Bayesian Analysis

Laplace Approximation for Logistic Gaussian Process Density Estimation and Regression

Jaakko Riihimäki and Aki Vehtari

Full-text: Open access

Enhanced PDF (702 KB)

Abstract

Logistic Gaussian process (LGP) priors provide a flexible alternative for modelling unknown densities. The smoothness properties of the density estimates can be controlled through the prior covariance structure of the LGP, but the challenge is the analytically intractable inference. In this paper, we present approximate Bayesian inference for LGP density estimation in a grid using Laplace’s method to integrate over the non-Gaussian posterior distribution of latent function values and to determine the covariance function parameters with type-II maximum a posteriori (MAP) estimation. We demonstrate that Laplace’s method with MAP is sufficiently fast for practical interactive visualisation of 1D and 2D densities. Our experiments with simulated and real 1D data sets show that the estimation accuracy is close to a Markov chain Monte Carlo approximation and state-of-the-art hierarchical infinite Gaussian mixture models. We also construct a reduced-rank approximation to speed up the computations for dense 2D grids, and demonstrate density regression with the proposed Laplace approach.

Article information

Source
Bayesian Anal., Volume 9, Number 2 (2014), 425-448.

Dates
First available in Project Euclid: 26 May 2014

Permanent link to this document
https://projecteuclid.org/euclid.ba/1401148315

Digital Object Identifier
doi:10.1214/14-BA872

Mathematical Reviews number (MathSciNet)
MR3217002

Zentralblatt MATH identifier
1327.62248

Keywords
Gaussian process logistic transformation density estimation density regression approximate inference Laplace’s method

Citation

Riihimäki, Jaakko; Vehtari, Aki. Laplace Approximation for Logistic Gaussian Process Density Estimation and Regression. Bayesian Anal. 9 (2014), no. 2, 425--448. doi:10.1214/14-BA872. https://projecteuclid.org/euclid.ba/1401148315


Export citation

References

  • Adams, R. P. (2009). “Kernel Methods for Nonparametric Bayesian Inference of Probabilities and Point Processes.” Ph.D. thesis, University of Cambridge.
  • Adams, R. P., Murray, I., and MacKay, D. J. (2009). “The Gaussian process density sampler.” In Advances in Neural Information Processing Systems 21, 9–16. NIPS foundation.
  • Brooks, S. P. and Gelman, A. (1998). “General Methods for Monitoring Convergence of Iterative Simulations.” Journal of Computational and Graphical Statistics, 7(4): 434–455.
    Mathematical Reviews (MathSciNet): MR1665662
  • Cseke, B. and Heskes, T. (2011). “Approximate marginals in latent Gaussian models.” Journal of Machine Learning Research, 12: 417–454.
    Mathematical Reviews (MathSciNet): MR2783173
  • Cunningham, J. P., Shenoy, K. V., and Sahani, M. (2008). “Fast Gaussian process methods for point process intensity estimation.” In McCallum, A. and Roweis, S. (eds.), Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008), 192–199. Omnipress.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models.” Bayesian Analysis, 1(3): 515–533.
    Mathematical Reviews (MathSciNet): MR2221284
    Digital Object Identifier: doi:10.1214/06-BA117A
  • Geweke, J. (1989). “Bayesian Inference in Econometric Models Using Monte Carlo Integration.” Econometrica, 57(6): 1317–1339.
    Mathematical Reviews (MathSciNet): MR1035115
    Digital Object Identifier: doi:10.2307/1913710
  • Geyer, C. J. (1992). “Practical Markov chain Monte Carlo.” Statistical Science, 7(4): 473–483.
  • Griebel, M. and Hegland, M. (2010). “A Finite Element Method for Density Estimation with Gaussian Process Priors.” SIAM Journal of Numerical Analysis, 47(6): 4759–4792.
    Mathematical Reviews (MathSciNet): MR2595057
    Digital Object Identifier: doi:10.1137/080736478
  • Griffin, J. (2010). “Default priors for density estimation with mixture models.” Bayesian Analysis, 5(1): 45–64.
    Mathematical Reviews (MathSciNet): MR2596435
    Digital Object Identifier: doi:10.1214/10-BA502
    Project Euclid: euclid.ba/1340369792
  • Harville, D. A. (1997). Matrix Algebra From a Statistician’s Perspective. Springer-Verlag.
    Mathematical Reviews (MathSciNet): MR1467237
  • Hoffmann, M. and Gelman, A. (2013). “The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo.” Journal of Machine Learning Research. In press.
  • Jylänki, P., Vanhatalo, J., and Vehtari, A. (2011). “Robust Gaussian Process Regression with a Student-t Likelihood.” Journal of Machine Learning Research, 12: 3227–3257.
    Mathematical Reviews (MathSciNet): MR2877599
  • Kong, A., Liu, J. S., and Wong, W. H. (1994). “Sequential Imputations and Bayesian Missing Data Problems.” Journal of the American Statistical Association, 89(425): 278–288.
  • Kundu, S. and Dunson, D. B. (2011). “Latent Factor Models for Density Estimation.” ArXiv:1108.2720v2.
    arXiv: 1108.2720v2
  • Lenk, P. J. (1991). “Towards a Practicable Bayesian Nonparametric Density Estimator.” Biometrika, 78(3): 531–543.
    Mathematical Reviews (MathSciNet): MR1130921
    Digital Object Identifier: doi:10.1093/biomet/78.3.531
  • — (2003). “Bayesian Semiparametric Density Estimation and Model Verification Using a Logistic-Gaussian Process.” Journal of Computational and Graphical Statistics, 12(3): 548–565.
    Mathematical Reviews (MathSciNet): MR2002635
    Digital Object Identifier: doi:10.1198/1061860032021
  • Leonard, T. (1978). “Density Estimation, Stochastic Processes, and Prior Information.” Journal of the Royal Statistical Society: Series B (Methodological), 40(2): 113–146.
    Mathematical Reviews (MathSciNet): MR517434
  • Neal, R. (1998). “Regression and Classification Using Gaussian Process Priors.” In Bernardo, J. M., Berger, J. O., David, A. P., and Smith, A. P. M. (eds.), Bayesian Statistics 6, 475–501. Oxford University Press.
    Mathematical Reviews (MathSciNet): MR1723510
  • Nickisch, H. and Rasmussen, C. E. (2008). “Approximations for Binary Gaussian Process Classification.” Journal of Machine Learning Research, 9: 2035–2078.
    Mathematical Reviews (MathSciNet): MR2452620
  • O’Hagan, A. (1978). “Curve fitting and optimal design for prediction.” Journal of the Royal Statistical Society. Series B (Methodological), 40(1): 1–42.
    Mathematical Reviews (MathSciNet): MR512140
  • Quiñonero-Candela, J. and Rasmussen, C. E. (2005). “A Unifying View of Sparse Approximate Gaussian Process Regression.” Journal of Machine Learning Research, 6(3): 1939–1959.
    Mathematical Reviews (MathSciNet): MR2249877
  • Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. Cambridge: MIT Press.
    Mathematical Reviews (MathSciNet): MR2514435
  • Riihimäki, J., Jylänki, P., and Vehtari, A. (2013). “Nested Expectation Propagation for Gaussian Process Classification with a Multinomial Probit Likelihood.” Journal of Machine Learning Research, 14: 75–109.
    Mathematical Reviews (MathSciNet): MR3033326
  • Rubin, D. B. (1981). “The Bayesian Bootstrap.” Annals of Statistics, 9(1): 130–134.
    Mathematical Reviews (MathSciNet): MR600538
    Digital Object Identifier: doi:10.1214/aos/1176345338
    Project Euclid: euclid.aos/1176345338
  • Rue, H., Martino, S., and Chopin, N. (2009). “Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations.” Journal of the Royal Statistical Society. Series B (Statistical Methodology), 71(2): 319–392.
    Mathematical Reviews (MathSciNet): MR2649602
    Digital Object Identifier: doi:10.1111/j.1467-9868.2008.00700.x
  • Snelson, E. and Ghahramani, Z. (2006). “Sparse Gaussian Processes Using Pseudo-Inputs.” In Weiss, Y., Schölkopf, B., and Platt, J. (eds.), Advances in Neural Information Processing Systems 18. The MIT Press.
    Mathematical Reviews (MathSciNet): MR2441315
  • Thorburn, D. (1986). “A Bayesian Approach to Density Estimation.” Biometrika, 73(1): 65–75.
    Mathematical Reviews (MathSciNet): MR836435
    Digital Object Identifier: doi:10.2307/2336272
  • Tokdar, S. T. (2007). “Towards a Faster Implementation of Density Estimation With Logistic Gaussian Process Priors.” Journal of Computational and Graphical Statistics, 16(3): 633–655.
    Mathematical Reviews (MathSciNet): MR2351083
    Digital Object Identifier: doi:10.1198/106186007X210206
  • Tokdar, S. T. and Ghosh, J. K. (2007). “Posterior consistency of logistic Gaussian process priors in density estimation.” Journal of Statistical Planning and Inference, 137(1): 34–42.
    Mathematical Reviews (MathSciNet): MR2292838
    Digital Object Identifier: doi:10.1016/j.jspi.2005.09.005
  • Tokdar, S. T., Zhu, Y. M., and Ghosh, J. K. (2010). “Bayesian Density Regression with Logistic Gaussian Process and Subspace Projection.” Bayesian Analysis, 5(2): 319–344.
    Mathematical Reviews (MathSciNet): MR2719655
    Digital Object Identifier: doi:10.1214/10-BA605
    Project Euclid: euclid.ba/1340218341
  • van der Vaart, A. W. and van Zanten, J. H. (2009). “Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth.” The Annals of Statistics, 37(5B): 2655–2675.
    Mathematical Reviews (MathSciNet): MR2541442
    Digital Object Identifier: doi:10.1214/08-AOS678
    Project Euclid: euclid.aos/1247836664
  • Vanhatalo, J., Pietiläinen, V., and Vehtari, A. (2010). “Approximate inference for disease mapping with sparse Gaussian processes.” Statistics in Medicine, 29(15): 1580–1607.
    Mathematical Reviews (MathSciNet): MR2758849
  • Vanhatalo, J., Riihimäki, J., Hartikainen, J., Jylänki, P., Tolvanen, V., and Vehtari, A. (2013). “GPstuff: Bayesian Modeling with Gaussian Processes.” Journal of Machine Learning Research, 14: 1175–1179.
    Mathematical Reviews (MathSciNet): MR3063621
  • Vehtari, A. and Lampinen, J. (2002). “Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities.” Neural Computation, 14(10): 2439–2468.
  • Villani, M. and Larsson, R. (2006). “The Multivariate Split Normal Distribution and Asymmetric Principal Components Analysis.” Communications in Statistics: Theory & Methods, 35(6): 1123–1140.
    Mathematical Reviews (MathSciNet): MR2256269
    Digital Object Identifier: doi:10.1080/03610920600672252
  • Williams, C. K. I. and Barber, D. (1998). “Bayesian classification with Gaussian processes.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12): 1342–1351.

International Society for Bayesian Analysis

Bayesian Analysis

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more