The Annals of Statistics

Rates of contraction of posterior distributions based on Gaussian process priors

A. W. van der Vaart and J. H. van Zanten

Full-text: Open access

Abstract

We derive rates of contraction of posterior distributions on nonparametric or semiparametric models based on Gaussian processes. The rate of contraction is shown to depend on the position of the true parameter relative to the reproducing kernel Hilbert space of the Gaussian process and the small ball probabilities of the Gaussian process. We determine these quantities for a range of examples of Gaussian priors and in several statistical settings. For instance, we consider the rate of contraction of the posterior distribution based on sampling from a smooth density model when the prior models the log density as a (fractionally integrated) Brownian motion. We also consider regression with Gaussian errors and smooth classification under a logistic or probit link function combined with various priors.

Article information

Source
Ann. Statist. Volume 36, Number 3 (2008), 1435-1463.

Dates
First available in Project Euclid: 26 May 2008

Permanent link to this document
http://projecteuclid.org/euclid.aos/1211819570

Digital Object Identifier
doi:10.1214/009053607000000613

Mathematical Reviews number (MathSciNet)
MR2418663

Zentralblatt MATH identifier
1141.60018

Subjects
Primary: 60G15: Gaussian processes 62G05: Estimation

Keywords
Rate of convergence Bayesian inference nonparametric density estimation nonparametric regression classification

Citation

van der Vaart, A. W.; van Zanten, J. H. Rates of contraction of posterior distributions based on Gaussian process priors. Ann. Statist. 36 (2008), no. 3, 1435--1463. doi:10.1214/009053607000000613. http://projecteuclid.org/euclid.aos/1211819570.


Export citation

References

  • [1] Albert, J. H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. J. Amer. Statist. Assoc. 88 422, 669–679.
  • [2] Ayache, A. and Taqqu, M. S. (2003). Rate optimality of wavelet series approximations of fractional Brownian motion. J. Fourier Anal. Appl. 9 451–471.
  • [3] Borell, C. (1975). The Brunn–Minkowski inequality in Gauss space. Invent. Math. 30 207–216.
  • [4] Choi, T. and Schervish, M. (2004). Posterior consistency in nonparametric regression problem under Gaussian process prior. Preprint.
  • [5] Choudhuri, N., Ghosal, S. and Roy, A. (2007). Nonparametric binary regression using a Gaussian process prior. Statist. Methodol. 4 227–243.
  • [6] Dzhaparidze, K. and van Zanten, H. (2004). A series expansion of fractional Brownian motion. Probab. Theory Related Fields 130 39–55.
  • [7] Dzhaparidze, K. and van Zanten, H. (2005). Krein’s spectral theory and the Paley–Wiener expansion for fractional Brownian motion. Ann. Probab. 33 620–644.
  • [8] Ghosal, S., Ghosh, J. K. and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist. 28 500–531.
  • [9] Ghosal, S. and Roy, A. (2006). Posterior consistency in nonparametric regression problem under Gaussian process prior. Ann. Statist. 34 2413–2429.
  • [10] Ghosal, S. and van der Vaart, A. (2007). Posterior convergence rates of Dirichlet mixtures at smooth densities. Ann. Statist. 35 697–723.
  • [11] Ghosal, S. and van der Vaart, A. (2007). Convergence rates of posterior distributions for noniid observations. Ann. Statist. 35 192–223.
  • [12] Ghosh, J. K. and Ramamoorthi, R. V. (2003). Bayesian Nonparametrics. Springer, New York.
  • [13] Hult, H. (2003). Approximating some Volterra type stochastic integrals with applications to parameter estimation. Stochastic Process. Appl. 105 1–32.
  • [14] Iglói, E. (2005). A rate-optimal trigonometric series expansion of the fractional Brownian motion. Electron. J. Probab. 10 1381–1397 (electronic).
  • [15] Kimeldorf, G. S. and Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Statist. 41 495–502.
  • [16] Kuelbs, J., Li, W. V. and Linde, W. (1994). The Gaussian measure of shifted balls. Probab. Theory Related Fields 98 143–162.
  • [17] Kühn, T. and Linde, W. (2002). Optimal series representation of fractional Brownian sheets. Bernoulli 8 669–696.
  • [18] Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. Springer, Berlin.
  • [19] Lenk, P. J. (1988). The logistic normal distribution for Bayesian, nonparametric, predictive densities. J. Amer. Statist. Assoc. 83 509–516.
  • [20] Lenk, P. J. (1991). Towards a practicable Bayesian nonparametric density estimator. Biometrika 78 531–543.
  • [21] Lenk, P. J. (1999). Bayesian inference for semiparametric regression using a Fourier representation. J. R. Stat. Soc. Ser. B Stat. Methodol. 61 863–879.
  • [22] Leonard, T. (1978). Density estimation, stochastic processes and prior information (with discussion). J. Roy. Statist. Soc. Ser. B 40 113–146.
  • [23] Li, W. V. and Linde, W. (1998). Existence of small ball constants for fractional Brownian motions. C. R. Acad. Sci. Paris Sér. I Math. 326 1329–1334.
  • [24] Li, W. V. and Shao, Q.-M. (2001). Gaussian processes: Inequalities, small ball probabilities and applications. In Stochastic Processes: Theory and Methods. Handbook of Statist. 19 533–597. North-Holland, Amsterdam.
  • [25] Mandelbrot, B. B. and Van Ness, J. W. (1968). Fractional Brownian motions, fractional noises and applications. SIAM Rev. 10 422–437.
  • [26] Neal, R. (1996). Bayesian Learning for Neural Networks. Lecture Notes in Statist. 118. Springer, New York.
  • [27] Rasmussen, C. E. and Williams, C. K. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.
  • [28] Samko, S. G., Kilbas, A. A. and Marichev, O. I. (1993). Fractional Integrals and Derivatives. Gordon and Breach, Yverdon.
  • [29] Tokdar, S. and Ghosh, J. (2005). Posterior consistency of Gaussian process priors in density estimation. J. Statist. Plann. Inference 137 34–42.
  • [30] van der Vaart, A. and Van Zanten, J. (2007). Reproducing kernel Hilbert spaces of Gaussian priors. In Festschrift for J. K. Ghosh. IMS Lecture Note Series, 2008. To appear.
  • [31] van der Vaart, A. W. (1988). Statistical Estimation in Large Parameter Spaces. Math. Centrum, Amsterdam.
  • [32] van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. Springer, New York.
  • [33] Wahba, G. (1978). Improper priors, spline smoothing and the problem of guarding against model errors in regression. J. Roy. Statist. Soc. Ser. B 40 364–372.
  • [34] Wood, S. and Kohn, R. (1998). A Bayesian approach to robust nonparametric binary regression. J. Amer. Statist. Assoc. 93 203–213.