Rates of contraction of posterior distributions based on Gaussian process priors
A. W. van der Vaart and J. H. van Zanten
Source: Ann. Statist. Volume 36, Number 3
(2008), 1435-1463.
Abstract
We derive rates of contraction of posterior distributions on nonparametric or semiparametric models based on Gaussian processes. The rate of contraction is shown to depend on the position of the true parameter relative to the reproducing kernel Hilbert space of the Gaussian process and the small ball probabilities of the Gaussian process. We determine these quantities for a range of examples of Gaussian priors and in several statistical settings. For instance, we consider the rate of contraction of the posterior distribution based on sampling from a smooth density model when the prior models the log density as a (fractionally integrated) Brownian motion. We also consider regression with Gaussian errors and smooth classification under a logistic or probit link function combined with various priors.
First Page:
Show
Hide
Keywords: Rate of convergence; Bayesian inference; nonparametric density estimation; nonparametric regression; classification
Full-text: Open access
Links and Identifiers
Permanent link to this document: http://projecteuclid.org/euclid.aos/1211819570
Digital Object Identifier: doi:10.1214/009053607000000613
Mathematical Reviews number (MathSciNet): MR2418663
Zentralblatt MATH identifier: 1141.60018
References
[1] Albert, J. H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. J. Amer. Statist. Assoc. 88 422, 669–679.
Mathematical Reviews (MathSciNet): MR1224394
Digital Object Identifier: doi:10.2307/2290350
JSTOR: links.jstor.org
Zentralblatt MATH: 0774.62031
[2] Ayache, A. and Taqqu, M. S. (2003). Rate optimality of wavelet series approximations of fractional Brownian motion. J. Fourier Anal. Appl. 9 451–471.
Mathematical Reviews (MathSciNet): MR2027888
Digital Object Identifier: doi:10.1007/s00041-003-0022-0
Zentralblatt MATH: 1050.60043
[3] Borell, C. (1975). The Brunn–Minkowski inequality in Gauss space. Invent. Math. 30 207–216.
Mathematical Reviews (MathSciNet): MR399402
Digital Object Identifier: doi:10.1007/BF01425510
Zentralblatt MATH: 0311.60007
[4] Choi, T. and Schervish, M. (2004). Posterior consistency in nonparametric regression problem under Gaussian process prior. Preprint.
Mathematical Reviews (MathSciNet): MR2396949
Digital Object Identifier: doi:10.1016/j.jmva.2007.01.004
Zentralblatt MATH: 1138.62020
[5] Choudhuri, N., Ghosal, S. and Roy, A. (2007). Nonparametric binary regression using a Gaussian process prior. Statist. Methodol. 4 227–243.
Mathematical Reviews (MathSciNet): MR2368147
Digital Object Identifier: doi:10.1016/j.stamet.2006.07.003
[6] Dzhaparidze, K. and van Zanten, H. (2004). A series expansion of fractional Brownian motion. Probab. Theory Related Fields 130 39–55.
Mathematical Reviews (MathSciNet): MR2092872
Digital Object Identifier: doi:10.1007/s00440-003-0310-2
Zentralblatt MATH: 1059.60048
[7] Dzhaparidze, K. and van Zanten, H. (2005). Krein’s spectral theory and the Paley–Wiener expansion for fractional Brownian motion. Ann. Probab. 33 620–644.
[8] Ghosal, S., Ghosh, J. K. and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist. 28 500–531.
Mathematical Reviews (MathSciNet): MR1790007
Digital Object Identifier: doi:10.1214/aos/1016218228
Project Euclid: euclid.aos/1016218228
[9] Ghosal, S. and Roy, A. (2006). Posterior consistency in nonparametric regression problem under Gaussian process prior. Ann. Statist. 34 2413–2429.
Mathematical Reviews (MathSciNet): MR2291505
Digital Object Identifier: doi:10.1214/009053606000000795
Project Euclid: euclid.aos/1169571802
Zentralblatt MATH: 1106.62039
[10] Ghosal, S. and van der Vaart, A. (2007). Posterior convergence rates of Dirichlet mixtures at smooth densities. Ann. Statist. 35 697–723.
Mathematical Reviews (MathSciNet): MR2336864
Digital Object Identifier: doi:10.1214/009053606000001271
Project Euclid: euclid.aos/1183667289
Zentralblatt MATH: 1117.62046
[11] Ghosal, S. and van der Vaart, A. (2007). Convergence rates of posterior distributions for noniid observations. Ann. Statist. 35 192–223.
[12] Ghosh, J. K. and Ramamoorthi, R. V. (2003). Bayesian Nonparametrics. Springer, New York.
Mathematical Reviews (MathSciNet): MR1992245
Zentralblatt MATH: 1029.62004
[13] Hult, H. (2003). Approximating some Volterra type stochastic integrals with applications to parameter estimation. Stochastic Process. Appl. 105 1–32.
Mathematical Reviews (MathSciNet): MR1972287
Digital Object Identifier: doi:10.1016/S0304-4149(02)00250-8
Zentralblatt MATH: 1075.60532
[14] Iglói, E. (2005). A rate-optimal trigonometric series expansion of the fractional Brownian motion. Electron. J. Probab. 10 1381–1397 (electronic).
[15] Kimeldorf, G. S. and Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Statist. 41 495–502.
Mathematical Reviews (MathSciNet): MR254999
Digital Object Identifier: doi:10.1214/aoms/1177697089
Project Euclid: euclid.aoms/1177697089
[16] Kuelbs, J., Li, W. V. and Linde, W. (1994). The Gaussian measure of shifted balls. Probab. Theory Related Fields 98 143–162.
Mathematical Reviews (MathSciNet): MR1258983
Digital Object Identifier: doi:10.1007/BF01192511
Zentralblatt MATH: 0792.60004
[17] Kühn, T. and Linde, W. (2002). Optimal series representation of fractional Brownian sheets. Bernoulli 8 669–696.
[18] Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. Springer, Berlin.
Mathematical Reviews (MathSciNet): MR1102015
Zentralblatt MATH: 0748.60004
[19] Lenk, P. J. (1988). The logistic normal distribution for Bayesian, nonparametric, predictive densities. J. Amer. Statist. Assoc. 83 509–516.
Mathematical Reviews (MathSciNet): MR971380
Digital Object Identifier: doi:10.2307/2288870
JSTOR: links.jstor.org
Zentralblatt MATH: 0648.62034
[20] Lenk, P. J. (1991). Towards a practicable Bayesian nonparametric density estimator. Biometrika 78 531–543.
Mathematical Reviews (MathSciNet): MR1130921
Zentralblatt MATH: 0737.62035
Digital Object Identifier: doi:10.1093/biomet/78.3.531
JSTOR: links.jstor.org
[21] Lenk, P. J. (1999). Bayesian inference for semiparametric regression using a Fourier representation. J. R. Stat. Soc. Ser. B Stat. Methodol. 61 863–879.
Mathematical Reviews (MathSciNet): MR1722244
Digital Object Identifier: doi:10.1111/1467-9868.00207
JSTOR: links.jstor.org
Zentralblatt MATH: 0940.62021
[22] Leonard, T. (1978). Density estimation, stochastic processes and prior information (with discussion). J. Roy. Statist. Soc. Ser. B 40 113–146.
Mathematical Reviews (MathSciNet): MR517434
JSTOR: links.jstor.org
[23] Li, W. V. and Linde, W. (1998). Existence of small ball constants for fractional Brownian motions. C. R. Acad. Sci. Paris Sér. I Math. 326 1329–1334.
Mathematical Reviews (MathSciNet): MR1649147
Digital Object Identifier: doi:10.1016/S0764-4442(98)80220-6
[24] Li, W. V. and Shao, Q.-M. (2001). Gaussian processes: Inequalities, small ball probabilities and applications. In Stochastic Processes: Theory and Methods. Handbook of Statist. 19 533–597. North-Holland, Amsterdam.
Mathematical Reviews (MathSciNet): MR1861734
Zentralblatt MATH: 0987.60053
[25] Mandelbrot, B. B. and Van Ness, J. W. (1968). Fractional Brownian motions, fractional noises and applications. SIAM Rev. 10 422–437.
Mathematical Reviews (MathSciNet): MR242239
Digital Object Identifier: doi:10.1137/1010093
JSTOR: links.jstor.org
Zentralblatt MATH: 0179.47801
[26] Neal, R. (1996). Bayesian Learning for Neural Networks. Lecture Notes in Statist. 118. Springer, New York.
[27] Rasmussen, C. E. and Williams, C. K. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.
Mathematical Reviews (MathSciNet): MR2514435
Zentralblatt MATH: 1177.68165
[28] Samko, S. G., Kilbas, A. A. and Marichev, O. I. (1993). Fractional Integrals and Derivatives. Gordon and Breach, Yverdon.
Mathematical Reviews (MathSciNet): MR1347689
Zentralblatt MATH: 0818.26003
[29] Tokdar, S. and Ghosh, J. (2005). Posterior consistency of Gaussian process priors in density estimation. J. Statist. Plann. Inference 137 34–42.
[30] van der Vaart, A. and Van Zanten, J. (2007). Reproducing kernel Hilbert spaces of Gaussian priors. In Festschrift for J. K. Ghosh. IMS Lecture Note Series, 2008. To appear.
Mathematical Reviews (MathSciNet): MR2459226
Digital Object Identifier: doi:10.1214/074921708000000156
[31] van der Vaart, A. W. (1988). Statistical Estimation in Large Parameter Spaces. Math. Centrum, Amsterdam.
Mathematical Reviews (MathSciNet): MR927725
Zentralblatt MATH: 0629.62035
[32] van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. Springer, New York.
Mathematical Reviews (MathSciNet): MR1385671
Zentralblatt MATH: 0862.60002
[33] Wahba, G. (1978). Improper priors, spline smoothing and the problem of guarding against model errors in regression. J. Roy. Statist. Soc. Ser. B 40 364–372.
Mathematical Reviews (MathSciNet): MR522220
JSTOR: links.jstor.org
[34] Wood, S. and Kohn, R. (1998). A Bayesian approach to robust nonparametric binary regression. J. Amer. Statist. Assoc. 93 203–213.
