Bernoulli

• Bernoulli
• Volume 26, Number 1 (2020), 616-641.

On frequentist coverage errors of Bayesian credible sets in moderately high dimensions

Abstract

In this paper, we study frequentist coverage errors of Bayesian credible sets for an approximately linear regression model with (moderately) high dimensional regressors, where the dimension of the regressors may increase with but is smaller than the sample size. Specifically, we consider quasi-Bayesian inference on the slope vector under the quasi-likelihood with Gaussian error distribution. Under this setup, we derive finite sample bounds on frequentist coverage errors of Bayesian credible rectangles. Derivation of those bounds builds on a novel Berry–Esseen type bound on quasi-posterior distributions and recent results on high-dimensional CLT on hyperrectangles. We use this general result to quantify coverage errors of Castillo–Nickl and $L^{\infty}$-credible bands for Gaussian white noise models, linear inverse problems, and (possibly non-Gaussian) nonparametric regression models. In particular, we show that Bayesian credible bands for those nonparametric models have coverage errors decaying polynomially fast in the sample size, implying advantages of Bayesian credible bands over confidence bands based on extreme value theory.

Article information

Source
Bernoulli, Volume 26, Number 1 (2020), 616-641.

Dates
Revised: June 2019
First available in Project Euclid: 26 November 2019

https://projecteuclid.org/euclid.bj/1574758840

Digital Object Identifier
doi:10.3150/19-BEJ1142

Mathematical Reviews number (MathSciNet)
MR4036046

Zentralblatt MATH identifier
07140511

Citation

Yano, Keisuke; Kato, Kengo. On frequentist coverage errors of Bayesian credible sets in moderately high dimensions. Bernoulli 26 (2020), no. 1, 616--641. doi:10.3150/19-BEJ1142. https://projecteuclid.org/euclid.bj/1574758840

References

• [1] Abramovich, F. and Silverman, B.W. (1998). Wavelet decomposition approaches to statistical inverse problems. Biometrika 85 115–129.
• [2] Anderson, T.W. (1955). The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities. Proc. Amer. Math. Soc. 6 170–176.
• [3] Atchadé, Y.A. (2017). On the contraction properties of some high-dimensional quasi-posterior distributions. Ann. Statist. 45 2248–2273.
• [4] Belloni, A., Chernozhukov, V., Chetverikov, D. and Kato, K. (2015). Some new asymptotic theory for least squares series: Pointwise and uniform results. J. Econometrics 186 345–366.
• [5] Bickel, P.J. and Rosenblatt, M. (1973). On some global measures of the deviations of density function estimates. Ann. Statist. 1 1071–1095.
• [6] Bontemps, D. (2011). Bernstein–von Mises theorems for Gaussian regression with increasing number of regressors. Ann. Statist. 39 2557–2584.
• [7] Boucheron, S. and Gassiat, E. (2009). A Bernstein–von Mises theorem for discrete probability distributions. Electron. J. Stat. 3 114–148.
• [8] Castillo, I. (2014). On Bayesian supremum norm contraction rates. Ann. Statist. 42 2058–2091.
• [9] Castillo, I. and Nickl, R. (2013). Nonparametric Bernstein–von Mises theorems in Gaussian white noise. Ann. Statist. 41 1999–2028.
• [10] Castillo, I. and Nickl, R. (2014). On the Bernstein–von Mises phenomenon for nonparametric Bayes procedures. Ann. Statist. 42 1941–1969.
• [11] Castillo, I. and Rousseau, J. (2015). A Bernstein–von Mises theorem for smooth functionals in semiparametric models. Ann. Statist. 43 2353–2383.
• [12] Castillo, I., Schmidt-Hieber, J. and van der Vaart, A. (2015). Bayesian linear regression with sparse priors. Ann. Statist. 43 1986–2018.
• [13] Chernozhukov, V., Chetverikov, D. and Kato, K. (2013). Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. Ann. Statist. 41 2786–2819.
• [14] Chernozhukov, V., Chetverikov, D. and Kato, K. (2014). Anti-concentration and honest, adaptive confidence bands. Ann. Statist. 42 1787–1818.
• [15] Chernozhukov, V., Chetverikov, D. and Kato, K. (2017). Central limit theorems and bootstrap in high dimensions. Ann. Probab. 45 2309–2352.
• [16] Chernozhukov, V. and Hong, H. (2003). An MCMC approach to classical estimation. J. Econometrics 115 293–346.
• [17] Claeskens, G. and Van Keilegom, I. (2003). Bootstrap confidence bands for regression curves and their derivatives. Ann. Statist. 31 1852–1884.
• [18] Clarke, B. and Ghosal, S. (2010). Reference priors for exponential families with increasing dimension. Electron. J. Stat. 4 737–780.
• [19] Cox, D.D. (1993). An analysis of Bayesian inference for nonparametric regression. Ann. Statist. 21 903–923.
• [20] DeVore, R.A. and Lorentz, G.G. (1993). Constructive Approximation. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 303. Berlin: Springer.
• [21] Donoho, D.L. (1995). Nonlinear solution of linear inverse problems by wavelet-vaguelette decomposition. Appl. Comput. Harmon. Anal. 2 101–126.
• [22] Florens, J.-P. and Simoni, A. (2012). Nonparametric estimation of an instrumental regression: A quasi-Bayesian approach based on regularized posterior. J. Econometrics 170 458–475.
• [23] Freedman, D. (1999). On the Bernstein–von Mises theorem with infinite-dimensional parameters. Ann. Statist. 27 1119–1140.
• [24] Gao, C. and Zhou, H.H. (2016). Bernstein–von Mises theorems for functionals of the covariance matrix. Electron. J. Stat. 10 1751–1806.
• [25] Ghosal, S. (1999). Asymptotic normality of posterior distributions in high-dimensional linear models. Bernoulli 5 315–331.
• [26] Ghosal, S. (2000). Asymptotic normality of posterior distributions for exponential families when the number of parameters tends to infinity. J. Multivariate Anal. 74 49–68.
• [27] Ghosal, S. and van der Vaart, A. (2017). Fundamentals of Nonparametric Bayesian Inference. Cambridge Series in Statistical and Probabilistic Mathematics 44. Cambridge: Cambridge Univ. Press.
• [28] Giné, E. and Nickl, R. (2010). Confidence bands in density estimation. Ann. Statist. 38 1122–1170.
• [29] Giné, E. and Nickl, R. (2011). Rates of contraction for posterior distributions in $L^{r}$-metrics, $1\leq r\leq\infty$. Ann. Statist. 39 2883–2911.
• [30] Giné, E. and Nickl, R. (2016). Mathematical Foundations of Infinite-Dimensional Statistical Models. Cambridge Series in Statistical and Probabilistic Mathematics. New York: Cambridge Univ. Press.
• [31] Hall, P. (1991). On convergence rates of suprema. Probab. Theory Related Fields 89 447–455.
• [32] Hoffmann, M., Rousseau, J. and Schmidt-Hieber, J. (2015). On adaptive posterior concentration rates. Ann. Statist. 43 2259–2295.
• [33] Johnstone, I. Gaussian Estimation: Sequence and Wavelet Models. Unpublished manuscript.
• [34] Johnstone, I.M. (2010). High dimensional Bernstein–von Mises: Simple examples. In Borrowing Strength: Theory Powering Application – a Festschrift for Lawrence D. Brown. Inst. Math. Stat. (IMS) Collect. 6 87–98. Beachwood, OH: IMS.
• [35] Kato, K. (2013). Quasi-Bayesian analysis of nonparametric instrumental variables models. Ann. Statist. 41 2359–2390.
• [36] Knapik, B.T., van der Vaart, A.W. and van Zanten, J.H. (2011). Bayesian inverse problems with Gaussian priors. Ann. Statist. 39 2626–2657.
• [37] Leahu, H. (2011). On the Bernstein–von Mises phenomenon in the Gaussian white noise model. Electron. J. Stat. 5 373–404.
• [38] Nazarov, F. (2003). On the maximal perimeter of a convex set in ${\mathbb{R}}^{n}$ with respect to a Gaussian measure. In Geometric Aspects of Functional Analysis. Lecture Notes in Math. 1807 169–187. Berlin: Springer.
• [39] Nickl, R. (2019). Bernstein–von Mises theorems for statistical inverse problems I: Schrödinger equation. J. Eur. Math. Soc. (JEMS). To appear.
• [40] Nickl, R. and Söhl, J. (2017). Nonparametric Bayesian posterior contraction rates for discretely observed scalar diffusions. Ann. Statist. 45 1664–1693.
• [41] Panov, M. and Spokoiny, V. (2015). Finite sample Bernstein–von Mises theorem for semiparametric problems. Bayesian Anal. 10 665–710.
• [42] Ray, K. (2017). Adaptive Bernstein–von Mises theorems in Gaussian white noise. Ann. Statist. 45 2511–2536.
• [43] Rivoirard, V. and Rousseau, J. (2012). Bernstein–von Mises theorem for linear functionals of the density. Ann. Statist. 40 1489–1523.
• [44] Smirnov, N.V. (1950). On the construction of confidence regions for the density of distribution of random variables. Dokl. Akad. Nauk SSSR 74 189–191.
• [45] Sniekers, S. and van der Vaart, A. (2015). Credible sets in the fixed design model with Brownian motion prior. J. Statist. Plann. Inference 166 78–86.
• [46] Spokoiny, V. Bernstein–von Mises Theorem for growing parameter dimension. Available at arXiv:1302.3430.
• [47] Szabó, B., van der Vaart, A. and van Zanten, H. (2015). Honest Bayesian confidence sets for the $L^{2}$-norm. J. Statist. Plann. Inference 166 36–51.
• [48] Szabó, B., van der Vaart, A.W. and van Zanten, J.H. (2015). Frequentist coverage of adaptive nonparametric Bayesian credible sets. Ann. Statist. 43 1391–1428.
• [49] van der Vaart, A.W. (1998). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics 3. Cambridge: Cambridge Univ. Press.
• [50] Yang, Y., Bhattacharya, A. and Pati, D. Frequentist coverage and sup-norm convergence rate in Gaussian process regression. Available at arXiv:1708.04753.
• [51] Yano, K. and Kato, K. (2020). Supplement to “On frequentist coverage errors of Bayesian credible sets in moderately high dimensions.” https://doi.org/10.3150/19-BEJ1142SUPP.
• [52] Yoo, W., Rousseau, J. and Rivoirard, V. Adaptive supremum norm posterior contraction: Wavelet spike-and-slab and anisotropic Besov spaces. Available at arXiv:1708.01909.
• [53] Yoo, W.W. and Ghosal, S. (2016). Supremum norm posterior contraction and credible sets for nonparametric multivariate regression. Ann. Statist. 44 1069–1102.

Supplemental materials

• Supplement to “On frequentist coverage errors of Bayesian credible sets in high dimensions”. The supplementary material contains the proofs omitted in the main text.