Electronic Journal of Statistics

Empirical Bayes scaling of Gaussian priors in the white noise model

B. T. Szabó, A. W. van der Vaart, and J. H. van Zanten

Full-text: Open access

Abstract

The performance of nonparametric estimators is heavily dependent on a bandwidth parameter. In nonparametric Bayesian methods this parameter can be specified as a hyperparameter of the nonparametric prior. The value of this hyperparameter may be made dependent on the data. The empirical Bayes method is to set its value by maximizing the marginal likelihood of the data in the Bayesian framework. In this paper we analyze a particular version of this method, common in practice, that the hyperparameter scales the prior variance. We characterize the behavior of the random hyperparameter, and show that a nonparametric Bayes method using it gives optimal recovery over a scale of regularity classes. This scale is limited, however, by the regularity of the unscaled prior. While a prior can be scaled up to make it appropriate for arbitrarily rough truths, scaling cannot increase the nominal smoothness by much. Surprisingy the standard empirical Bayes method is even more limited in this respect than an oracle, deterministic scaling method. The same can be said for the hierarchical Bayes method.

Article information

Source
Electron. J. Statist. Volume 7 (2013), 991-1018.

Dates
First available in Project Euclid: 15 April 2013

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1366031048

Digital Object Identifier
doi:10.1214/13-EJS798

Mathematical Reviews number (MathSciNet)
MR3044507

Zentralblatt MATH identifier
1336.62039

Subjects
Primary: 62G05: Estimation 62G15: Tolerance and confidence regions
Secondary: 62G20: Asymptotic properties

Keywords
Adaptation hyper-rectangle Gaussian white noise normal means model bandwidth rate of contraction

Citation

Szabó, B. T.; van der Vaart, A. W.; van Zanten, J. H. Empirical Bayes scaling of Gaussian priors in the white noise model. Electron. J. Statist. 7 (2013), 991--1018. doi:10.1214/13-EJS798. https://projecteuclid.org/euclid.ejs/1366031048


Export citation

References

  • [1] Belitser, E., and Enikeeva, F. Empirical Bayesian test of the smoothness., Math. Methods Statist. 17, 1 (2008), 1–18.
  • [2] Belitser, E., and Ghosal, S. Adaptive Bayesian inference on the mean of an infinite-dimensional normal distribution., Ann. Statist. 31, 2 (2003), 536–559. Dedicated to the memory of Herbert E. Robbins.
  • [3] Brown, L. D., and Zhao, L. H. Estimators for Gaussian models having a block-wise structure., Statist. Sinica 19, 3 (2009), 885–903.
  • [4] Cai, T. T., Low, M. G., and Zhao, L. H. Sharp adaptive estimation by a blockwise method., J. Nonparametr. Stat. 21, 7 (2009), 839–850.
  • [5] Castillo, I. Lower bounds for posterior rates with Gaussian process priors., Electron. J. Stat. 2 (2008), 1281–1299.
  • [6] Cox, D. D. An analysis of Bayesian inference for nonparametric regression., Ann. Statist. 21, 2 (1993), 903–923.
  • [7] Donoho, D. L. Statistical estimation and optimal recovery., Ann. Statist. 22, 1 (1994), 238–270.
  • [8] Donoho, D. L., Liu, R. C., and MacGibbon, B. Minimax risk over hyperrectangles, and implications., Ann. Statist. 18, 3 (1990), 1416–1437.
  • [9] Efroĭmovich, S. Y., and Pinsker, M. S. Estimation of square-integrable probability density of a random variable., Problemy Peredachi Informatsii 18, 3 (1982), 19–38.
  • [10] Efron, B., and Morris, C. Limiting the risk of Bayes and empirical Bayes estimators. II. The empirical Bayes case., J. Amer. Statist. Assoc. 67 (1972), 130–139.
  • [11] Florens, J., and Simoni, A. Regularizing priors for linear inverse problems., preprint .
  • [12] Freedman, D. On the Bernstein-von Mises theorem with infinite-dimensional parameters., Ann. Statist. 27, 4 (1999), 1119–1140.
  • [13] Ghosal, S., Ghosh, J. K., and van der Vaart, A. W. Convergence rates of posterior distributions., Ann. Statist. 28, 2 (2000), 500–531.
  • [14] Ghosal, S., Lember, J., and van der Vaart, A. Nonparametric Bayesian model selection and averaging., Electron. J. Stat. 2 (2008), 63–89.
  • [15] Ghosh, J. K., and Ramamoorthi, R. V., Bayesian nonparametrics. Springer Series in Statistics. Springer-Verlag, New York, 2003.
  • [16] Jiang, W., and Zhang, C.-H. General maximum likelihood empirical Bayes estimation of normal means., Ann. Statist. 37, 4 (2009), 1647–1684.
  • [17] Johnstone, I. M., and Silverman, B. W. Needles and straw in haystacks: empirical Bayes estimates of possibly sparse sequences., Ann. Statist. 32, 4 (2004), 1594–1649.
  • [18] Knapik, B. T., Szabó, B., van der Vaart, A. W., and van Zanten, J. H. Bayes procedures for adaptive inference in nonparametric inverse problems., Preprint, arXiv :1209.3628 [math.ST] (2012).
  • [19] Knapik, B. T., van der Vaart, A. W., and van Zanten, J. H. Bayesian inverse problems with Gaussian priors., Ann. Statist. 39, 5 (2011), 2626–2657.
  • [20] Lember, J., and van der Vaart, A. On universal Bayesian adaptation., Statist. Decisions 25, 2 (2007), 127–152.
  • [21] Pinsker, M. S. Optimal filtration of square-integrable signals in Gaussian, noise.
  • [22] Robbins, H. An empirical Bayes approach to statistics. In, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1954–1955, vol. I (Berkeley and Los Angeles, 1956), University of California Press, pp. 157–163.
  • [23] Scricciolo, C. Convergence rates for Bayesian density estimation of infinite-dimensional exponential families., Ann. Statist. 34, 6 (2006), 2897–2920.
  • [24] Tsybakov, A. B., Introduction à l’estimation non-paramétrique, vol. 41 of Mathématiques & Applications (Berlin) [Mathematics & Applications]. Springer-Verlag, Berlin, 2004.
  • [25] van der Vaart, A., and van Zanten, H. Bayesian inference with rescaled Gaussian process priors., Electron. J. Stat. 1 (2007), 433–448 (electronic).
  • [26] van der Vaart, A. W., and van Zanten, J. H. Rates of contraction of posterior distributions based on Gaussian process priors., Ann. Statist. 36, 3 (2008), 1435–1463.
  • [27] van der Vaart, A. W., and van Zanten, J. H. Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth., Ann. Statist. 37, 5B (2009), 2655–2675.
  • [28] van der Vaart, A. W., and Wellner, J. A., Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York, 1996. With applications to statistics.
  • [29] Wahba, G. Improper priors, spline smoothing and the problem of guarding against model errors in regression., J. Roy. Statist. Soc. Ser. B 40, 3 (1978), 364–372.
  • [30] Zhang, C.-H. General empirical Bayes wavelet methods and exactly adaptive minimax estimation., Ann. Statist. 33, 1 (2005), 54–100.
  • [31] Zhao, L. H. Bayesian aspects of some nonparametric problems., Ann. Statist. 28, 2 (2000), 532–552.