Electronic Journal of Statistics

Criteria for posterior consistency and convergence at a rate

B. J. K. Kleijn and Y. Y. Zhao

Full-text: Open access

Abstract

Frequentist conditions for asymptotic consistency of Bayesian procedures with i.i.d. data focus on lower bounds for prior mass in Kullback-Leibler neighbourhoods of the data distribution. The goal of this paper is to investigate the flexibility in these criteria. We derive a versatile new posterior consistency theorem, which is used to consider Kullback-Leibler consistency and indicate when it is sufficient to have a prior that charges metric balls instead of KL-neighbourhoods. We generalize our proposal to sieved models with Barron’s negligible prior mass condition and to separable models with variations on Walker’s condition. Results are also applied in semi-parametric consistency: support boundary estimation is considered explicitly and consistency is proved in a model for which Kullback-Leibler priors do not exist. As a further demonstration of applicability, we consider metric consistency at a rate: under a mild integrability condition, the second-order Ghosal-Ghosh-van der Vaart prior mass condition can be relaxed to a lower bound for ordinary KL-neighbourhoods. The posterior rate is derived in a parametric model for heavy-tailed distributions in which the Ghosal-Ghosh-van der Vaart condition cannot be satisfied by any prior.

Article information

Source
Electron. J. Statist., Volume 13, Number 2 (2019), 4709-4742.

Dates
Received: October 2019
First available in Project Euclid: 20 November 2019

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1574240425

Digital Object Identifier
doi:10.1214/19-EJS1633

Mathematical Reviews number (MathSciNet)
MR4033683

Zentralblatt MATH identifier
07136628

Subjects
Primary: 62G07: Density estimation 62G20: Asymptotic properties

Keywords
Asymptotic consistency posterior consistency Bayesian consistency marginal consistency posterior rate of convergence

Rights
Creative Commons Attribution 4.0 International License.

Citation

Kleijn, B. J. K.; Zhao, Y. Y. Criteria for posterior consistency and convergence at a rate. Electron. J. Statist. 13 (2019), no. 2, 4709--4742. doi:10.1214/19-EJS1633. https://projecteuclid.org/euclid.ejs/1574240425


Export citation

References

  • [1] A. Barron, The exponential convergence of posterior probabilities with implications for Bayes estimators of density functions, Technical report nr. 7 (1988), Dept. of Statistics, Univ. of Illinois.
  • [2] A. Barron, M. Schervish and L. Wasserman, The Consistency of posterior distributions in nonparametric problems, Ann. Statist. 27 (1999), 536–561.
  • [3] P. Bickel and B. Kleijn, The semiparametric Bernstein-Von Mises theorem, Ann. Statist. 40 (2012), 206–237.
  • [4] L. Birgé, Approximation dans les espaces métriques et théorie de l’estimation, Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 65 (1983), 181–238.
  • [5] L. Birgé, Sur un théorème de minimax et son application aux tests, Probability and Mathematical Statistics 3 (1984), 259–282.
  • [6] P. De Blasi, A. Lijoi and I. Prünster, An asymptotic analysis of a class of discrete nonparametric priors, Statist. Sinica 23 (2013), 1299–1321.
  • [7] I. Castillo, On Bayesian supremum norm contraction rates, Ann. Statist. 42 (2014), 2058–2091.
  • [8] I. Castillo, J. Schmidt-Hieber, and A. van der Vaart, Bayesian linear regression with sparse priors, Ann. Statist. 43 (2015), 1986–2018.
  • [9] P. Diaconis and D. Freedman, On the consistency of Bayes estimates, Ann. Statist. 14 (1986), 1–26.
  • [10] J. Doob, Applications of the theory of martingales, Le calcul des Probabilités et ses Applications, Colloques Internationales du CNRS, Paris (1948), 22–28.
  • [11] D. Freedman, On the asymptotic behavior of Bayes estimates in the discrete case I, Ann. Math. Statist. 34 (1963), 1386–1403.
  • [12] D. Freedman, On the asymptotic behavior of Bayes estimates in the discrete case II, Ann. Math. Statist. 36 (1965), 454–456.
  • [13] S. Ghosal, J. Ghosh and R. Ramamoorthi, Non-informative priors via sieves and packing numbers, in “Advances in Statistical Decision Theory and Applications” (S. Panchapakesan and N. Balakrishnan, eds.), pp. 119–132, Birkhauser, Boston (1997).
  • [14] S. Ghosal, J. Ghosh and R. Ramamoorthi, Posterior consistency of Dirichlet mixtures in density estimation, Ann. Statist. 27 (1999), 143–158.
  • [15] S. Ghosal, J. Ghosh and A. van der Vaart, Convergence rates of posterior distributions, Ann. Statist. 28 (2000), 500–531.
  • [16] S. Ghosal, and A. van der Vaart, Fundamentals of Nonparametric Bayesian Inference, Cambridge University Press, Cambridge (2017).
  • [17] J. Ghosh and R. Ramamoorthi, Bayesian Nonparametrics, Springer, New York (2003).
  • [18] M. Hoffmann, J. Schmidt-Hieber, On adaptive posterior concentration rates, Ann. Statist. 43 (2015), 2259—2295.
  • [19] I. Ibragimov, R. Has’minskii, Statistical Estimation: Asymptotic Theory, Springer, New York (1981).
  • [20] B. Kleijn, Bayesian asymptotics under misspecification, PhD. Thesis, Free University Amsterdam (2004).
  • [21] B. Kleijn and A. van der Vaart, Misspecification in infinite-dimensional Bayesian statistics, Ann. Statist. 34 (2006), 837–877.
  • [22] B. Kleijn and A. van der Vaart, The Bernstein-Von-Mises theorem under misspecification, Electron. J. Statist. 6 (2012), 354–381.
  • [23] B. Kleijn and B. Knapik, Semiparametric posterior limits under local asymptotic exponentiality, 1210.6204.
  • [24] B. Kleijn, On the frequentist validity of Bayesian limits, 1611.08444.
  • [25] L. Le Cam, On the speed of convergence of posterior distributions, (unpublished preprint) University of California, Berkeley (197?).
  • [26] L. Le Cam, Convergence of estimates under dimensionality restrictions, Ann. Statist. 1 (1973), 38–55.
  • [27] L. Le Cam, On local and global properties in the theory of asymptotic normality of experiments, Stochastic Process. and Related Topics 1 (1975), 13–53. (ed. M.L. Puri), Academic Press, New York.
  • [28] L. Le Cam, Asymptotic Methods in Statistical Decision Theory, Springer, New York (1986).
  • [29] T. Leonard, Density estimation, stochastic processes and prior information, J. Roy. Statist. Soc. B40 (1978), 113–146.
  • [30] K. Matusita, Some propoerties of affinity and applications, Ann. Inst. Statist. Math. 23 (1971), 137–155.
  • [31] M. Reiss, J. Schmidt-Hieber, Nonparametric Bayesian analysis for support boundary recovery, 1703.08358.
  • [32] Y. Ritov, P. Bickel, A. Gamst and B. Kleijn, The Bayesian analysis of complex, high-dimensional models: can it be CODA? Statist. Sci. 29 (2014), 619–639.
  • [33] L. Schwartz, Consistency of Bayes procedures, PhD. thesis, UC Berkeley Statistics Department (1961).
  • [34] L. Schwartz, On Bayes procedures, Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 4 (1965), 10–26.
  • [35] X. Shen and L. Wasserman, Rates of convergence of posterior distributions, Ann. Statist. 29 (2001), 687–714.
  • [36] H. Strasser, Mathematical Theory of Statistics, de Gruyter, Berlin (1985).
  • [37] G. Toussiant, Some properties of Matusita’s measure of affinity of several distributions, Ann. Inst. Statist. Math. 26 (1974), 389–394.
  • [38] A. van der Vaart and J. Wellner, Weak Convergence and Empirical Processes, Springer, New York (1996).
  • [39] S. Walker, New approaches to Bayesian consistency, Ann. Statist. 32 (2004), 2028–2043.
  • [40] S. Walker, A. Lijoi and I. Prünster, On rates of convergence for posterior distributions in infinite-dimensional models, Ann. Statist. 35 (2007), 738–746.
  • [41] W.H. Wong and X. Shen, Probability inequalities for likelihood ratios and convergence rates of sieve MLEs, Ann. Statist. 23 (1995), 339–362.