The Annals of Statistics

Asymptotic equivalence of empirical likelihood and Bayesian MAP

Marian Grendár and George Judge

Full-text: Open access

Abstract

In this paper we are interested in empirical likelihood (EL) as a method of estimation, and we address the following two problems: (1) selecting among various empirical discrepancies in an EL framework and (2) demonstrating that EL has a well-defined probabilistic interpretation that would justify its use in a Bayesian context. Using the large deviations approach, a Bayesian law of large numbers is developed that implies that EL and the Bayesian maximum a posteriori probability (MAP) estimators are consistent under misspecification and that EL can be viewed as an asymptotic form of MAP. Estimators based on other empirical discrepancies are, in general, inconsistent under misspecification.

Article information

Source
Ann. Statist. Volume 37, Number 5A (2009), 2445-2457.

Dates
First available in Project Euclid: 15 July 2009

Permanent link to this document
https://projecteuclid.org/euclid.aos/1247663761

Digital Object Identifier
doi:10.1214/08-AOS645

Mathematical Reviews number (MathSciNet)
MR2543698

Zentralblatt MATH identifier
1173.62014

Subjects
Primary: 62G05: Estimation 62C10: Bayesian problems; characterization of Bayes procedures
Secondary: 60F10: Large deviations

Keywords
Maximum nonparametric likelihood estimating equations Bayesian nonparametric consistency Bayesian large deviations L-divergence Pólya sampling right censoring Kaplan–Meier estimator

Citation

Grendár, Marian; Judge, George. Asymptotic equivalence of empirical likelihood and Bayesian MAP. Ann. Statist. 37 (2009), no. 5A, 2445--2457. doi:10.1214/08-AOS645. https://projecteuclid.org/euclid.aos/1247663761.


Export citation

References

  • [1] Ben-Tal, A., Brown, D. E. and Smith, R. L. (1987). Posterior convergence under incomplete information. Technical Report 87-23, Univ. Michigan, Ann Arbor.
  • [2] Ben-Tal, A., Brown, D. E. and Smith, R. L. (1988). Relative entropy and the convergence of the posterior and empirical distributions under incomplete and conflicting information. Technical Report 88-12, Univ. Michigan.
  • [3] Brown, B. M. and Chen, S. X. (1998). Combined and least squares empirical likelihood. Ann. Inst. Statist. Math. 90 443–450.
  • [4] Cressie, N. and Read, T. (1984). Multinomial goodness-of-fit tests. J. Roy. Statist. Soc. Ser. B 46 440–464.
  • [5] Csiszár, I. (1998). The method of types. IEEE Trans. Inform. Theory 44 2505–2523.
  • [6] Csiszár, I. and Shields, P. (2004). Notes on information theory and statistics: A tutorial. Found. Trends Comm. Inform. Theory 1 1–111.
  • [7] Dembo, A. and Zeitouni, O. (1998). Large Deviations Techniques and Applications, 2nd ed. Springer, New York.
  • [8] Florens, J.-P. and Rolin, J.-M. (1994). Bayes, bootstrap, moments. Discussion Paper 94.13, Institute de Statistique, Université catholique de Louvain.
  • [9] Freedman, D. A. (1963). On the asymptotic behavior of Bayes’ estimates in the discrete case. Ann. Math. Statist. 34 1386–1403.
  • [10] Freedman, D. A. (1999). On the Bernstein–von Mises theorem with infinite-dimensional parameters. Ann. Statist. 27 1119–1140.
  • [11] Ganesh, A. and O’Connell, N. (1999). An inverse of Sanov’s Theorem. Statist. Probab. Lett. 42 201–206.
  • [12] Ghosal, A., Ghosh, J. K. and Ramamoorthi, R. V. (1999). Consistency issues in Bayesian nonanparametrics. In Asymptotics, Nonparametrics and Time Series: A Tribute to Madan Lal Puri 639–667. Dekker.
  • [13] Ghosh, J. K. and Ramamoorthi, R. V. (2003). Bayesian Nonparametrics. Springer, New York.
  • [14] Godambe, V. P. and Kale, B. K. (1991). Estimating functions: An overview. In Estimating Functions (V. P. Godambe, ed.) 3–20. Oxford Univ. Press, New York.
  • [15] Grendár. M. (2005). Conditioning by rare sources. Acta Univ. M. Belii Ser. Math. 12 19–29.
  • [16] Grendár, M. and Judge, G. (2008). Large deviations theory and empirical estimator choice. Econometric Rev. 27 513–525.
  • [17] Hjort, N. L., Mckeague, I. W. and Van Keilegom, I. (2007). Extending the scope of empirical likelihood. Ann. Statist. To appear.
  • [18] Imbens, G., Spady, R. and Johnson, P. (1998). Information theoretic approaches to inference in moment condition models. Econometrica 66 333–357.
  • [19] Judge, G. G. and Mittelhammer, R. C. (2007). Estimation and inference in the case of competing sets of estimating equations. J. Econometrics 138 513–531.
  • [20] Kečkić, J. D. and Vasić, P. M. (1971). Some inequalities for the gamma function. Publ. Inst. Math. (Beograd) (N.S.) 11 107–114.
  • [21] Kerridge, D. F. (1961). Inaccuracy and inference. J. Roy. Statist. Soc. Ser. B 23 284–294.
  • [22] Kitamura, Y. and Stutzer, M. (1997). An information-theoretic alternative to generalized method of moments estimation. Econometrica 65 861–874.
  • [23] Kitamura, Y. and Stutzer, M. (2002). Connections between entropic and linear projections in asset pricing estimation. J. Econometrics 107 159–174.
  • [24] Kleijn, B. J. K. and van der Vaart, A. W. (2006). Misspecification in infinite-dimensional Bayesian statistics. Ann. Statist. 34 837–877.
  • [25] Kulhavý, R. (1996). Recursive Nonlinear Estimation: A Geometric Approach. Lecture Notes in Control and Information Sciences 216. Springer, London.
  • [26] Lazar, N. (2003). Bayesian empirical likelihood. Biometrika 90 319–326.
  • [27] Mittelhammer, R., Judge, G. and Miller, D. (2000). Econometric Foundations. Cambridge Univ. Press, Cambridge.
  • [28] Monahan, J. F. and Boos, D. D. (1992). Proper likelihoods for Bayesian analysis. Biometrika 79 271–278.
  • [29] Niven, R. K. (2005). Combinatorial information theory: I. Philosophical basis of cross-entropy and entropy. Available at arXiv:cond-mat/0512017.
  • [30] Owen, A. (1988). Empirical likelihood ratio confidence interval for a single functional. Biometrika 75 237–249.
  • [31] Owen, A. (2001). Empirical Likelihood. Chapman & Hall/CRC Press, New York.
  • [32] Qin, J. and Lawless, J. (1994). Empirical likelihood and general estimating equations. Ann. Statist. 22 300–325.
  • [33] Ragusa, G. (2006). Bayesian likelihoods for moment condition models. Working paper, Univ. California, Irvine.
  • [34] Sanov, I. N. (1957). On the probability of large deviations of random variables. Sb. Math. 42 11–44. (In Russian.)
  • [35] Schennach, S. (2005). Bayesian exponentially tilted empirical likelihood. Biometrika 92 31–46.
  • [36] Schwartz, L. (1965). On Bayes procedures. Z. Wahrsch. Verw. Gebiete 4 10–26.
  • [37] Susarla, V. and Van Ryzin, J. (1976). Nonparametric Bayesian estimation of survival curves from incomplete observations. J. Amer. Statist. Assoc. 71 897–902.
  • [38] Walker, S. (2004). New approaches to Bayesian consistency. Ann. Statist. 32 2028–2043.
  • [39] Walker, S. and Damien, P. (2000). Practical Bayesian asymptotics. Working Paper 00-007, Business School, Univ. Michigan.
  • [40] Walker, S., Lijoi, A. and Prünster, I. (2004). Contributions to the understanding of Bayesian consistency. Discussion Paper 13/2004, ICER, Applied Mathematics Series.
  • [41] Wang, D. and Chen, S. X. (2007). Empirical likelihood for estimating equations with missing values. Ann. Statist. To appear.