The Annals of Statistics

Consistency of Bayesian procedures for variable selection

George Casella, F. Javier Girón, M. Lina Martínez, and Elías Moreno

Full-text: Open access

Abstract

It has long been known that for the comparison of pairwise nested models, a decision based on the Bayes factor produces a consistent model selector (in the frequentist sense). Here we go beyond the usual consistency for nested pairwise models, and show that for a wide class of prior distributions, including intrinsic priors, the corresponding Bayesian procedure for variable selection in normal regression is consistent in the entire class of normal linear models. We find that the asymptotics of the Bayes factors for intrinsic priors are equivalent to those of the Schwarz (BIC) criterion. Also, recall that the Jeffreys–Lindley paradox refers to the well-known fact that a point null hypothesis on the normal mean parameter is always accepted when the variance of the conjugate prior goes to infinity. This implies that some limiting forms of proper prior distributions are not necessarily suitable for testing problems. Intrinsic priors are limits of proper prior distributions, and for finite sample sizes they have been proved to behave extremely well for variable selection in regression; a consequence of our results is that for intrinsic priors Lindley’s paradox does not arise.

Article information

Source
Ann. Statist., Volume 37, Number 3 (2009), 1207-1228.

Dates
First available in Project Euclid: 10 April 2009

Permanent link to this document
https://projecteuclid.org/euclid.aos/1239369020

Digital Object Identifier
doi:10.1214/08-AOS606

Mathematical Reviews number (MathSciNet)
MR2509072

Zentralblatt MATH identifier
1160.62004

Subjects
Primary: 62F05: Asymptotic properties of tests
Secondary: 62J15: Paired and multiple comparisons

Keywords
Bayes factors intrinsic priors linear models consistency

Citation

Casella, George; Girón, F. Javier; Martínez, M. Lina; Moreno, Elías. Consistency of Bayesian procedures for variable selection. Ann. Statist. 37 (2009), no. 3, 1207--1228. doi:10.1214/08-AOS606. https://projecteuclid.org/euclid.aos/1239369020


Export citation

References

  • Abramowitz, M. and Stegun, I. A. (1972). Handbook of Mathematical Functions. U.S. Government Printing Office, Washington, DC.
  • Berger, J. O. and Bernardo, J. M. (1992). On the development of the reference prior method. In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. David and A. F. M. Smith, eds.) 35–60. Oxford Univ. Press, London.
  • Berger, J. O. and Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. J. Amer. Statist. Assoc. 91 109–122.
  • Berger, J. O. and Pericchi, L. R. (2004). Training samples in objective Bayesian model selection. Ann. Statist. 32 841–869.
  • Casella, G. and Moreno, E. (2006). Objective Bayesian variable selection. J. Amer. Statist. Assoc. 101 157–167.
  • Dawid, A. P. (1992). Prequential analysis, stochastic complexity and Bayesian inference. In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. David and A. F. M. Smith, eds.) 109–125. Oxford Univ. Press, London.
  • Diaconis, P. and Friedman, D. (1986). On the consistency of Bayes estimates (with discussion). Ann. Statist. 14 1–67.
  • Girón, F. J., Martínez, M. L., Moreno, E. and Torres, F. (2006). Objective testing procedures in linear models. Calibration of the p-values. Scandinavian J. Statist. 33 765–784.
  • Girón, F. J., Moreno, E. and Casella, G. (2007). Objective Bayesian analysis of multiple changepoints for linear models (with discussion). In Bayesian Statistics 8 (J. M. Bernardo, M. J. Bayarri and J. O. Berger, eds.) 227–252. Oxford Univ. Press, London.
  • Girón, F. J., Moreno, E. and Martínez, M. L. (2006). An objective Bayesian procedure for variable selection in regression. In Advances on Distribution Theory, Order Statistics and Inference (N. Balakrishnan et al., eds.) 393–408. Birkhäuser, Boston.
  • Jeffreys, H. (1967). Theory of Probability. Oxford Univ. Press, London.
  • Kass, R. E. and Wasserman, L. (1995). A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. J. Amer. Statist. Assoc. 90 928–934.
  • Lindley, D. V. (1957). A statistical paradox. Biometrika 44 187–192.
  • Little, R. J. (2006). Calibrated Bayes: A Bayes/frequentist roadmap. Amer. Statist. 60 213–223.
  • Moreno, E. and Girón, F. J. (2005). Consistency of Bayes factors for linear models. C. R. Acad. Sci. Paris Ser. I Math. 340 911–914.
  • Moreno, E. and Girón, F. J. (2008). Comparison of Bayesian objective procedures for variable selection in linear regression. Test 3 472–492.
  • Moreno, E., Bertolino, F. and Racugno, W. (1998). An intrinsic limiting procedure for model selection and hypothesis testing. J. Amer. Statist. Assoc. 93 1451–1460.
  • O’Hagan, A. and Forster, J. (2004). Kendall’s Advanced Theory of Statistics: Vol. 2B: Bayesian Inference. Arnold, London.
  • Robert, C. P. (1993). A note on Jeffreys–Lindley paradox. Statist. Sinica 3 601–608.
  • Schwarz, G. (1978). Estimating the dimension of a model. Ann. Statist. 6 461–464.
  • Shao, J. (1997). An asymptotic theory for linear model selection. Statist. Sinica 7 221–264.