The Annals of Statistics

Criteria for Bayesian model choice with application to variable selection

Abstract

In objective Bayesian model selection, no single criterion has emerged as dominant in defining objective prior distributions. Indeed, many criteria have been separately proposed and utilized to propose differing prior choices. We first formalize the most general and compelling of the various criteria that have been suggested, together with a new criterion. We then illustrate the potential of these criteria in determining objective model selection priors by considering their application to the problem of variable selection in normal linear models. This results in a new model selection objective prior with a number of compelling properties.

Article information

Source
Ann. Statist. Volume 40, Number 3 (2012), 1550-1577.

Dates
First available in Project Euclid: 5 September 2012

http://projecteuclid.org/euclid.aos/1346850065

Digital Object Identifier
doi:10.1214/12-AOS1013

Mathematical Reviews number (MathSciNet)
MR3015035

Zentralblatt MATH identifier
1257.62023

Citation

Bayarri, M. J.; Berger, J. O.; Forte, A.; García-Donato, G. Criteria for Bayesian model choice with application to variable selection. Ann. Statist. 40 (2012), no. 3, 1550--1577. doi:10.1214/12-AOS1013. http://projecteuclid.org/euclid.aos/1346850065.

References

• Abramowitz, M. and Stegun, I. A. (1964). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Dover, New York.
• Bayarri, M. J. and García-Donato, G. (2008). Generalization of Jeffreys divergence-based priors for Bayesian hypothesis testing. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 981–1003.
• Berger, J. (1980). A robust generalized Bayes estimator and confidence region for a multivariate normal mean. Ann. Statist. 8 716–761.
• Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis, 2nd ed. Springer, New York.
• Berger, J. O., Bayarri, M. J. and Pericchi, L. R. (2012). The effective sample size. Econometric Reviews. To appear.
• Berger, J. O., Ghosh, J. K. and Mukhopadhyay, N. (2003). Approximations and consistency of Bayes factors as model dimension grows. J. Statist. Plann. Inference 112 241–258.
• Berger, J. O. and Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. J. Amer. Statist. Assoc. 91 109–122.
• Berger, J. O., Pericchi, L. R. and Varshavsky, J. A. (1998). Bayes factors and marginal distributions in invariant situations. Sankhyā Ser. A 60 307–321.
• Berger, J. O. and Pericchi, L. R. (2001). Objective Bayesian methods for model selection: Introduction and comparison. In Model Selection. Institute of Mathematical Statistics Lecture Notes—Monograph Series 38 135–207. IMS, Beachwood, OH.
• Casella, G., Girón, F. J., Martínez, M. L. and Moreno, E. (2009). Consistency of Bayesian procedures for variable selection. Ann. Statist. 37 1207–1228.
• Cui, W. and George, E. I. (2008). Empirical Bayes vs. fully Bayes variable selection. J. Statist. Plann. Inference 138 888–900.
• De Santis, F. and Spezzaferri, F. (1999). Methods for default and roubst Bayesian model comparison: The fractional Bayes factor approach. International Statistical Review 67 267–286.
• Fernández, C., Ley, E. and Steel, M. F. J. (2001). Benchmark priors for Bayesian model averaging. J. Econometrics 100 381–427.
• Forte, A. (2011). Objective Bayesian criteria for variable selection. Ph.D. thesis, Univ. de Valencia.
• Ghosh, J. K. and Samanta, T. (2002). Nonsubjective Bayes testing—an overview. J. Statist. Plann. Inference 103 205–223.
• Guo, R. and Speckman, P. L. (2009). Bayes factors consistency in linear models. Presented in O’Bayes 09 conference.
• Hsiao, C. K. (1997). Approximate Bayes factors when a mode occurs on the boundary. J. Amer. Statist. Assoc. 92 656–663.
• Jeffreys, H. (1961). Theory of Probability, 3rd ed. Clarendon, Oxford.
• Kass, R. E. and Raftery, A. E. (1995). Bayes factors. J. Amer. Statist. Assoc. 90 773–795.
• Kass, R. E. and Vaidyanathan, S. K. (1992). Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. J. Roy. Statist. Soc. Ser. B 54 129–144.
• Kass, R. E. and Wasserman, L. (1995). A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. J. Amer. Statist. Assoc. 90 928–934.
• Laud, P. W. and Ibrahim, J. G. (1995). Predictive model selection. J. Roy. Statist. Soc. Ser. B 57 247–262.
• Liang, F., Paulo, R., Molina, G., Clyde, M. A. and Berger, J. O. (2008). Mixtures of $g$ priors for Bayesian variable selection. J. Amer. Statist. Assoc. 103 410–423.
• Maruyama, Y. and George, E. I. (2008). gBF: A fully Bayes factor with a generalized $g$-prior. Available at arXiv:0801.4410v2 [stat.ME].
• Maruyama, Y. and Strawderman, W. E. (2010). Robust Bayesian variable selection with sub-harmonic priors. Available at arXiv:1009.1926v2 [stat.ME].
• Moreno, E., Bertolino, F. and Racugno, W. (1998). An intrinsic limiting procedure for model selection and hypotheses testing. J. Amer. Statist. Assoc. 93 1451–1460.
• Pérez, J. M. and Berger, J. O. (2002). Expected-posterior prior distributions for model selection. Biometrika 89 491–511.
• Robert, C. P., Chopin, N. and Rousseau, J. (2009). Harold Jeffreys’s theory of probability revisited. Statist. Sci. 24 141–172.
• Scott, J. G. and Berger, J. O. (2010). Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann. Statist. 38 2587–2619.
• Spiegelhalter, D. J. and Smith, A. F. M. (1982). Bayes factors for linear and log-linear models with vague prior information. J. Roy. Statist. Soc. Ser. B 44 377–387.
• Strawderman, W. E. (1971). Proper Bayes minimax estimators of the multivariate normal mean. Ann. Math. Statist. 42 385–388.
• Suzuki, Y. (1983). On Bayesian approach to model selection. In Proceedings of the International Statistical Institute 288–291. ISI Publications, Voorburg.
• Weisstein, E. W. (2009). Appell hypergeometric function from mathworld—a Wolfram web resource. Available at http://mathworld.wolfram.com/AppellHypergeometricFunction.html.
• Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with $g$-prior distributions. In Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti (A. Zellner, ed.) 389–399. North-Holland, Amsterdam.
• Zellner, A. and Siow, A. (1980). Posterior odds ratio for selected regression hypotheses. In Bayesian Statistics 1 (J. M. Bernardo, M. H. DeGroot, D. V. Lindley and A. F. M. Smith, eds.) 585–603. Univeristy Press, Valencia.
• Zellner, A. and Siow, A. (1984). Basic Issues in Econometrics. University of Chicago Press, Chicago.