Bayesian Analysis

A review of Bayesian variable selection methods: what, how and which

R. B. O'Hara and M. J. Sillanpää

Full-text: Open access

Abstract

The selection of variables in regression problems has occupied the minds of many statisticians. Several Bayesian variable selection methods have been developed, and we concentrate on the following methods: Kuo & Mallick, Gibbs Variable Selection (GVS), Stochastic Search Variable Selection (SSVS), adaptive shrinkage with Jeffreys' prior or a Laplacian prior, and reversible jump MCMC. We review these methods, in the context of their different properties. We then implement the methods in BUGS, using both real and simulated data as examples, and investigate how the different methods perform in practice. Our results suggest that SSVS, reversible jump MCMC and adaptive shrinkage methods can all work well, but the choice of which method is better will depend on the priors that are used, and also on how they are implemented.

Article information

Source
Bayesian Anal. Volume 4, Number 1 (2009), 85-117.

Dates
First available in Project Euclid: 22 June 2012

Permanent link to this document
https://projecteuclid.org/euclid.ba/1340370391

Digital Object Identifier
doi:10.1214/09-BA403

Mathematical Reviews number (MathSciNet)
MR2486240

Zentralblatt MATH identifier
1330.62291

Keywords
Variable Selection MCMC BUGS

Citation

O'Hara, R. B.; Sillanpää, M. J. A review of Bayesian variable selection methods: what, how and which. Bayesian Anal. 4 (2009), no. 1, 85--117. doi:10.1214/09-BA403. https://projecteuclid.org/euclid.ba/1340370391.


Export citation

References

  • Ball, R. D. (2001). “Bayesian methods for quantitative trait loci based on model selection: approximate analysis using Bayesian information criteria.” Genetics, 159: 1351–1364.
  • Broman, K. W. and Speed, T. P. (2002). “A model selection approach for identification of quantitative trait loci in experimental crosses.” J. Roy. Stat. Soc. B., 64: 641–656.
  • Brown, P. J., Vannucci, M., and Fearn, T. (1998). “Multivariate Bayesian variable selection and prediction.” J. Roy. Stat. Soc. B., 60: 627–641.
  • Burnham, K. and Anderson, D. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. New York, U.S.A.: Springer, 2nd edition.
  • Carlin, B. R. and Chib, S. (1995). “Bayesian model choice via Markov Chain Monte Carlo methods.” J. Roy. Stat. Soc. B., 57: 473–484.
  • Dellaportas, P., Forster, J. J., and Ntzoufras, I. (2000). “Bayesian variable selection using the Gibbs sampling.” In Dey, D. K., Ghosh, S. K., and Mallick, B. K. (eds.), Generalized linear models: a Bayesian perspective, 273–286. Marcel Dekker, Inc., New York.
  • Dellaportas, P., J.J.Forster, and Ntzoufras, I. (1997). “On Bayesian model and variable selection using MCMC. Technical report.” Technical report, Department of Statistics, Athens University of Economics and Business, Athens Greece.
  • Figueiredo, M. A. T. (2003). “Adaptive sparseness for supervised learning.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 25: 1150–1159.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper).” Bayesian Analysis, 1: 515–534.
  • George, E. I. and McCulloch, R. E. (1993). “Variable selection via Gibbs sampling.” J. Am. Stat. Assoc., 85: 398–409.
  • — (1997). “Approaches for Bayesian variable selection.” Statistica Sinica, 7: 339–373.
  • Geweke, J. (1996). “Variable selection and model comparison in regression.” In Bernando, J. M., Berger, J. O., Dawid, A. P., and Smith, A. F. M. (eds.), Bayesian Statistics 5, 609–620. Oxford University Press, Oxford, UK.
  • Geyer, C. J. (1992). “Practical Markov chain Monte Carlo.” Statist. Sci., 7: 473–483.
  • Godsill, S. J. (2001). “On the relationship between MCMC model uncertainty methods.” J. Comput. Graph. Stat., 10: 230–248.
  • Göring, H. H. H., Terwilliger, J. D., and Blangero, J. (2001). “Large upward bias in estimation of locus-specific effects from genomewide scans.” Am. J. Hum. Genet., 69: 1357–1369.
  • Green, P. J. (1995). “Reversible jump Markov chain Monte Carlo computation and Bayesian model determination.” Biometrika, 82: 711–732.
  • Hopert, J. P. and Casella, G. (1996). “The effect of improper priors on Gibbs sampling in hierarchical mixed models.” J. Am. Stat. Assoc., 91: 1461–1473.
  • Hoti, F. and Sillanpää, M. J. (2006). “Bayesian mapping of genotype x expression interactions in quantitative and qualitative traits.” Heredity, 97: 4–18.
  • Iswaran, H. and Rao, J. S. (2005). “Spike and slab variable selection: frequentist and Bayesian strategies.” The Annals of Statistics, 33: 730–773.
  • Kass, R. E. and Raftery, A. E. (1995). “Bayes factors.” J. Am. Stat. Assoc., 90: 773–795.
  • Kilpikari, R. and Sillanpää, M. J. (2003). “Bayesian analysis of multilocus association in quantitative and qualitative traits.” Genet. Epidemiol., 25: 122–135.
  • Knapp, S., Bridges, W., and Birkes, D. (1990). “Mapping quantitative trait loci using molecular marker linkage maps.” Theor. Appl. Genet., 79: 583–592.
  • Kuo, L. and Mallick, B. (1998). “Variable selection for regression models.” Sankhya Ser. B, 60: 65–81.
  • Lande, R. and Thompson, R. (1990). “Efficiency of marker-assisted selection in the improvement of quantitative traits.” Genetics, 124: 743–756.
  • Lunn, D. J., Whittaker, J. C., and Best, N. (2006). “A Bayesian toolkit for genetic association studies.” Genet. Epidemiol., 30: 231–247.
  • McDonald, G. C. and Schwing, R. C. (1973). “Instabilities of regression estimates relating air pollution to mortality.” Technometrics, 15: 463–482.
  • Meuwissen, T. H. E. and Goddard, M. E. (2004). “Mapping multiple QTL using linkage disequilibrium and linkage analysis information and multitrait data.” Genet. Sel. Evol., 36: 261–279.
  • Meuwissen, T. H. E., Hayes, B. J., and Goddard, M. E. (2001). “Prediction of total genetic value using genome-wide dense marker map.” Genetics, 157: 1819–1829.
  • Miller, A. (2002). Subset Selection in Regression. Boca Raton, Florida, U.S.A.: Chap- man & Hall/CRC.
  • Park, T. and Casella, G. (2008). “The Bayesian Lasso.” J. Am. Stat. Assoc., 103: 681–686.
  • Plummer, M., Best, N., Cowles, K., and Vines, K. (2008). coda: Output analysis and diagnostics for MCMC. R package version 0.13-3.
  • Robert, C. and Casella, G. (2004). Monte Carlo Statistical Methods. New York: Springer-Verlag, 2nd edition.
  • Sen, S. and Churchill, G. A. (2001). “A statistical framework for quantitative trait mapping.” Genetics, 159: 371–387.
  • Sillanpää, M., Gasbarra, D., and Arjas, E. (2004). “Comment on “On the Metropolis-Hastings acceptance probability to add or drop a quantitative trait locus in Markov Chain Monte Carlo-based Bayesian analyses”.” Genetics, 167: 1037.
  • Sillanpää, M. J. and Arjas, E. (1998). “Bayesian mapping of multiple quantitative trait loci from incomplete inbred line cross data.” Genetics, 148: 1373–1388.
  • Sillanpää, M. J. and Bhattacharjee, M. (2005). “Bayesian association-based fine mapping in small chromosomal segments.” Genetics, 169: 427–439.
  • — (2006). “Association mapping of complex trait loci with context-dependent effects and unknown context-variable.” Genetics, 174: 1597–1611.
  • Sillanpää, M. J. and Corander, J. (2002). “Model choice in gene mapping: what and why.” Trends Genet., 18: 301–307.
  • Smith, M. and Kohn, R. (2002). “Parsimonious covariance matrix estimation for longitudinal data.” J. Am. Stat. Assoc., 97: 1141–1153.
  • Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and der Linde, A. V. (2002). “Bayesian measures of model complexity and fit (with discussion).” J. R. Stat. Soc. B., 64: 583–616.
  • Sturtz, S., Ligges, U., and Gelman, A. (2005). “R2WinBUGS: A Package for Running WinBUGS from R.” Journal of Statistical Software, 12: 1–16.
  • ter Braak, C. J. F., Boer, M. P., and Bink, M. C. A. M. (2005). “Extending Xu’s Bayesian model for estimating polygenic effects using markers of the entire genome.” Genetics, 170: 1435–1438.
  • Thomas, A., O’Hara, B., Ligges, U., and Sturtz, S. (2006). “Making BUGS Open.” R News, 9: 12–17.
  • Tibshirani, R. (1996). “Regression shrinkage and selection via the lasso.” J. R. Stat. Soc. B., 58: 267–288.
  • Tinker, N. A., Mather, D. E., Rossnagel, B. G., Kasha, K. J., and Keinhofs, A. (1996). “Regions of the genome that affect agronomic performance in two-row barley.” Crop. Sci., 36: 1053–1062.
  • Tipping, M. E. (2004). “Bayesian inference: an introduction to principles and practice in machine learning.” In Bousquet, O., von Luxburg, U., and Rätsch, G. (eds.), Advanced Lectures on Machine Learning, 41–46. Springer.
  • Uimari, P. and Hoeschele, I. (1997). “Mapping-linked quantitative trait loci using Bayesian analysis and Markov chain Monte Carlo algoritm.” Genetics, 146: 735–743.
  • Xu, S. (2003). “Estimating polygenic effects using markers of the entire genome.” Genetics, 163: 789–801.
  • Yi, N. (2004). “A unified Markov chain Monte Carlo framework for mapping multiple quantitative trait loci.” Genetics, 167: 967–975.
  • Yi, N., George, V., and Allison, D. B. (2003). “Stochastic search variable selection for identifying multiple quantitative trait loci.” Genetics, 164: 1129–1138.
  • Yi, N. and Xu, S. (2008). “Bayesian LASSO for quantitative trait loci mapping.” Genetics, 179: 1045–1055.
  • Zhang, Y.-M. and Xu, S. (2005). “A penalized maximum likelihood method for estimating epistatic effects of QTL.” Heredity, 95: 96–104.

Supplemental materials