This paper presents a method for fast Bayesian variable selection in the normal linear regression model with high dimensional data. A novel approach is adopted in which an explicit posterior probability for including a covariate is obtained. The method is sequential but not order dependent, one deals with each covariate one by one, and a spike and slab prior is only assigned to the coefficient under investigation. We adopt the well-known spike and slab Gaussian priors with a sample size dependent variance, which achieves strong selection consistency for marginal posterior probabilities even when the number of covariates grows almost exponentially with sample size. Numerical illustrations are presented where it is shown that the new approach provides essentially equivalent results to the standard spike and slab priors, i.e. the same marginal posterior probabilities of the coefficients being nonzero, which are estimated via Gibbs sampling. Hence, we obtain the same results via the direct calculation of $p$ probabilities, compared to a stochastic search over a space of $2^{p}$ elements. Our procedure only requires $p$ probabilities to be calculated, which can be done exactly, hence parallel computation when $p$ is large is feasible.
Electron. J. Statist.
13(1):
284-309
(2019).
DOI: 10.1214/18-EJS1529
[1] Barbieri, M. M., Berger, J. O. et al. (2004). Optimal predictive model selection., Annals of Statistics 32 870–897. 1092.62033 10.1214/009053604000000238 euclid.aos/1085408489[1] Barbieri, M. M., Berger, J. O. et al. (2004). Optimal predictive model selection., Annals of Statistics 32 870–897. 1092.62033 10.1214/009053604000000238 euclid.aos/1085408489
[2] Bondell, H. D. and Reich, B. J. (2012). Consistent high-dimensional Bayesian variable selection via penalized credible regions., Journal of the American Statistical Association 107 1610–1624. 1258.62026 10.1080/01621459.2012.716344[2] Bondell, H. D. and Reich, B. J. (2012). Consistent high-dimensional Bayesian variable selection via penalized credible regions., Journal of the American Statistical Association 107 1610–1624. 1258.62026 10.1080/01621459.2012.716344
[3] Brown, P. J., Vannucci, M. and Fearn, T. (1998). Multivariate Bayesian variable selection and prediction., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60 627–641. 0909.62022 10.1111/1467-9868.00144[3] Brown, P. J., Vannucci, M. and Fearn, T. (1998). Multivariate Bayesian variable selection and prediction., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60 627–641. 0909.62022 10.1111/1467-9868.00144
[4] Candes, E., Tao, T. et al. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$., Annals of Statistics 35 2313–2351. 1139.62019 10.1214/009053606000001523 euclid.aos/1201012958[4] Candes, E., Tao, T. et al. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$., Annals of Statistics 35 2313–2351. 1139.62019 10.1214/009053606000001523 euclid.aos/1201012958
[5] Casella, G. and Moreno, E. (2006). Objective Bayesian variable selection., Journal of the American Statistical Association 101 157–167. 1118.62313 10.1198/016214505000000646[5] Casella, G. and Moreno, E. (2006). Objective Bayesian variable selection., Journal of the American Statistical Association 101 157–167. 1118.62313 10.1198/016214505000000646
[6] Castillo, I., Schmidt-Hieber, J., van der Vaart, A. et al. (2015). Bayesian linear regression with sparse priors., Annals of Statistics 43 1986–2018. 06502640 10.1214/15-AOS1334 euclid.aos/1438606851[6] Castillo, I., Schmidt-Hieber, J., van der Vaart, A. et al. (2015). Bayesian linear regression with sparse priors., Annals of Statistics 43 1986–2018. 06502640 10.1214/15-AOS1334 euclid.aos/1438606851
[7] Clyde, M. and George, E. I. (2004). Model uncertainty., Statistical Science 81–94. 1062.62044 10.1214/088342304000000035 euclid.ss/1089808274[7] Clyde, M. and George, E. I. (2004). Model uncertainty., Statistical Science 81–94. 1062.62044 10.1214/088342304000000035 euclid.ss/1089808274
[8] Fan, J., Feng, Y. and Song, R. (2011). Nonparametric independence screening in sparse ultra-high-dimensional additive models., Journal of the American Statistical Association 106 544–557. 1232.62064 10.1198/jasa.2011.tm09779[8] Fan, J., Feng, Y. and Song, R. (2011). Nonparametric independence screening in sparse ultra-high-dimensional additive models., Journal of the American Statistical Association 106 544–557. 1232.62064 10.1198/jasa.2011.tm09779
[9] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association 96 1348–1360. 1073.62547 10.1198/016214501753382273[9] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association 96 1348–1360. 1073.62547 10.1198/016214501753382273
[10] Fan, J. and Lv, J. (2010). A selective overview of variable selection in high dimensional feature space., Statistica Sinica 20 101. 1180.62080[10] Fan, J. and Lv, J. (2010). A selective overview of variable selection in high dimensional feature space., Statistica Sinica 20 101. 1180.62080
[11] George, E. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection., Biometrika 87 731–747. 1029.62008 10.1093/biomet/87.4.731[11] George, E. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection., Biometrika 87 731–747. 1029.62008 10.1093/biomet/87.4.731
[13] George, E. I. and McCulloch, R. E. (1997). Approaches for Bayesian variable selection., Statistica sinica 339–373. 0884.62031[13] George, E. I. and McCulloch, R. E. (1997). Approaches for Bayesian variable selection., Statistica sinica 339–373. 0884.62031
[14] Ishwaran, H. and Rao, J. S. (2005). Spike and slab variable selection: Frequentist and Bayesian strategies., Annals of Statistics 730–773. 1068.62079 10.1214/009053604000001147 euclid.aos/1117114335[14] Ishwaran, H. and Rao, J. S. (2005). Spike and slab variable selection: Frequentist and Bayesian strategies., Annals of Statistics 730–773. 1068.62079 10.1214/009053604000001147 euclid.aos/1117114335
[15] Johnson, V. E. and Rossell, D. (2012). Bayesian model selection in high-dimensional settings., Journal of the American Statistical Association 107 649–660. 1261.62024 10.1080/01621459.2012.682536[15] Johnson, V. E. and Rossell, D. (2012). Bayesian model selection in high-dimensional settings., Journal of the American Statistical Association 107 649–660. 1261.62024 10.1080/01621459.2012.682536
[16] Liang, F., Song, Q. and Yu, K. (2013). Bayesian subset modeling for high-dimensional generalized linear models., Journal of the American Statistical Association 108 589–606. 06195963 10.1080/01621459.2012.761942[16] Liang, F., Song, Q. and Yu, K. (2013). Bayesian subset modeling for high-dimensional generalized linear models., Journal of the American Statistical Association 108 589–606. 06195963 10.1080/01621459.2012.761942
[17] Mitchell, T. J. and Beauchamp, J. J. (1988). Bayesian variable selection in linear regression., Journal of the American Statistical Association 83 1023–1032. 0673.62051 10.1080/01621459.1988.10478694[17] Mitchell, T. J. and Beauchamp, J. J. (1988). Bayesian variable selection in linear regression., Journal of the American Statistical Association 83 1023–1032. 0673.62051 10.1080/01621459.1988.10478694
[18] Narisetty, N. N., He, X. et al. (2014). Bayesian variable selection with shrinking and diffusing priors., Annals of Statistics 42 789–817. 1302.62158 10.1214/14-AOS1207 euclid.aos/1400592178[18] Narisetty, N. N., He, X. et al. (2014). Bayesian variable selection with shrinking and diffusing priors., Annals of Statistics 42 789–817. 1302.62158 10.1214/14-AOS1207 euclid.aos/1400592178
[19] O’Hara, R. B., Sillanpää, M. J. et al. (2009). A review of Bayesian variable selection methods: what, how and which., Bayesian Analysis 4 85–117.[19] O’Hara, R. B., Sillanpää, M. J. et al. (2009). A review of Bayesian variable selection methods: what, how and which., Bayesian Analysis 4 85–117.
[21] Song, R., Yi, F. and Zou, H. (2014). On varying-coefficient independence screening for high-dimensional varying-coefficient models., Statistica Sinica 24 1735. 06497322[21] Song, R., Yi, F. and Zou, H. (2014). On varying-coefficient independence screening for high-dimensional varying-coefficient models., Statistica Sinica 24 1735. 06497322
[22] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso., Journal of the Royal Statistical Society. Series B (Methodological) 267–288. 0850.62538 10.1111/j.2517-6161.1996.tb02080.x[22] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso., Journal of the Royal Statistical Society. Series B (Methodological) 267–288. 0850.62538 10.1111/j.2517-6161.1996.tb02080.x
[23] Zou, H. (2006). The adaptive lasso and its oracle properties., Journal of the American Statistical Association 101 1418–1429. 1171.62326 10.1198/016214506000000735[23] Zou, H. (2006). The adaptive lasso and its oracle properties., Journal of the American Statistical Association 101 1418–1429. 1171.62326 10.1198/016214506000000735