Bernoulli
- Bernoulli
- Volume 23, Number 3 (2017), 1822-1847.
Empirical Bayes posterior concentration in sparse high-dimensional linear models
Ryan Martin, Raymond Mess, and Stephen G. Walker
Full-text: Open access
Abstract
We propose a new empirical Bayes approach for inference in the $p\gg n$ normal linear model. The novelty is the use of data in the prior in two ways, for centering and regularization. Under suitable sparsity assumptions, we establish a variety of concentration rate results for the empirical Bayes posterior distribution, relevant for both estimation and model selection. Computation is straightforward and fast, and simulation results demonstrate the strong finite-sample performance of the empirical Bayes model selection procedure.
Article information
Source
Bernoulli, Volume 23, Number 3 (2017), 1822-1847.
Dates
Received: September 2015
Revised: December 2015
First available in Project Euclid: 17 March 2017
Permanent link to this document
https://projecteuclid.org/euclid.bj/1489737626
Digital Object Identifier
doi:10.3150/15-BEJ797
Mathematical Reviews number (MathSciNet)
MR3624879
Zentralblatt MATH identifier
06714320
Keywords
data-dependent prior fractional likelihood minimax regression variable selection
Citation
Martin, Ryan; Mess, Raymond; Walker, Stephen G. Empirical Bayes posterior concentration in sparse high-dimensional linear models. Bernoulli 23 (2017), no. 3, 1822--1847. doi:10.3150/15-BEJ797. https://projecteuclid.org/euclid.bj/1489737626
References
- [1] Abramovich, F. and Grinshtein, V. (2010). MAP model selection in Gaussian regression. Electron. J. Stat. 4 932–949.
- [2] Arias-Castro, E. and Lounici, K. (2014). Estimation and variable selection with exponential weights. Electron. J. Stat. 8 328–354.Mathematical Reviews (MathSciNet): MR3195119
Zentralblatt MATH: 1294.62164
Digital Object Identifier: doi:10.1214/14-EJS883
Project Euclid: euclid.ejs/1397826704 - [3] Barbieri, M.M. and Berger, J.O. (2004). Optimal predictive model selection. Ann. Statist. 32 870–897.
- [4] Barron, A.R. and Cover, T.M. (1991). Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 1034–1054.
- [5] Bondell, H.D. and Reich, B.J. (2012). Consistent high-dimensional Bayesian variable selection via penalized credible regions. J. Amer. Statist. Assoc. 107 1610–1624.
- [6] Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Series in Statistics. Heidelberg: Springer.
- [7] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351.
- [8] Castillo, I., Schmidt-Hieber, J. and van der Vaart, A. (2015). Bayesian linear regression with sparse priors. Ann. Statist. 43 1986–2018.
- [9] Castillo, I. and van der Vaart, A. (2012). Needles and straw in a haystack: Posterior concentration for possibly sparse sequences. Ann. Statist. 40 2069–2101.
- [10] Chen, J. and Chen, Z. (2008). Extended Bayesian information criteria for model selection with large model spaces. Biometrika 95 759–771.Zentralblatt MATH: 05609546
- [11] Clyde, M. and George, E.I. (2004). Model uncertainty. Statist. Sci. 19 81–94.
- [12] Dalalyan, A.S. and Tsybakov, A.B. (2008). Aggregation by exponential weighting, sharp PAC-Bayesian bounds, and sparsity. Mach. Learn. 72 39–61.
- [13] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
- [14] Fan, J. and Lv, J. (2010). A selective overview of variable selection in high dimensional feature space. Statist. Sinica 20 101–148.Zentralblatt MATH: 1180.62080
- [15] Gao, C., van der Vaart, A.W. and Zhou, H.H. (2015). A general framework for Bayes structured linear models. Unpublished manuscript. Available at arXiv:1506.02174.arXiv: 1506.02174
- [16] George, E.I. and McCullogh, R.E. (1993). Variable selection via Gibbs sampling. J. Amer. Statist. Assoc. 88 881–889.
- [17] Ghosal, S., Ghosh, J.K. and van der Vaart, A.W. (2000). Convergence rates of posterior distributions. Ann. Statist. 28 500–531.Mathematical Reviews (MathSciNet): MR1790007
Zentralblatt MATH: 1105.62315
Digital Object Identifier: doi:10.1214/aos/1016218228
Project Euclid: euclid.aos/1016218228 - [18] Grünwald, P. and van Ommen, T. (2014). Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Unpublished manuscript. Available at arXiv:1412.3730.arXiv: 1412.3730
- [19] Heaton, M.J. and Scott, J.G. (2010). Bayesian Computation and the Linear Model. In Frontiers of Statistical Decision Making and Bayesian Analysis (M.-H. Cheh, D. Dey, P. Müller, D. Sun and K. Ye, eds.) 527–545. New York: Springer.
- [20] Ishwaran, H. and Rao, J.S. (2005). Spike and slab gene selection for multigroup microarray data. J. Amer. Statist. Assoc. 100 764–780.
- [21] Ishwaran, H. and Rao, J.S. (2005). Spike and slab variable selection: Frequentist and Bayesian strategies. Ann. Statist. 33 730–773.
- [22] James, G.M. and Radchenko, P. (2009). A generalized Dantzig selector with shrinkage tuning. Biometrika 96 323–337.Mathematical Reviews (MathSciNet): MR2507146
Zentralblatt MATH: 1163.62054
Digital Object Identifier: doi:10.1093/biomet/asp013 - [23] James, G.M., Radchenko, P. and Lv, J. (2009). DASSO: Connections between the Dantzig selector and lasso. J. R. Stat. Soc. Ser. B. Stat. Methodol. 71 127–142.
- [24] Jiang, W. (2007). Bayesian variable selection for high dimensional generalized linear models: Convergence rates of the fitted densities. Ann. Statist. 35 1487–1511.
- [25] Jiang, W. and Tanner, M.A. (2008). Gibbs posterior for variable selection in high-dimensional classification and data mining. Ann. Statist. 36 2207–2231.
- [26] Johnson, V.E. and Rossell, D. (2012). Bayesian model selection in high-dimensional settings. J. Amer. Statist. Assoc. 107 649–660.
- [27] Martin, R. and Walker, S.G. (2014). Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector. Electron. J. Stat. 8 2188–2206.
- [28] Narisetty, N.N. and He, X. (2014). Bayesian variable selection with shrinking and diffusing priors. Ann. Statist. 42 789–817.
- [29] Park, T. and Casella, G. (2008). The Bayesian lasso. J. Amer. Statist. Assoc. 103 681–686.
- [30] Reid, S., Tibshirani, R. and Friedman, J. (2014). A study of error variance estimation in lasso regression. Unpublished manuscript. Available at arXiv:1311.5274.arXiv: 1311.5274
- [31] Rigollet, P. and Tsybakov, A. (2011). Exponential screening and optimal rates of sparse estimation. Ann. Statist. 39 731–771.
- [32] Rigollet, P. and Tsybakov, A.B. (2012). Sparse estimation by exponential weighting. Statist. Sci. 27 558–575.
- [33] Shen, X. and Wasserman, L. (2001). Rates of convergence of posterior distributions. Ann. Statist. 29 687–714.
- [34] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B. Stat. Methodol. 58 267–288.Zentralblatt MATH: 0850.62538
- [35] Walker, S. and Hjort, N.L. (2001). On Bayesian consistency. J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 811–821.
- [36] Walker, S.G., Lijoi, A. and Prünster, I. (2007). On rates of convergence for posterior distributions in infinite-dimensional models. Ann. Statist. 35 738–746.
- [37] Wellcome Trust Case Control Consortium (2007). Genome-wide association study of 14,000 cases of seven common diseases and 3000 shared controls. Nature 447 661–678.
- [38] Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with $g$-prior distributions. In Bayesian Inference and Decision Techniques. Stud. Bayesian Econometrics Statist. 6 233–243. North-Holland, Amsterdam.Zentralblatt MATH: 0655.62071
- [39] Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the LASSO selection in high-dimensional linear regression. Ann. Statist. 36 1567–1594.
- [40] Zhang, T. (2006). From $\varepsilon $-entropy to KL-entropy: Analysis of minimum information complexity density estimation. Ann. Statist. 34 2180–2210.
- [41] Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418–1429.
- [42] Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 301–320.
- You have access to this content.
- You have partial access to this content.
- You do not have access to this content.
More like this
- Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector
Martin, Ryan and Walker, Stephen G., Electronic Journal of Statistics, 2014 - Posterior contraction in sparse Bayesian factor models for massive covariance matrices
Pati, Debdeep, Bhattacharya, Anirban, Pillai, Natesh S., and Dunson, David, Annals of Statistics, 2014 - Asymptotic Optimality of One-Group Shrinkage Priors in Sparse High-dimensional Problems
Ghosh, Prasenjit and Chakrabarti, Arijit, Bayesian Analysis, 2017
- Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector
Martin, Ryan and Walker, Stephen G., Electronic Journal of Statistics, 2014 - Posterior contraction in sparse Bayesian factor models for massive covariance matrices
Pati, Debdeep, Bhattacharya, Anirban, Pillai, Natesh S., and Dunson, David, Annals of Statistics, 2014 - Asymptotic Optimality of One-Group Shrinkage Priors in Sparse High-dimensional Problems
Ghosh, Prasenjit and Chakrabarti, Arijit, Bayesian Analysis, 2017 - Bayesian linear regression with sparse priors
Castillo, Ismaël, Schmidt-Hieber, Johannes, and van der Vaart, Aad, Annals of Statistics, 2015 - Projective inference in high-dimensional problems: Prediction and feature selection
Piironen, Juho, Paasiniemi, Markus, and Vehtari, Aki, Electronic Journal of Statistics, 2020 - Empirical Bayes selection of wavelet thresholds
Johnstone, Iain M. and Silverman, Bernard W., Annals of Statistics, 2005 - Gene network reconstruction using global-local shrinkage priors
Leday, Gwenaël G. R., de Gunst, Mathisca C. M., Kpogbezan, Gino B., van der Vaart, Aad W., van Wieringen, Wessel N., and van de Wiel, Mark A., Annals of Applied Statistics, 2017 - Adaptive posterior contraction rates for the horseshoe
van der Pas, Stéphanie, Szabó, Botond, and van der Vaart, Aad, Electronic Journal of Statistics, 2017 - Posterior graph selection and estimation consistency for high-dimensional Bayesian DAG models
Cao, Xuan, Khare, Kshitij, and Ghosh, Malay, Annals of Statistics, 2019 - Bayesian variable selection for globally sparse probabilistic PCA
Bouveyron, Charles, Latouche, Pierre, and Mattei, Pierre-Alexandre, Electronic Journal of Statistics, 2018