Bayesian Analysis

Consistent Group Selection with Bayesian High Dimensional Modeling

Xinming Yang and Naveen N. Narisetty

Advance publication

This article is in its final form and can be cited using the date of online publication and the DOI.

Full-text: Open access

Abstract

In many applications with high dimensional covariates, the covariates are naturally structured into different groups which can be used to perform efficient statistical inference. We propose a Bayesian hierarchical model with a spike and slab prior specification to perform group selection in high dimensional linear regression models. While several penalization methods and more recently, some Bayesian approaches are proposed for group selection, theoretical properties of Bayesian approaches have not been studied extensively. In this paper, we provide novel theoretical results for group selection consistency under spike and slab priors which demonstrate that the proposed Bayesian approach has advantages compared to penalization approaches. Our theoretical results accommodate flexible conditions on the design matrix and can be applied to commonly used statistical models such as nonparametric additive models for which very limited theoretical results are available for the Bayesian methods. A shotgun stochastic search algorithm is adopted for the implementation of our proposed approach. We illustrate through simulation studies that the proposed method has better performance for group selection compared to a variety of existing methods.

Article information

Source
Bayesian Anal., Advance publication (2018), 27 pages.

Dates
First available in Project Euclid: 9 October 2019

Permanent link to this document
https://projecteuclid.org/euclid.ba/1570586977

Digital Object Identifier
doi:10.1214/19-BA1178

Keywords
group selection spike and slab priors Bayesian variable selection shotgun stochastic search

Rights
Creative Commons Attribution 4.0 International License.

Citation

Yang, Xinming; Narisetty, Naveen N. Consistent Group Selection with Bayesian High Dimensional Modeling. Bayesian Anal., advance publication, 9 October 2019. doi:10.1214/19-BA1178. https://projecteuclid.org/euclid.ba/1570586977


Export citation

References

  • Bach, F. R. (2008). “Consistency of the group Lasso and multiple kernel learning.” Journal of Machine Learning Research, 9(Jun): 1179–1225.
  • Bernardo, J. M. (1979). “Reference posterior distributions for Bayesian inference.” Journal of the Royal Statistical Society: Series B (Methodological), 41(2): 113–128.
  • Breheny, P. (2015). “The group exponential Lasso for bi-level variable selection.” Biometrics, 71(3): 731–740.
  • Breheny, P. and Huang, J. (2009). “Penalized methods for bi-level variable selection.” Statistics and Its Interface, 2(3): 369.
  • Fan, J. and Li, R. (2001). “Variable selection via nonconcave penalized likelihood and its oracle properties.” Journal of the American Statistical Association, 96(456): 1348–1360.
  • Fan, J. and Lv, J. (2008). “Sure independence screening for ultrahigh dimensional feature space.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5): 849–911.
  • Frank, L. E. and Friedman, J. H. (1993). “A statistical view of some chemometrics regression tools.” Technometrics, 35(2): 109–135.
  • Gan, L., Narisetty, N. N., and Liang, F. (2018). “Bayesian regularization for graphical models with unequal shrinkage.” Journal of the American Statistical Association, 1–14.
  • George, E. I. and Foster, D. P. (2000). “Calibration and empirical Bayes variable selection.” Biometrika, 87(4): 731–747.
  • George, E. I. and McCulloch, R. E. (1993). “Variable selection via Gibbs sampling.” Journal of the American Statistical Association, 88(423): 881–889.
  • George, E. I. and McCulloch, R. E. (1997). “Approaches for Bayesian variable selection.” Statistica Sinica, 339–373.
  • Greenlaw, K., Szefer, E., Graham, J., Lesperance, M., Nathoo, F. S., and Initiative, A. D. N. (2017). “A Bayesian group sparse multi-task regression model for imaging genetics.” Bioinformatics, 33(16): 2513–2522.
  • Hans, C., Dobra, A., and West, M. (2007). “Shotgun stochastic search for “large p” regression.” Journal of the American Statistical Association, 102(478): 507–516.
  • He, X., Wang, L., and Hong, H. G. (2013). “Quantile-adaptive model-free variable screening for high-dimensional heterogeneous data.” The Annals of Statistics, 41(1): 342–369.
  • Hernández-Lobato, D., Hernández-Lobato, J. M., and Dupont, P. (2013). “Generalized spike-and-slab priors for Bayesian group feature selection using expectation propagation.” The Journal of Machine Learning Research, 14(1): 1891–1945.
  • Huang, J., Breheny, P., and Ma, S. (2012). “A selective review of group selection in high-dimensional models.” Statistical Science, 27(4).
  • Huang, J., Horowitz, J. L., and Wei, F. (2010). “Variable selection in nonparametric additive models.” The Annals of Statistics, 38(4): 2282.
  • Huang, J., Ma, S., Xie, H., and Zhang, C.-H. (2009). “A group bridge approach for variable selection.” Biometrika, 96(2): 339–355.
  • Ishwaran, H. and Rao, J. S. (2005). “Spike and slab variable selection: frequentist and Bayesian strategies.” The Annals of Statistics, 730–773.
  • Jeffreys, H. (1961). Theory of Probability (3rd edition). Oxford University Press.
  • Johnson, V. E. and Rossell, D. (2012). “Bayesian model selection in high-dimensional settings.” Journal of the American Statistical Association, 107(498): 649–660.
  • Keller, M., Gatti, D., Schueler, K., Rabaglia, M., Stapleton, D., Simecek, P., Vincent, M., Allen, S., Broman, A., Bacher, R., Kendziorski, C., Broman, K., Yandell, B., Churchill, G., and Attie, A. (2018a). “Data from: Genetic drivers of pancreatic islet function.” URL https://doi.org/10.5061/dryad.pj105.
  • Keller, M. P., Gatti, D. M., Schueler, K. L., Rabaglia, M. E., Stapleton, D. S., Simecek, P., Vincent, M., Allen, S., Broman, A. T., Bacher, R., Kendziorski, C., Broman, K. W., Yandell, B. S., Churchill, G. A., and Attie, A. D. (2018b). “Genetic Drivers of Pancreatic Islet Function.” Genetics, 209(1): 335–356. URL http://www.genetics.org/content/209/1/335.
  • Kyung, M., Gill, J., Ghosh, M., and Casella, G. (2010). “Penalized regression, standard errors, and Bayesian Lassos.” Bayesian Analysis, 5(2): 369–411.
  • Liang, F., Liu, C., and Carroll, R. J. (2007). “Stochastic approximation in Monte Carlo computation.” Journal of the American Statistical Association, 102(477): 305–320.
  • Liang, F., Song, Q., and Yu, K. (2013). “Bayesian subset modeling for high-dimensional generalized linear models.” Journal of the American Statistical Association, 108(502): 589–606.
  • Liquet, B., Mengersen, K., Pettitt, A., and Sutton, M. (2017). “Bayesian variable selection regression of multivariate responses for group data.” Bayesian Analysis, 12(4): 1039–1067.
  • Lounici, K., Pontil, M., Van De Geer, S., and Tsybakov, A. B. (2011). “Oracle inequalities and optimal inference under group sparsity.” The Annals of Statistics, 39(4): 2164–2204.
  • Nardi, Y. and Rinaldo, A. (2008). “On the asymptotic properties of the group Lasso estimator for linear models.” Electronic Journal of Statistics, 2: 605–633.
  • Narisetty, N. N. and He, X. (2014). “Bayesian variable selection with shrinking and diffusing priors.” The Annals of Statistics, 42(2): 789–817.
  • Obozinski, G., Wainwright, M. J., and Jordan, M. I. (2011). “Support union recovery in high-dimensional multivariate regression.” The Annals of Statistics, 39(1): 1–47.
  • Park, T. and Casella, G. (2008). “The Bayesian Lasso.” Journal of the American Statistical Association, 103(482): 681–686.
  • Raman, S., Fuchs, T. J., Wild, P. J., Dahl, E., and Roth, V. (2009). “The Bayesian group-Lasso for analyzing contingency tables.” In Proceedings of the 26th Annual International Conference on Machine Learning, 881–888. ACM.
  • Ravikumar, P., Lafferty, J., Liu, H., and Wasserman, L. (2009). “Sparse additive models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(5): 1009–1030.
  • Ročková, V. and George, E. I. (2014). “EMVS: The EM approach to Bayesian variable selection.” Journal of the American Statistical Association, 109(506): 828–846.
  • Ročková, V. and George, E. I. (2016). “Fast Bayesian factor analysis via automatic rotations to sparsity.” Journal of the American Statistical Association, 111(516): 1608–1622.
  • Ročková, V. and George, E. I. (2018). “The spike-and-slab Lasso.” Journal of the American Statistical Association, 113(521): 431–444.
  • Scott, J. G. and Berger, J. O. (2010). “Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem.” The Annals of Statistics, 38(5): 2587–2619.
  • Shang, Z. and Li, P. (2014). “High-dimensional Bayesian inference in nonparametric additive models.” Electronic Journal of Statistics, 8(2): 2804–2847.
  • Simon, N., Friedman, J., Hastie, T., and Tibshirani, R. (2013). “A sparse-group Lasso.” Journal of Computational and Graphical Statistics, 22(2): 231–245.
  • Spitzner, D. J. (2019). “Subjective Bayesian testing using calibrated prior probabilities.” Brazilian Journal of Probability and Statistics, 33(4): 861–893.
  • Tibshirani, R. (1996). “Regression shrinkage and selection via the Lasso.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 267–288.
  • Wang, H. and Leng, C. (2008). “A note on adaptive group Lasso.” Computational Statistics & Data Analysis, 52(12): 5277–5286.
  • Wang, L., Li, H., and Huang, J. Z. (2008). “Variable selection in nonparametric varying-coefficient models for analysis of repeated measurements.” Journal of the American Statistical Association, 103(484): 1556–1569.
  • Wei, F. and Huang, J. (2010). “Consistent group selection in high-dimensional linear regression.” Bernoulli: official journal of the Bernoulli Society for Mathematical Statistics and Probability, 16(4): 1369.
  • Xu, X. and Ghosh, M. (2015). “Bayesian variable selection and estimation for group Lasso.” Bayesian Analysis, 10(4): 909–936.
  • Yang, X. and Narisetty, N. N. (2019). “Consistent Group Selection with Bayesian High Dimensional Modeling: Supplementary Material.” Bayesian Analysis.
  • Yuan, M. and Lin, Y. (2006). “Model selection and estimation in regression with grouped variables.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1): 49–67.
  • Zellner, A. (1962). “An efficient method of estimating seemingly unrelated regressions and tests for aggregation bias.” Journal of the American Statistical Association, 57(298): 348–368.
  • Zellner, A. (1986). “On Assessing Prior Distributions and Bayesian Regression Analysis with g-Prior Distributions.” Bayesian inference and decision techniques: Essays in honor of Bruno de Finetti, 233–243.
  • Zhang, C.-H. (2010). “Nearly unbiased variable selection under minimax concave penalty.” The Annals of Statistics, 38(2): 894–942.
  • Zhang, L., Baladandayuthapani, V., Mallick, B. K., Manyam, G. C., Thompson, P. A., Bondy, M. L., and Do, K.-A. (2014). “Bayesian hierarchical structured variable selection methods with application to molecular inversion probe studies in breast cancer.” Journal of the Royal Statistical Society: Series C (Applied Statistics), 63(4): 595–620.
  • Zhao, P. and Yu, B. (2006). “On model selection consistency of Lasso.” Journal of Machine Learning Research, 7(Nov): 2541–2563.
  • Zou, H. (2006). “The adaptive Lasso and its oracle properties.” Journal of the American Statistical Association, 101(476): 1418–1429.

Supplemental materials

  • Consistent Group Selection with Bayesian High Dimensional Modeling: Supplementary Material. Proofs of Theorem 2.1 and Theorem 3.1 and a Gibbs sampler for drawing samples from P(Z, beta, sigma2 | Y) are provided in the Supplementary Material.