Electronic Journal of Statistics

Geometric ergodicity of Gibbs samplers in Bayesian penalized regression models

Dootika Vats

Full-text: Open access

Abstract

We consider three Bayesian penalized regression models and show that the respective deterministic scan Gibbs samplers are geometrically ergodic regardless of the dimension of the regression problem. We prove geometric ergodicity of the Gibbs samplers for the Bayesian fused lasso, the Bayesian group lasso, and the Bayesian sparse group lasso. Geometric ergodicity along with a moment condition results in the existence of a Markov chain central limit theorem for Monte Carlo averages and ensures reliable output analysis. Our results of geometric ergodicity allow us to also provide default starting values for the Gibbs samplers.

Article information

Source
Electron. J. Statist., Volume 11, Number 2 (2017), 4033-4064.

Dates
Received: September 2016
First available in Project Euclid: 19 October 2017

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1508378637

Digital Object Identifier
doi:10.1214/17-EJS1351

Mathematical Reviews number (MathSciNet)
MR3714307

Zentralblatt MATH identifier
1374.60127

Subjects
Primary: 60J05: Discrete-time Markov processes on general state spaces
Secondary: 62F15: Bayesian inference

Keywords
Markov chains geometric ergodicity Bayesian lassos starting values

Rights
Creative Commons Attribution 4.0 International License.

Citation

Vats, Dootika. Geometric ergodicity of Gibbs samplers in Bayesian penalized regression models. Electron. J. Statist. 11 (2017), no. 2, 4033--4064. doi:10.1214/17-EJS1351. https://projecteuclid.org/euclid.ejs/1508378637


Export citation

References

  • [1] Andelić, M. and Da Fonseca, C. (2011). Sufficient conditions for positive definiteness of tridiagonal matrices revisited., Positivity 15 155–159.
  • [2] Bhattacharya, A., Chakraborty, A. and Mallick, B. K. (2016). Fast sampling with Gaussian scale mixture priors in high-dimensional regression., Biometrika 103 985–991.
  • [3] Doss, H. and Hobert, J. P. (2010). Estimation of Bayes factors in a class of hierarchical random effects models using a geometrically ergodic MCMC algorithm., Journal of Computational and Graphical Statistics 19 295–312.
  • [4] Doss, C. R., Flegal, J. M., Jones, G. L. and Neath, R. C. (2014). Markov Chain Monte Carlo estimation of quantiles., Electronic Journal of Statistics 8 2448–2478.
  • [5] Fan, Y., Wang, X. and Peng, Q. (2017). Inference of gene regulatory networks using Bayesian nonparametric regression and topology information., Computational and Mathematical Methods in Medicine 2017.
  • [6] Flegal, J. M. and Gong, L. (2015). Relative fixed-width stopping rules for Markov chain Monte Carlo simulations., Statistica Sinica 25 655–676.
  • [7] Flegal, J. M. and Jones, G. L. (2010). Batch means and spectral variance estimators in Markov Chain Monte Carlo., The Annals of Statistics 38 1034–1070.
  • [8] Gong, L. and Flegal, J. M. (2016). A practical sequential stopping rule for high-dimensional Markov chain Monte Carlo., Journal of Computational and Graphical Statistics 684–700.
  • [9] Griffin, J. E. and Brown, P. J. (2010). Inference with normal-gamma prior distributions in regression problems., Bayesian Analysis 5 171–188.
  • [10] Gu, X., Yin, G. and Lee, J. J. (2013). Bayesian two-step lasso strategy for biomarker selection in personalized medicine development for time-to-event endpoints., Contemporary clinical trials 36 642–650.
  • [11] Guan, Y. and Stephens, M. (2011). Bayesian variable selection regression for genome-wide association studies and other large-scale problems., The Annals of Applied Statistics 1780–1815.
  • [12] Hobert, J. P., Jones, G. L., Presnell, B. and Rosenthal, J. S. (2002). On the applicability of regenerative simulation in Markov chain Monte Carlo., Biometrika 89 731–743.
  • [13] Ishwaran, H. and Rao, J. S. (2005). Spike and slab variable selection: frequentist and Bayesian strategies., Annals of Statistics 730–773.
  • [14] Johnson, A. A. and Jones, G. L. (2015). Geometric ergodicity of random scan Gibbs samplers for hierarchical one-way random effects models., Journal of Multivariate Analysis 140 325–342.
  • [15] Jones, G. L. and Hobert, J. P. (2001). Honest exploration of intractable probability distributions via Markov chain Monte Carlo., Statistical Science 16 312–334.
  • [16] Jones, G. L. and Hobert, J. P. (2004). Sufficient burn-in for Gibbs samplers for a hierarchical random effects model., The Annals of Statistics 32 784–817.
  • [17] Jones, G. L., Haran, M., Caffo, B. S. and Neath, R. (2006). Fixed-width output analysis for Markov chain Monte Carlo., Journal of the American Statistical Association 101 1537–1547.
  • [18] Khare, K. and Hobert, J. P. (2012). Geometric ergodicity of the Gibbs sampler for Bayesian quantile regression., Journal of Multivariate Analysis 112 108–116.
  • [19] Khare, K. and Hobert, J. P. (2013). Geometric ergodicity of the Bayesian lasso., Electronic Journal of Statistics 7 2150–2163.
  • [20] Kyung, M., Gill, J., Ghosh, M. and Casella, G. (2010). Penalized regression, standard errors, and Bayesian lassos., Bayesian Analysis 5 369–411.
  • [21] Liu, J. S., Wong, W. H. and Kong, A. (1994). Covariance structure of the Gibbs sampler with applications to the comparisons of estimators and augmentation schemes., Biometrika 81 27–40.
  • [22] Meyn, S. P. and Tweedie, R. L. (2009)., Markov Chains and Stochastic Stability. Cambridge University Press.
  • [23] Nathoo, F. S., Greenlaw, K. and Lesperance, M. (2016). Regularization parameter selection for a Bayesian multi-level group lasso regression model with application to imaging genomics., arXiv preprint arXiv:1603.08163.
  • [24] Pal, S. and Khare, K. (2014). Geometric ergodicity for Bayesian shrinkage models., Electronic Journal of Statistics 8 604–645.
  • [25] Park, T. and Casella, G. (2008). The Bayesian lasso., Journal of the American Statistical Association 103 681–686.
  • [26] Rajaratnam, B. and Sparks, D. (2015). MCMC-based inference in the era of big data: A fundamental analysis of the convergence complexity of high-dimensional chains., arXiv preprint arXiv:1508.00947.
  • [27] Raman, S., Fuchs, T. J., Wild, P. J., Dahl, E., Buhmann, J. M. and Roth, V. (2010). Infinite mixture-of-experts model for sparse survival regression with application to breast cancer., BMC bioinformatics 11 1.
  • [28] Roberts, G. O. and Rosenthal, J. S. (1997). Geometric ergodicity and hybrid Markov chains., Electronic Communications in Probability 2 13–25.
  • [29] Roberts, G. O. and Rosenthal, J. S. (2004). General state space Markov chains and MCMC algorithms., Probability Surveys 1 20–71.
  • [30] Rosenthal, J. S. (1995). Minorization conditions and convergence rates for Markov chain Monte Carlo., Journal of the American Statistical Association 90 558–566.
  • [31] Roy, V. and Chakraborty, S. (2017). Selection of tuning parameters, solution paths and standard errors for Bayesian lassos., Bayesian Analysis 12 753–778.
  • [32] Simon, N., Friedman, J., Hastie, T. and Tibshirani, R. (2013). A sparse-group lasso., Journal of Computational and Graphical Statistics 22 231–245.
  • [33] Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005). Sparsity and smoothness via the fused lasso., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67 91–108.
  • [34] Vats, D., Flegal, J. M. and Jones, G. L. (2015a). Multivariate output analysis for Markov chain Monte Carlo., arXiv preprint arXiv:1512.07713.
  • [35] Vats, D., Flegal, J. M. and Jones, G. L. (2015b). Strong consistency of multivariate spectral variance estimators in Markov chain Monte Carlo., Bernoulli (to appear).
  • [36] Xu, X. and Ghosh, M. (2015). Bayesian variable selection and estimation for group lasso., Bayesian Analysis 10 909–936.
  • [37] Yang, Y., Wainwright, M. J. and Jordan, M. I. (2016). On the computational complexity of high-dimensional Bayesian variable selection., The Annals of Statistics 44 2497–2532.
  • [38] Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 68 49–67.
  • [39] Zhang, L., Baladandayuthapani, V., Mallick, B. K., Manyam, G. C., Thompson, P. A., Bondy, M. L. and Do, K.-A. (2014). Bayesian hierarchical structured variable selection methods with application to molecular inversion probe studies in breast cancer., Journal of the Royal Statistical Society: Series C (Applied Statistics) 63 595–620.