Electronic Journal of Statistics

Geometric ergodicity of the Bayesian lasso

Abstract

Consider the standard linear model $\mathbf{y}=X\boldsymbol{\beta}+\sigma\epsilon$, where the components of $\epsilon$ are iid standard normal errors. Park and Casella [14] consider a Bayesian treatment of this model with a Laplace/Inverse-Gamma prior on $(\beta,\sigma)$. They introduce a Data Augmentation approach that can be used to explore the resulting intractable posterior density, and call it the Bayesian lasso algorithm. In this paper, the Markov chain underlying the Bayesian lasso algorithm is shown to be geometrically ergodic, for arbitrary values of the sample size $n$ and the number of variables $p$. This is important, as geometric ergodicity provides theoretical justification for the use of Markov chain CLT, which can then be used to obtain asymptotic standard errors for Markov chain based estimates of posterior quantities. Kyung et al. [12] provide a proof of geometric ergodicity for the restricted case $n\geq p$, but as we explain in this paper, their proof is incorrect. Our approach is different and more direct, and enables us to establish geometric ergodicity for arbitrary $n$ and $p$.

Article information

Source
Electron. J. Statist., Volume 7 (2013), 2150-2163.

Dates
First available in Project Euclid: 10 September 2013

https://projecteuclid.org/euclid.ejs/1378817879

Digital Object Identifier
doi:10.1214/13-EJS841

Mathematical Reviews number (MathSciNet)
MR3104915

Zentralblatt MATH identifier
1349.60124

Subjects
Primary: 60J27: Continuous-time Markov processes on discrete state spaces
Secondary: 62F15: Bayesian inference

Citation

Khare, Kshitij; Hobert, James P. Geometric ergodicity of the Bayesian lasso. Electron. J. Statist. 7 (2013), 2150--2163. doi:10.1214/13-EJS841. https://projecteuclid.org/euclid.ejs/1378817879

References

• [1] Asmussen, S. and Glynn, P.W. (2011). A new proof of convergence of MCMC via the ergodic theorem, Statistics & Probability Letters 81, 1482–1485.
• [2] Bednorz, W. and Latuszynski, K. (2007). A few remarks on “Fixed-Width Output Analysis for Markov Chain Monte Carlo” by Jones et al., JASA 102, 1485–1486.
• [3] Bae, K. and Mallick, B.K. (2004). Gene selection using a two-level hierarchical Bayesian model, Bioinformatics 20, 3423–3430.
• [4] Bhattacharya, A., Pati, D., Pillai, N.S., and Dunson, D.B. (2012). Bayesian shrinkage, arXiv preprint.
• [5] Carvalho, C.M., Polson, N.G., and Scott, J.G. (2009). Handling sparsity via the horseshoe, Journal of Machine Learning Research W & CP 5, 73–80.
• [6] Carvalho, C.M., Polson, N.G., and Scott, J.G. (2010). The horseshoe estimator for sparse signals, Biometrika 97, 465–480.
• [7] Figueiredo, M.A.T. (2003). Adaptive sparseness for supervised learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 1150–1159.
• [8] Flegal, J.M., Haran, M., and Jones, G.M. (2008). Markov chain Monte Carlo: Can we trust the third significant figure?, Statistical Science 23, 250–260.
• [9] Flegal, J.M. and Jones, G.L. (2010). Batch means and spectral variance estimators in Markov chain Monte Carlo, Annals of Statistics 38, 1034–1070.
• [10] Griffin, J.E. and Brown, P.J. (2010). Inference with normal-gamma prior distributions in regression problems, Bayesian Analysis 5, 171–188.
• [11] Jones, G.L., Haran, M., Caffo, B.S., and Neath, R. (2006). Fixed-width output analysis for Markov chain Monte Carlo, Journal of the American Statistical Association 101, 1537–1547.
• [12] Kyung, M., Gill, J., Ghosh, M., and Casella, G. (2010). Penalized regression, standard errors and Bayesian lassos, Bayesian Analysis 5, 369–412.
• [13] Meyn, S.P. and Tweedie, R.L. (1993)., Markov Chains and Stochastic Stability, Springer-Verlag, London.
• [14] Park, T. and Casella, G. (2008). The Bayesian lasso, Journal of the American Statistical Association 103, 681–686.
• [15] Polson, N.G. and Scott, J.G. (2010). Shrink globally, act locally: Sparse Bayesian regularization and prediction, Bayesian Statistics 9, J.M. Bernardo, M.J. Bayarri, J.O. Berger, A.P. Dawid, D. Heckerman, A.F.M. Smith, and M. West, eds., Oxford University Press, New York.
• [16] Roberts, G.O. and Rosenthal, J.S. (1998). Markov chain Monte Carlo: Some practical implications of theoretical results (with discussion), Canadian Journal of Statistics 26, 5–31.
• [17] Roberts, G.O. and Rosenthal, J.S. (2004). General state space Markov chains and MCMC algorithms, Probability Surveys, 20–71.
• [18] Rosenthal, J.S. (1995). Minorization conditions and convergence rates for Markov chain Monte Carlo, JASA 90, 558–566.
• [19] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society B 58, 267–288.
• [20] Yuan, M., and Lin, Y. (2005). Efficient empirical Bayes variable selection and esimation in linear models, Journal of the American Statistical Association 100, 1215–1225.