Bayesian Analysis

Selection of Tuning Parameters, Solution Paths and Standard Errors for Bayesian Lassos

Vivekananda Roy and Sounak Chakraborty

Full-text: Open access

Enhanced PDF (721 KB)

Abstract

Penalized regression methods such as the lasso and elastic net (EN) have become popular for simultaneous variable selection and coefficient estimation. Implementation of these methods require selection of the penalty parameters. We propose an empirical Bayes (EB) methodology for selecting these tuning parameters as well as computation of the regularization path plots. The EB method does not suffer from the “double shrinkage problem” of frequentist EN. Also it avoids the difficulty of constructing an appropriate prior on the penalty parameters. The EB methodology is implemented by efficient importance sampling method based on multiple Gibbs sampler chains. Since the Markov chains underlying the Gibbs sampler are proved to be geometrically ergodic, Markov chain central limit theorem can be used to provide asymptotically valid confidence band for profiles of EN coefficients. The practical effectiveness of our method is illustrated by several simulation examples and two real life case studies. Although this article considers lasso and EN for brevity, the proposed EB method is general and can be used to select shrinkage parameters in other regularization methods.

Article information

Source
Bayesian Anal. Volume 12, Number 3 (2017), 753-778.

Dates
First available in Project Euclid: 7 September 2016

Permanent link to this document
http://projecteuclid.org/euclid.ba/1473276258

Digital Object Identifier
doi:10.1214/16-BA1025

Subjects
Primary: 62F15: Bayesian inference 62J07: Ridge regression; shrinkage estimators
Secondary: 60J05: Discrete-time Markov processes on general state spaces

Keywords
Bayesian lasso elastic net empirical Bayes geometric ergodicity importance sampling Markov chain Monte Carlo shrinkage standard errors

Rights
Creative Commons Attribution 4.0 International License.

Citation

Roy, Vivekananda; Chakraborty, Sounak. Selection of Tuning Parameters, Solution Paths and Standard Errors for Bayesian Lassos. Bayesian Anal. 12 (2017), no. 3, 753--778. doi:10.1214/16-BA1025. http://projecteuclid.org/euclid.ba/1473276258.


Export citation

References

  • Andrews, D. F. and Mallows, C. F. (1974). “Scale mixtures of normal distributions.” Journal of the Royal Statistical Society, Series B, 36: 99–102.
    Zentralblatt MATH: 0282.62017
  • Asmussen, S. and Glynn, P. W. (2011). “A new proof of convergence of MCMC via the ergodic theorem.” Statistics and Probability Letters, 81: 1482–1485.
    Zentralblatt MATH: 1232.65015
    Digital Object Identifier: doi:10.1016/j.spl.2011.05.004
  • Bondell, H. and Reich, B. (2008). “Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with Oscar.” Biometrics, 64: 115–123.
    Zentralblatt MATH: 1146.62051
    Digital Object Identifier: doi:10.1111/j.1541-0420.2007.00843.x
  • Buta, E. and Doss, H. (2011). “Computational approaches for empirical Bayes methods and Bayesian sensitivity analysis.” The Annals of Statistics, 39: 2658–2685.
    Mathematical Reviews (MathSciNet): MR2906882
    Zentralblatt MATH: 1231.62008
    Digital Object Identifier: doi:10.1214/11-AOS913
    Project Euclid: euclid.aos/1324563351
  • Carvalho, C., Polson, N., and Scott, J. (2010). “The horseshoe estimator for sparse signals.” Biometrika, 97: 465–480.
    Mathematical Reviews (MathSciNet): MR2650751
    Zentralblatt MATH: 05773446
    Digital Object Identifier: doi:10.1093/biomet/asq017
  • Doss, H. (2010). “Estimation of large families of Bayes factors from Markov chain output.” Statistica Sinica, 20: 537–560.
    Mathematical Reviews (MathSciNet): MR2682629
    Zentralblatt MATH: 1187.62052
  • Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). “Least angle regression.” The Annals of Statistics, 32: 407–499.
    Mathematical Reviews (MathSciNet): MR2060166
    Zentralblatt MATH: 1091.62054
    Digital Object Identifier: doi:10.1214/009053604000000067
    Project Euclid: euclid.aos/1083178935
  • Fan, J. and Li, R. (2001). “Variable selection via nonconcave penalized likelihood and its oracle property.” Journal of the American Statistical Association, 96: 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Zentralblatt MATH: 1073.62547
    Digital Object Identifier: doi:10.1198/016214501753382273
  • Flegal, J. M. and Jones, G. L. (2010). “Batch means and spectral variance estimators in Markov chain Monte Carlo.” The Annals of Statistics, 38: 1034–1070.
    Zentralblatt MATH: 1184.62161
    Digital Object Identifier: doi:10.1214/09-AOS735
    Project Euclid: euclid.aos/1266586622
  • Geyer, C. J. (1994). “Estimating normalizing constants and reweighting mixtures in Markov chain Monte Carlo.” Technical report 568, School of Statistics, University of Minnesota.
  • Geyer, C. J. (1996). Markov Chain Monte Carlo in Practice, chapter Estimation and optimization of functions, 241–258. Boca Raton, FL: Chapman and Hall/CRC Press.
  • Hoerl, A. E. and Kennard, R. W. (1970). “Ridge regression: Biased estimation for nonorthogonal problems.” Technometrics, 12: 55–67.
    Zentralblatt MATH: 0202.17205
    Digital Object Identifier: doi:10.1080/00401706.1970.10488634
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning with Applications in R. Springer, New York, USA.
    Zentralblatt MATH: 1281.62147
  • Johnson, V. E. and Rossell, D. (2012). “Bayesian model selection in high-dimensional settings.” Journal of the American Statistical Association, 107: 649–660.
    Mathematical Reviews (MathSciNet): MR2980074
    Zentralblatt MATH: 1261.62024
    Digital Object Identifier: doi:10.1080/01621459.2012.682536
  • Jones, G. L. and Hobert, J. P. (2001). “Honest exploration of intractable probability distributions via Markov chain Monte Carlo.” Statistical Science, 16: 312–34.
    Mathematical Reviews (MathSciNet): MR1888447
    Zentralblatt MATH: 1127.60309
    Digital Object Identifier: doi:10.1214/ss/1015346317
    Project Euclid: euclid.ss/1015346317
  • Kass, R. E. and Raftery, A. E. (1995). “Bayes factors.” Journal of the American Statistical Association, 90: 773–795.
    Mathematical Reviews (MathSciNet): MR3363402
    Zentralblatt MATH: 0846.62028
    Digital Object Identifier: doi:10.1080/01621459.1995.10476572
  • Khare, K. and Hobert, J. P. (2013). “Geometric ergodicity of Bayesian lasso.” Electronic Journal of Statistics, 7: 2150–2163.
    Mathematical Reviews (MathSciNet): MR3104915
    Zentralblatt MATH: 1349.60124
    Digital Object Identifier: doi:10.1214/13-EJS841
    Project Euclid: euclid.ejs/1378817879
  • Knight, K. and Fu, W. (2000). “Asymptotics for lasso-type estimators.” The Annals of Statistics, 28: 1356–1378.
    Mathematical Reviews (MathSciNet): MR1805787
    Zentralblatt MATH: 1105.62357
    Digital Object Identifier: doi:10.1214/aos/1015957397
    Project Euclid: euclid.aos/1015957397
  • Kyung, M., Gill, J., Ghosh, M., and Casella, G. (2010). “Penalized regression, standard errors, and Bayesian lassos.” Bayesian Analysis, 5: 369–412.
    Zentralblatt MATH: 1330.62289
    Digital Object Identifier: doi:10.1214/10-BA607
  • Lee, K. H., Chakraborty, S., and Sun, J. (2015). “Survival prediction and variable selection with simultaneous shrinkage and grouping priors.” Statistical Analysis and Data Mining, 8: 114–127.
  • Li, Q. and Lin, N. (2010). “The Bayesian elastic net.” Bayesian Analysis, 5: 151–170.
    Zentralblatt MATH: 1330.65026
    Digital Object Identifier: doi:10.1214/10-BA506
  • Liang, F., Paulo, R., Molina, G., Clyde, M. A., and Berger, J. O. (2008). “Mixtures of $g$-priors for Bayesian variable selection.” Journal of the American Statistical Association, 103: 410–423.
    Zentralblatt MATH: 1335.62026
    Digital Object Identifier: doi:10.1198/016214507000001337
  • Liu, F., Chakraborty, S., Li, F., Liu, Y., and Lozano, A. C. (2014). “Bayesian regularization via graph Laplacian.” Bayesian Analysis, 9: 449–474.
    Zentralblatt MATH: 1327.62152
    Digital Object Identifier: doi:10.1214/14-BA860
  • Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. London: Springer Verlag.
    Zentralblatt MATH: 0925.60001
  • Newton, M. and Raftery, A. (1994). “Approximate Bayesian inference with the weighted likelihood bootstrap (with discussion).” Journal of the Royal Statistical Society, Series B, 56: 3–48.
    Mathematical Reviews (MathSciNet): MR1257793
    Zentralblatt MATH: 0788.62026
  • Park, T. and Casella, G. (2008). “The Bayesian lasso.” Journal of the American Statistical Association, 103: 681–686.
    Zentralblatt MATH: 1330.62292
    Digital Object Identifier: doi:10.1198/016214508000000337
  • Polson, N. and Scott, J. (2010). “Shrink globally, act locally: sparse Bayesian regularization and prediction.” Bayesian Statistics, 9: 501–538.
  • Roberts, G. O. and Rosenthal, J. S. (1997). “Geometric ergodicity and hybrid Markov chains.” Electronic Communications in Probability, 2: 13–25.
    Zentralblatt MATH: 0890.60061
    Digital Object Identifier: doi:10.1214/ECP.v2-981
  • Rosenthal, J. S. (1995). “Minorization conditions and convergence rates for Markov chain Monte Carlo.” Journal of the American Statistical Association, 90: 558–566.
    Zentralblatt MATH: 0824.60077
    Digital Object Identifier: doi:10.1080/01621459.1995.10476548
  • Roy, V. (2014). “Efficient estimation of the link function parameter in a robust Bayesian binary regression model.” Computational Statistics and Data Analysis, 73: 87–102.
  • Roy, V. and Chakraborty, S. (2016). “Supplementary material for “Selection of tuning parameters, solution paths and standard errors for Bayesian lassos”.” Bayesian Analysis.
  • Roy, V., Evangelou, E., and Zhu, Z. (2016). “Efficient estimation and prediction for the Bayesian binary spatial model with flexible link functions.” Biometrics, 72: 289–298.
    Zentralblatt MATH: 06578779
  • Roy, V. and Hobert, J. P. (2007). “Convergence rates and asymptotic standard errors for MCMC algorithms for Bayesian probit regression.” Journal of the Royal Statistical Society, Series B, 69: 607–623.
  • Roy, V., Tan, A., and Flegal, J. (2015). “Estimating standard errors for importance sampling estimators with multiple Markov chains.” Technical report, Iowa State University.
  • Tibshirani, R. (1996). “Regression shrinkage and selection via the lasso.” Journal of the Royal Statistical Society, Series B, 58: 267–288.
    Zentralblatt MATH: 0850.62538
  • Wolpert, R. L. and Schmidler, S. C. (2012). “$\alpha$-stable limit laws for harmonic mean estimators of marginal likelihoods.” Statistica Sinica, 22: 1233–1251.
    Zentralblatt MATH: 06072105
  • Yuan, M. and Lin, Y. (2006). “Model selection and estimation in regression with grouped variables.” Journal of the Royal Statistical Society, Series B, 68: 49–67.
    Mathematical Reviews (MathSciNet): MR2212574
    Zentralblatt MATH: 1141.62030
    Digital Object Identifier: doi:10.1111/j.1467-9868.2005.00532.x
  • Zou, H. (2006). “The adaptive lasso and its oracle properties.” Journal of the American Statistical Association, 101: 1418–1429.
    Zentralblatt MATH: 1171.62326
    Digital Object Identifier: doi:10.1198/016214506000000735
  • Zou, H. and Hastie, T. (2005). “Regularization and variable selection via the elastic net.” Journal of the Royal Statistical Society, Series B, 67: 301–320.
    Mathematical Reviews (MathSciNet): MR2210692
    Zentralblatt MATH: 0962.83045

Supplemental materials

  • Supplementary Material: Supplementary Material for “Selection of Tuning Parameters, Solution Paths and Standard Errors for Bayesian Lassos”. The online supplementary materials contain the proofs of lemmas. Also a summary of the steps involved in the estimation of the tuning parameters and the solution paths is given in the supplementary materials.
    Digital Object Identifier: doi:10.1214/16-BA1025SUPP
    Supplemental PDF (151 KB)

International Society for Bayesian Analysis

Bayesian Analysis

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more