Bayesian Analysis

Uncertainty Quantification for the Horseshoe

Stéphanie van der Pas, Botond Szabó, and Aad van der Vaart

Advance publication

This article is in its final form and can be cited using the date of online publication and the DOI.

Full-text: Open access

Abstract

We investigate the credible sets and marginal credible intervals resulting from the horseshoe prior in the sparse multivariate normal means model. We do so in an adaptive setting without assuming knowledge of the sparsity level (number of signals). We consider both the hierarchical Bayes method of putting a prior on the unknown sparsity level and the empirical Bayes method with the sparsity level estimated by maximum marginal likelihood. We show that credible balls and marginal credible intervals have good frequentist coverage and optimal size if the sparsity level of the prior is set correctly. By general theory honest confidence sets cannot adapt in size to an unknown sparsity level. Accordingly the hierarchical and empirical Bayes credible sets based on the horseshoe prior are not honest over the full parameter space. We show that this is due to over-shrinkage for certain parameters and characterise the set of parameters for which credible balls and marginal credible intervals do give correct uncertainty quantification. In particular we show that the fraction of false discoveries by the marginal Bayesian procedure is controlled by a correct choice of cut-off.

Article information

Source
Bayesian Anal. (2017), 29 pages.

Dates
First available in Project Euclid: 1 September 2017

Permanent link to this document
https://projecteuclid.org/euclid.ba/1504231319

Digital Object Identifier
doi:10.1214/17-BA1065

Subjects
Primary: 62G15: Tolerance and confidence regions
Secondary: 62F15: Bayesian inference

Keywords
credible sets horseshoe sparsity nearly black vectors normal means problem frequentist Bayes

Rights
Creative Commons Attribution 4.0 International License.

Citation

van der Pas, Stéphanie; Szabó, Botond; van der Vaart, Aad. Uncertainty Quantification for the Horseshoe. Bayesian Anal., advance publication, 1 September 2017. doi: 10.1214/17-BA1065. https://projecteuclid.org/euclid.ba/1504231319


Export citation

References

  • Armagan, A., Dunson, D. B., and Lee, J. (2013). “Generalized Double Pareto Shrinkage.” Statistica Sinica, 23: 119–143.
  • Belitser, E. (2017). “On coverage and local radial rates of credible sets.” Annals of Statistics, 45(3): 1124–1151.
  • Belitser, E. and Nurushev, N. (2015). “Needles and straw in a haystack: empirical Bayes confidence for possibly sparse sequences.” ArXiv e-prints.
  • Bhadra, A., Datta, J., Polson, N. G., and Willard, B. (2017). “The Horseshoe+ Estimator of Ultra-Sparse Signals.” Advance publication. http://dx.doi.org/10.1214/16-BA1028
  • Bhattacharya, A., Chakraborty, A., and Mallick, B. K. (2015). “Fast sampling with Gaussian scale-mixture priors in high-dimensional regression.” ArXiv e-prints.
  • Bhattacharya, A., Pati, D., Pillai, N. S., and Dunson, D. B. (2014). “Dirichlet-Laplace Priors for Optimal Shrinkage.” ArXiv:1401.5398.
  • Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data. Springer-Verlag Berlin Heidelberg.
  • Bull, A. (2012). “Honest adaptive confidence bands and self-similar functions.” Electronic Journal of Statistics, 6: 1490–1516. http://projecteuclid.org/euclid.ejs/1346421602
  • Caron, F. and Doucet, A. (2008). “Sparse Bayesian Nonparametric Regression.” In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, 88–95. New York, NY, USA: ACM.
  • Carvalho, C. M., Polson, N. G., and Scott, J. G. (2009). “Handling Sparsity via the Horseshoe.” Journal of Machine Learning Research, W&CP, 5: 73–80.
  • Carvalho, C. M., Polson, N. G., and Scott, J. G. (2010). “The Horseshoe Estimator for Sparse Signals.” Biometrika, 97(2): 465–480.
  • Castillo, I. and Nickl, R. (2014). “On the Bernstein von Mises phenomenon for nonparametric Bayes procedures.” Annals of Statistics, 42(5): 1941–1969.
  • Castillo, I., Schmidt-Hieber, J., and van der Vaart, A. (2015). “Bayesian linear regression with sparse priors.” Annals of Statistics, 43(5): 1986–2018.
  • Castillo, I. and Van der Vaart, A. W. (2012). “Needles and Straw in a Haystack: Posterior Concentration for Possibly Sparse Sequences.” Annals of Statistics, 40(4): 2069–2101.
  • Datta, J. and Ghosh, J. K. (2013). “Asymptotic Properties of Bayes Risk for the Horseshoe Prior.” Bayesian Analysis, 8(1): 111–132.
  • Ghosh, P. and Chakrabarti, A. (2015). “Posterior Concentration Properties of a General Class of Shrinkage Estimators around Nearly Black Vectors.” ArXiv:1412.8161v2.
  • Giné, E. and Nickl, R. (2010). “Confidence bands in density estimation.” Annals of Statistics, 38(2): 1122–1170.
  • Gramacy, R. B. (2014). monomvn: Estimation for multivariate normal and Student-t data with monotone missingness. R package version 1.9-5. http://CRAN.R-project.org/package=monomvn
  • Griffin, J. E. and Brown, P. J. (2010). “Inference with Normal-Gamma Prior Distributions in Regression Problems.” Bayesian Analysis, 5(1): 171–188.
  • Hahn, R. P., He, J., and Lopes, H. (2016). fastHorseshoe: The Elliptical Slice Sampler for Bayesian Horseshoe Regression. R package version 0.1.0. https://cran.r-project.org/package=fastHorseshoe
  • Jiang, W. and Zhang, C.-H. (2009). “General maximum likelihood empirical Bayes estimation of normal means.” Annals of Statistics, 37(4): 1647–1684.
  • Johnson, V. E. and Rossell, D. (2010). “On the use of non-local prior densities in Bayesian hypothesis tests.” Journal of the Royal Statistical Society. Series B, Statistical Methodology, 72(2): 143–170.
  • Johnstone, I. M. and Silverman, B. W. (2004). “Needles and Straw in Haystacks: Empirical Bayes Estimates of Possibly Sparse Sequences.” Annals of Statistics, 32(4): 1594–1649.
  • Li, K.-C. (1989). “Honest confidence regions for nonparametric regression.” Annals of Statistics, 17(3): 1001–1008.
  • Liu, H. and Yu, B. (2013). “Asymptotic properties of Lasso+mLS and Lasso+Ridge in sparse high-dimensional linear regression.” Electronic Journal of Statistics, 7: 3124–3169.
  • Makalic, E. and Schmidt, D. F. (2016). “A Simple Sampler for the Horseshoe Estimator.” IEEE Signal Processing Letters, 23(1): 179–182.
  • Nickl, R. and Szabo, B. (2016). “A sharp adaptive confidence ball for self-similar functions.” Stochastic Processes and their Applications, 126(12): 3913–3934. http://www.sciencedirect.com/science/article/pii/S0304414916300394
  • Nickl, R. and van de Geer, S. (2013). “Confidence sets in sparse regression.” Annals of Statistics, 41(6): 2852–2876.
  • Picard, D. and Tribouley, K. (2000). “Adaptive confidence interval for pointwise curve estimation.” Annals of Statistics, 28(1): 298–335. http://projecteuclid.org/euclid.aos/1016120374
  • Polson, N. G. and Scott, J. G. (2010). “Shrink Globally, Act Locally: Sparse Bayesian Regularization and Prediction.” In Bernardo, J., Bayarri, M., Berger, J., Dawid, A., Heckerman, D., Smith, A., and West, M. (eds.), Bayesian Statistics 9. Oxford University Press.
  • Polson, N. G. and Scott, J. G. (2012a). “Good, Great or Lucky? Screening for Firms with Sustained Superior Performance Using Heavy-Tailed Priors.” The Annals of Applied Statistics, 6(1): 161–185.
  • Polson, N. G. and Scott, J. G. (2012b). “On the Half-Cauchy Prior for a Global Scale Parameter.” Bayesian Analysis, 7(4): 887–902.
  • Ray, K. (2014). “Adaptive Bernstein-von Mises theorems in Gaussian white noise.” ArXiv e-prints.
  • Robins, J. and van der Vaart, A. (2006). “Adaptive nonparametric confidence sets.” Annals of Statistics, 34(1): 229–253.
  • Ročková, V. (2015). “Bayesian estimation of sparse signals with a continuous spike-and-slab prior.”
  • Rousseau, J. and Szabo, B. (2016). “Asymptotic frequentist coverage properties of Bayesian credible sets for sieve priors in general settings.” ArXiv e-prints.
  • Scott, J. G. (2010). “Parameter Expansion in Local-Shrinkage Models.” ArXiv: 1010.5265.
  • Scott, J. G. (2011). “Bayesian Estimation of Intensity Surfaces on the Sphere via Needlet Shrinkage and Selection.” Bayesian Analysis, 6(2): 307–328.
  • Serra, P. and Krivobokova, T. (2017). “Adaptive Empirical Bayesian Smoothing Splines.” Bayesian Analysis, 12(1): 219–238.
  • Sniekers, S. and van der Vaart, A. (2015a). “Adaptive Bayesian credible sets in regression with a Gaussian process prior.” Electronic Journal of Statistics, 9(2): 2475–2527.
  • Sniekers, S. and van der Vaart, A. (2015b). “Adaptive credible bands in nonparametric regression with Brownian motion prior.” preprint.
  • Sniekers, S. and van der Vaart, A. (2015c). “Credible sets in the fixed design model with Brownian motion prior.” Journal of Statistical Planning and Inference, 166: 78–86.
  • Szabó, B., van der Vaart, A., and van Zanten, H. (2015a). “Honest Bayesian confidence sets for the L2-norm.” Journal of Statistical Planning and Inference, 166: 36–51. Special Issue on Bayesian Nonparametrics. http://www.sciencedirect.com/science/article/pii/S0378375814001244
  • Szabó, B., van der Vaart, A. W., and van Zanten, J. H. (2015b). “Frequentist coverage of adaptive nonparametric Bayesian credible sets.” Annals of Statistics, 43(4): 1391–1428.
  • Tibshirani, R. (1996). “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society. Series B, Statistical Methodology, 58(1): 267–288.
  • van de Geer, S., Bühlmann, P., Ritov, Y., and Dezeure, R. (2014). “On asymptotically optimal confidence regions and tests for high-dimensional models.” Annals of Statistics, 42(3): 1166–1202.
  • van de Geer, S., Bühlmann, P., and Zhou, S. (2011). “The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso).” Electronic Journal of Statistics, 5: 688–749.
  • van der Pas, S., Salomond, J.-B., and Schmidt-Hieber, J. (2016a). “Conditions for posterior contraction in the sparse normal means problem.” Electronic Journal of Statistics, 10(1): 976–1000.
  • van der Pas, S., Scott, J., Chakraborty, A., and Bhattacharya, A. (2016b). horseshoe: Implementation of the Horseshoe Prior. R package version 0.1.0. https://CRAN.R-project.org/package=horseshoe
  • van der Pas, S., Szabó, B., and van der Vaart, A. (2017a). “Adaptive posterior contraction rates for the horseshoe.” To appear in Electronic Journal of Statistics.
  • van der Pas, S., Szabó, B., and van der Vaart, A. (2017b). “Supplement to: Uncertainty quantification for the horseshoe”. Bayesian Analysis.
  • van der Pas, S. L., Kleijn, B. J. K., and van der Vaart, A. W. (2014). “The horseshoe estimator: Posterior concentration around nearly black vectors.” Electronic Journal of Statistics, 8(2): 2585–2618.
  • Zhang, C.-H. and Zhang, S. S. (2014). “Confidence intervals for low dimensional parameters in high dimensional linear models.” Journal of the Royal Statistical Society. Series B, Statistical Methodology, 76(1): 217–242.

Supplemental materials