Annals of Statistics

Computational approaches for empirical Bayes methods and Bayesian sensitivity analysis

Eugenia Buta and Hani Doss

Full-text: Open access

Abstract

We consider situations in Bayesian analysis where we have a family of priors νh on the parameter θ, where h varies continuously over a space $\mathcal{H}$, and we deal with two related problems. The first involves sensitivity analysis and is stated as follows. Suppose we fix a function f of θ. How do we efficiently estimate the posterior expectation of f(θ) simultaneously for all h in $\mathcal{H}$? The second problem is how do we identify subsets of $\mathcal{H}$ which give rise to reasonable choices of νh? We assume that we are able to generate Markov chain samples from the posterior for a finite number of the priors, and we develop a methodology, based on a combination of importance sampling and the use of control variates, for dealing with these two problems. The methodology applies very generally, and we show how it applies in particular to a commonly used model for variable selection in Bayesian linear regression, and give an illustration on the US crime data of Vandaele.

Article information

Source
Ann. Statist., Volume 39, Number 5 (2011), 2658-2685.

Dates
First available in Project Euclid: 22 December 2011

Permanent link to this document
https://projecteuclid.org/euclid.aos/1324563351

Digital Object Identifier
doi:10.1214/11-AOS913

Mathematical Reviews number (MathSciNet)
MR2906882

Zentralblatt MATH identifier
1231.62008

Subjects
Primary: 62F15: Bayesian inference 91-08: Computational methods
Secondary: 62F12: Asymptotic properties of estimators

Keywords
Bayes factors control variates ergodicity hyperparameter selection importance sampling Markov chain Monte Carlo

Citation

Buta, Eugenia; Doss, Hani. Computational approaches for empirical Bayes methods and Bayesian sensitivity analysis. Ann. Statist. 39 (2011), no. 5, 2658--2685. doi:10.1214/11-AOS913. https://projecteuclid.org/euclid.aos/1324563351


Export citation

References

  • Antoniak, C. E. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. 2 1152–1174.
  • Athreya, K. B., Doss, H. and Sethuraman, J. (1996). On the convergence of the Markov chain simulation method. Ann. Statist. 24 69–100.
  • Burr, D. and Doss, H. (2005). A Bayesian semiparametric model for random-effects meta-analysis. J. Amer. Statist. Assoc. 100 242–251.
  • Buta, E. (2010). Computational approaches for empirical Bayes methods and Bayesian sensitivity analysis. Ph.D. thesis, Univ. Florida, Gainesville, FL.
  • Buta, E. and Doss, H. (2011). Supplement to “Computational approaches for empirical Bayes methods and Bayesian sensitivity analysis.” DOI:10.1214/11-AOS913SUPP.
  • Cui, W. and George, E. I. (2008). Empirical Bayes vs. fully Bayes variable selection. J. Statist. Plann. Inference 138 888–900.
  • Doss, H. (1994). Comment on “Markov chains for exploring posterior distributions,” by L. Tierney. Ann. Statist. 22 1728–1734.
  • Doss, H. (2007). Bayesian model selection: Some thoughts on future directions. Statist. Sinica 17 413–421.
  • Doss, H. (2010). Estimation of large families of Bayes factors from Markov chain output. Statist. Sinica 20 537–560.
  • Fernández, C., Ley, E. and Steel, M. F. J. (2001). Benchmark priors for Bayesian model averaging. J. Econometrics 100 381–427.
  • Flegal, J. M. and Jones, G. L. (2010). Batch means and spectral variance estimators in Markov chain Monte Carlo. Ann. Statist. 38 1034–1070.
  • George, E. I. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–747.
  • Geyer, C. J. (1994). Estimating normalizing constants and reweighting mixtures in Markov chain Monte Carlo. Technical Report 568r, Dept. Statistics, Univ. Minnesota.
  • Gill, R. D., Vardi, Y. and Wellner, J. A. (1988). Large sample theory of empirical distributions in biased sampling models. Ann. Statist. 16 1069–1112.
  • Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57 97–109.
  • Kong, A., McCullagh, P., Meng, X. L., Nicolae, D. and Tan, Z. (2003). A theory of statistical models for Monte Carlo integration (with discussion). J. R. Stat. Soc. Ser. B Stat. Methodol. 65 585–618.
  • Liang, F., Paulo, R., Molina, G., Clyde, M. A. and Berger, J. O. (2008). Mixtures of g-priors for Bayesian variable selection. J. Amer. Statist. Assoc. 103 410–423.
  • Meng, X.-L. and Wong, W. H. (1996). Simulating ratios of normalizing constants via a simple identity: A theoretical exploration. Statist. Sinica 6 831–860.
  • Mykland, P., Tierney, L. and Yu, B. (1995). Regeneration in Markov chain samplers. J. Amer. Statist. Assoc. 90 233–241.
  • Smith, M. and Kohn, R. (1996). Nonparametric regression using Bayesian variable selection. J. Econometrics 75 317–343.
  • Tan, Z. (2004). On a likelihood approach for Monte Carlo integration. J. Amer. Statist. Assoc. 99 1027–1036.
  • Tierney, L. (1994). Markov chains for exploring posterior distributions. Ann. Statist. 22 1701–1728.
  • Vandaele, W. (1978). Participation in illegitimate activities: Ehrlich revisited. In Deterrence and Incapacitation. U.S. National Academy of Sciences, Washington, DC.
  • Vardi, Y. (1985). Empirical distributions in selection bias models. Ann. Statist. 13 178–203.
  • Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g-prior distributions. In Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti (P. K. Goel and A. Zellner, eds.) 233–243. North-Holland, Amsterdam.

Supplemental materials

  • Supplementary material: Additional technical details. We show that when estimating the Bayes factors using control variates, the estimate that is optimal when the samples are i.i.d. sequences is no longer optimal when the samples are Markov chains. We also give technical arguments regarding the consistency of spectral estimates of the variance of our estimators.