Scalable Empirical Bayes Inference and Bayesian Sensitivity Analysis

Abstract

Consider a Bayesian setup in which we observe Y, whose distribution depends on a parameter θ, that is, $\mathit{Y}|\mathit{\theta }\sim {\mathit{\pi }}_{\mathit{Y}|\mathit{\theta }}$. The parameter θ is unknown and treated as random, and a prior distribution chosen from some parametric family $\{{\mathit{\pi }}_{\mathit{\theta }}(·;\mathit{h}),\mathit{h}\in \mathcal{H}\}$, is to be placed on it. For the subjective Bayesian there is a single prior in the family which represents his or her beliefs about θ, but determination of this prior is very often extremely difficult. In the empirical Bayes approach, the latent distribution on θ is estimated from the data. This is usually done by choosing the value of the hyperparameter h that maximizes some criterion. Arguably the most common way of doing this is to let $\mathit{m}(\mathit{h})$ be the marginal likelihood of h, that is, $\mathit{m}(\mathit{h})=\int {\mathit{\pi }}_{\mathit{Y}|\mathit{\theta }}{\mathit{\nu }}_{\mathit{h}}(\mathit{\theta })\mathit{d}\mathit{\theta }$, and choose the value of h that maximizes $\mathit{m}(·)$. Unfortunately, except for a handful of textbook examples, analytic evaluation of ${arg\hspace{0.17em}max}_{\mathit{h}}\mathit{m}(\mathit{h})$ is not feasible. The purpose of this paper is two-fold. First, we review the literature on estimating it and find that the most commonly used procedures are either potentially highly inaccurate or don’t scale well with the dimension of h, the dimension of θ, or both. Second, we present a method for estimating ${arg\hspace{0.17em}max}_{\mathit{h}}\mathit{m}(\mathit{h})$, based on Markov chain Monte Carlo, that applies very generally and scales well with dimension. Let g be a real-valued function of θ, and let $\mathit{I}(\mathit{h})$ be the posterior expectation of $\mathit{g}(\mathit{\theta })$ when the prior is ${\mathit{\nu }}_{\mathit{h}}$. As a byproduct of our approach, we show how to obtain point estimates and globally-valid confidence bands for the family $\mathit{I}(\mathit{h}),\mathit{h}\in \mathcal{H}$. To illustrate the scope of our methodology we provide three detailed examples, having different characters.