Inexact Laplace Approximation and the Use of Posterior Mean in Bayesian Inference

Abstract

The prominent Bernstein – von Mises (BvM) Theorem claims a kind of approximation of the posterior distribution by a Gaussian one with the covariance close to the inverse of the total Fisher information matrix. A more general Laplace approximation result states a similar Gaussian approximation of the posterior with the parameters depending on the prior. These two results build a basis for Bayesian inference and uncertainty quantification in a rather general situation. Spokoiny and Panov (2021) offered a new look at this problem which allows to state rather strong results on the quality of Gaussian approximation in non-asymptotic and dimension free form assuming linearity and concavity of log-likelihood function which can be misspecified. The established results provide explicit non-asymptotic bounds on the quality of a Gaussian approximation of the posterior distribution in total variation distance in terms of the so called effective dimension ${\mathtt{p}}_G$ defined as interplay between information contained in the data and in the prior distribution. This paper substantially improves and further develops the results from Spokoiny and Panov (2021) using the recent progress on high dimensional Laplace approximation. We address the question of effective and critical dimension in Bayesian inference, the relations between Laplace approximation and Bernstein–von Mises Theorem, and, particularly, the use of posterior mean instead of Maximum A Posteriori Probability estimator in Bayesian inference. The results are illustrated for the case of log-density estimation.