The TAP free energy for high-dimensional linear regression

Abstract

We derive a variational representation for the log-normalizing constant of the posterior distribution in Bayesian linear regression with a uniform spherical prior and an i.i.d. Gaussian design. We work under the “proportional” asymptotic regime, where the number of observations and the number of features grow at a proportional rate. Our representation holds when the variance of the additive noise is sufficiently large, which corresponds to a high-temperature condition in statistical physics. This rigorously establishes the Thouless–Anderson–Palmer (TAP) approximation arising from spin glass theory, and proves a conjecture of (In 2014 IEEE International Symposium on Information Theory (2014) 1499–1503 IEEE) in the special case of the spherical prior (at sufficiently high temperature).