Variational Bayes and a Problem of Reliable Communication I: Finite Systems

Abstract

This paper is the first in a two-part study of a variational Bayesian method and its application to a problem of reliable communication. The variational method expresses a Bayesian posterior distribution as the unique minimizer of a quantity dubbed apparent information. This has the same nature as the free energy of statistical mechanics. The minimum apparent information coincides with the full information of the observation.

Reliable communication over an error prone channel can be achieved by the use of random block coding, as originally proposed by Shannon. The primary Bayesian problem in this context, is that of estimating the transmitted block from observations of the output of the channel. Scaling limits for the various information quantities are derived for this problem; these show that the primary problem undergoes a second-order phase transition, in a very precise sense, at the channel capacity; the code rate is shown to play the role of absolute temperature.

Shannon’s reliability function is recovered as the scaling limit of the full information of a secondary Bayesian problem, in which the channel noise and random code are estimated from the observation of a block decoding error. This secondary problem undergoes a third-order phase transition at a second critical code rate.