The Annals of Probability

A Sandwich Proof of the Shannon-McMillan-Breiman Theorem

Paul H. Algoet and Thomas M. Cover

Full-text: Open access

Abstract

Let $\{X_t\}$ be a stationary ergodic process with distribution $P$ admitting densities $p(x_0,\ldots, x_{n-1})$ relative to a reference measure $M$ that is finite order Markov with stationary transition kernel. Let $I_M(P)$ denote the relative entropy rate. Then $n^{-1}\log p(X_0,\ldots, X_{n-1}) \rightarrow I_M(P) \mathrm{a.s.} (P).$ We present an elementary proof of the Shannon-McMillan-Breiman theorem and the preceding generalization, obviating the need to verify integrability conditions and also covering the case $I_M(P) = \infty$. A sandwich argument reduces the proof to direct applications of the ergodic theorem.

Article information

Source
Ann. Probab., Volume 16, Number 2 (1988), 899-909.

Dates
First available in Project Euclid: 19 April 2007

Permanent link to this document
https://projecteuclid.org/euclid.aop/1176991794

Digital Object Identifier
doi:10.1214/aop/1176991794

Mathematical Reviews number (MathSciNet)
MR929085

Zentralblatt MATH identifier
0653.28013

JSTOR
links.jstor.org

Subjects
Primary: 28D05: Measure-preserving transformations
Secondary: 94A17: Measures of information, entropy 28A65 28D20: Entropy and other invariants 60F15: Strong theorems

Keywords
Shannon-McMillan-Breiman theorem asymptotic equipartition property (AEP) ergodic theorem of information theory relative entropy rate likelihood ratio sandwich argument Markov approximation asymptotically mean stationary

Citation

Algoet, Paul H.; Cover, Thomas M. A Sandwich Proof of the Shannon-McMillan-Breiman Theorem. Ann. Probab. 16 (1988), no. 2, 899--909. doi:10.1214/aop/1176991794. https://projecteuclid.org/euclid.aop/1176991794


Export citation