The Annals of Mathematical Statistics

Infinite Codes for Memoryless Channels

David Blackwell

Full-text: Open access


For a memory less channel with finite input alphabet $A$, finite output alphabet $B$, and probability law $p(b\mid a)$, the capacity $C$ is defined as the maximum over all probability distributions $q$ on $A$ of $$\sum_{ab} q(a)p(b\mid a)\log_2(p(b\mid a)/\sum_a q(a)p(b\mid a)).$$ Shannon [1] has obtained the following result. Exponential error bound. For any $C_0 < C$ there is a number $\rho < 1$ such that, for every positive integer $N$, there is a set $S \subset A^{(N)}$ with at least $2^{C_0N}$ elements and a function $g$ from $B^{(N)}$ to $S$, such that, for every $s = (a_1, \cdots, a_N) \varepsilon S$, $$\sum p(b_1|a_1) \cdots p(b_N|a_N) < 2\rho^N,$$ where the sum extends over all sequences $b_1, \cdots, b_N$ for which $g(b_1, \cdots, b_N) \neq s$. Thus if the sender selects any $s \varepsilon S$ and places its letters $a_1, \cdots, a_N$ successively into the channel, and the receiver, on observing the resulting output sequence $b_1, \cdots, b_N$, decides that the input was $g(b_1, \cdots, b_N)$, the probability that he makes an error is less than $2\rho^N$, no matter what $s \varepsilon S$ was chosen. This result may be described as follows: it is possible to transmit at any rate $C_0 < C$, with arbitrarily small probability of error, by using block codes of sufficient length. We wish to draw a slightly stronger conclusion, as follows. We imagine an infinite sequence $x = (x_1, x_2, \cdots)$ of 0's and 1's, which we are required to transmit across the channel. At time $N$, the sender will have observed the first $\lbrack C_0N\rbrack$ coordinates of $x$, and will place the $N$th input symbol in the channel. The receiver, having at this point observed the first $N$ channel outputs, will estimate the first $M(N)$ coordinates of $x$. If $M(N)/C_0N \rightarrow 1$ as $N \rightarrow \infty$ and if, for every $x$, all but a finite number of his estimates are correct (i.e., agree with $x$ in every coordinate estimated) with probability 1, we shall say that the channel is being used at rate $C_0$. Our result is that, in this sense, a (memoryless) channel can be used at any rate $C_0 < C$. The result stated below is exactly this result, for the special case $C_0 = 1$. The general case involves no new ideas, but only more notation, and we shall restrict attention to the case $C_0 = 1$. The function $f_n$ of a code, as defined below, specifies the $n$th channel input symbol, as a function of the first $n$ coordinates of $x$. The number $M(n)$ is the number of $x$ coordinates to be estimated by the receiver after observing the first $n$ output symbols, and the function $g_n$ specifies the estimate. We now state the result precisely. For any finite set $S$, we denote by $S^{(N)}$ the set of all sequences $(s_1, \cdots, s_N)$, where $s_n \varepsilon S$ for $n = 1, 2, \cdots, N$. For a memoryless channel with finite input alphabet $A$, finite output alphabet $B$, an infinite code (for transmitting at rate 1) is defined as consisting of (a) a sequence $\{f_n\}$ of functions, where $f_n$ maps $I^{(n)}$ into $A$, and $I$ consists of the two elements 0 and 1, (b) a nondecreasing sequence $\{M(n)\}$ of positive integers such that $M(n)/n \rightarrow 1$ as $n \rightarrow \infty$, and (c) a sequence $\{g_n\}$ of functions, where $\{g_n\}$ maps $B^{(n)}$ into $I^{(M(n))}$. An infinite sequence $x = (x_1, x_2, \cdots)$ of 0's and 1's, together with an infinite code, defines a sequence of independent output variables $y_1, y_2, \cdots$, with $$\Pr \{y_n = b\} = p(b \mid f_n(x_1, \cdots, x_n)),$$ where $p(b \mid a)$ is the probability that the output symbol of the channel is $b$, given that the corresponding input symbol is $a$, and defines a sequence of estimated messages $t_1, t_2, \cdots$, where $t_n = g_n(y_1, \cdots, y_n)$. We shall say that the code is effective at $x$ if, with probability 1, $$t_n = (x_1, \cdots, x_{M(n)})$$ for all sufficiently large $n$, and shall say that the code is effective if it is effective for every $x$. The result of this note is the THEOREM: For any memoryless channel with capacity $C > 1$, there is an effective code.

Article information

Ann. Math. Statist., Volume 30, Number 4 (1959), 1242-1244.

First available in Project Euclid: 27 April 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier



Blackwell, David. Infinite Codes for Memoryless Channels. Ann. Math. Statist. 30 (1959), no. 4, 1242--1244. doi:10.1214/aoms/1177706107.

Export citation