Infinite Codes for Memoryless Channels

Abstract

For a memory less channel with finite input alphabet $A$, finite output alphabet $B$, and probability law $p(b\mid a)$, the capacity $C$ is defined as the maximum over all probability distributions $q$ on $A$ of $$\sum_{ab} q(a)p(b\mid a)\log_2(p(b\mid a)/\sum_a q(a)p(b\mid a)).$$ Shannon [1] has obtained the following result. Exponential error bound. For any $C_0 < C$ there is a number $\rho < 1$ such that, for every positive integer $N$, there is a set $S \subset A^{(N)}$ with at least $2^{C_0N}$ elements and a function $g$ from $B^{(N)}$ to $S$, such that, for every $s = (a_1, \cdots, a_N) \varepsilon S$, $$\sum p(b_1|a_1) \cdots p(b_N|a_N) < 2\rho^N,$$ where the sum extends over all sequences $b_1, \cdots, b_N$ for which $g(b_1, \cdots, b_N) \neq s$. Thus if the sender selects any $s \varepsilon S$ and places its letters $a_1, \cdots, a_N$ successively into the channel, and the receiver, on observing the resulting output sequence $b_1, \cdots, b_N$, decides that the input was $g(b_1, \cdots, b_N)$, the probability that he makes an error is less than $2\rho^N$, no matter what $s \varepsilon S$ was chosen. This result may be described as follows: it is possible to transmit at any rate $C_0 < C$, with arbitrarily small probability of error, by using block codes of sufficient length. We wish to draw a slightly stronger conclusion, as follows. We imagine an infinite sequence $x = (x_1, x_2, \cdots)$ of 0's and 1's, which we are required to transmit across the channel. At time $N$, the sender will have observed the first $\lbrack C_0N\rbrack$ coordinates of $x$, and will place the $N$th input symbol in the channel. The receiver, having at this point observed the first $N$ channel outputs, will estimate the first $M(N)$ coordinates of $x$. If $M(N)/C_0N \rightarrow 1$ as $N \rightarrow \infty$ and if, for every $x$, all but a finite number of his estimates are correct (i.e., agree with $x$ in every coordinate estimated) with probability 1, we shall say that the channel is being used at rate $C_0$. Our result is that, in this sense, a (memoryless) channel can be used at any rate $C_0 < C$. The result stated below is exactly this result, for the special case $C_0 = 1$. The general case involves no new ideas, but only more notation, and we shall restrict attention to the case $C_0 = 1$. The function $f_n$ of a code, as defined below, specifies the $n$th channel input symbol, as a function of the first $n$ coordinates of $x$. The number $M(n)$ is the number of $x$ coordinates to be estimated by the receiver after observing the first $n$ output symbols, and the function $g_n$ specifies the estimate. We now state the result precisely. For any finite set $S$, we denote by $S^{(N)}$ the set of all sequences $(s_1, \cdots, s_N)$, where $s_n \varepsilon S$ for $n = 1, 2, \cdots, N$. For a memoryless channel with finite input alphabet $A$, finite output alphabet $B$, an infinite code (for transmitting at rate 1) is defined as consisting of (a) a sequence $\{f_n\}$ of functions, where $f_n$ maps $I^{(n)}$ into $A$, and $I$ consists of the two elements 0 and 1, (b) a nondecreasing sequence $\{M(n)\}$ of positive integers such that $M(n)/n \rightarrow 1$ as $n \rightarrow \infty$, and (c) a sequence $\{g_n\}$ of functions, where $\{g_n\}$ maps $B^{(n)}$ into $I^{(M(n))}$. An infinite sequence $x = (x_1, x_2, \cdots)$ of 0's and 1's, together with an infinite code, defines a sequence of independent output variables $y_1, y_2, \cdots$, with $$\Pr \{y_n = b\} = p(b \mid f_n(x_1, \cdots, x_n)),$$ where $p(b \mid a)$ is the probability that the output symbol of the channel is $b$, given that the corresponding input symbol is $a$, and defines a sequence of estimated messages $t_1, t_2, \cdots$, where $t_n = g_n(y_1, \cdots, y_n)$. We shall say that the code is effective at $x$ if, with probability 1, $$t_n = (x_1, \cdots, x_{M(n)})$$ for all sufficiently large $n$, and shall say that the code is effective if it is effective for every $x$. The result of this note is the THEOREM: For any memoryless channel with capacity $C > 1$, there is an effective code.