Asymptotics of input-constrained binary symmetric channel capacity
Guangyue Han and Brian Marcus
Source: Ann. Appl. Probab. Volume 19, Number 3 (2009), 1063-1091.
Abstract
We study the classical problem of noisy constrained capacity in the case of the binary symmetric channel (BSC), namely, the capacity of a BSC whose inputs are sequences chosen from a constrained set. Motivated by a result of Ordentlich and Weissman [In Proceedings of IEEE Information Theory Workshop (2004) 117–122], we derive an asymptotic formula (when the noise parameter is small) for the entropy rate of a hidden Markov chain, observed when a Markov chain passes through a BSC. Using this result, we establish an asymptotic formula for the capacity of a BSC with input process supported on an irreducible finite type constraint, as the noise parameter tends to zero.
Primary Subjects: 60K99, 94A15
Secondary Subjects: 60J10
Keywords: Hidden Markov chain; entropy; constrained capacity
Full-text: Access denied (no subscription detected)
We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber.
If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text
Links and Identifiers
Permanent link to this document: http://projecteuclid.org/euclid.aoap/1245071019
Digital Object Identifier: doi:10.1214/08-AAP570
Zentralblatt MATH identifier:
05580233
Mathematical Reviews number (MathSciNet):
MR2537199
References
[1] Arimoto, S. (1972). An algorithm for computing the capacity of arbitrary discrete memoryless channels. IEEE Trans. Inform. Theory IT-18 14–20.
Mathematical Reviews (MathSciNet):
MR403796
Digital Object Identifier: doi:10.1109/TIT.1972.1054753
[2] Arnold, D. and Loeliger, H. (2001). The information rate of binary-input channels with memory. In Proceedings of 2001 IEEE International Conference on Communications (Helsinki, Finland) 2692–2695.
[3] Arnold, D. M., Loeliger, H.-A., Vontobel, P. O., Kavčić, A. and Zeng, W. (2006). Simulation-based computation of information rates for channels with memory. IEEE Trans. Inform. Theory 52 3498–3508.
Mathematical Reviews (MathSciNet):
MR2242361
Digital Object Identifier: doi:10.1109/TIT.2006.878110
[4] Arnold, L., Gundlach, V. M. and Demetrius, L. (1994). Evolutionary formalism for products of positive random matrices. Ann. Appl. Probab. 4 859–901.
Mathematical Reviews (MathSciNet):
MR1284989
Digital Object Identifier: doi:10.1214/aoap/1177004975
Project Euclid: euclid.aoap/1177004975
[5] Birch, J. J. (1962). Approximation for the entropy for functions of Markov chains. Ann. Math. Statist. 33 930–938.
Mathematical Reviews (MathSciNet):
MR141162
Digital Object Identifier: doi:10.1214/aoms/1177704462
Project Euclid: euclid.aoms/1177704462
[6] Blackwell, D. (1957). The entropy of functions of finite-state Markov chains. In Transactions of the First Prague Conference on Information Theory, Statistical Decision Functions, Random Processes 13–20. Publishing House of the Czechoslovak Academy of Sciences, Prague.
Mathematical Reviews (MathSciNet):
MR100297
[7] Blahut, R. E. (1972). Computation of channel capacity and rate-distortion functions. IEEE Trans. Inform. Theory IT-18 460–473.
Mathematical Reviews (MathSciNet):
MR476161
Digital Object Identifier: doi:10.1109/TIT.1972.1054855
[8] Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. Wiley, New York.
Mathematical Reviews (MathSciNet):
MR1122806
[9] Egner, S., Balakirsky, V., Tolhuizen, L., Baggen, S. and Hollmann, H. (2004). On the entropy rate of a hidden Markov model. In Proceedings of the 2004 IEEE International Symposium on Information Theory (Chicago, IL) 12.
[10] Ephraim, Y. and Merhav, N. (2002). Hidden Markov processes. IEEE Trans. Inform. Theory 48 1518–1569. Special issue on Shannon theory: Perspective, trends, and applications.
Mathematical Reviews (MathSciNet):
MR1909472
Digital Object Identifier: doi:10.1109/TIT.2002.1003838
[11] Gharavi, R. and Anantharam, V. (2005). An upper bound for the largest Lyapunov exponent of a Markovian product of nonnegative matrices. Theoret. Comput. Sci. 332 543–557.
Mathematical Reviews (MathSciNet):
MR2122519
Digital Object Identifier: doi:10.1016/j.tcs.2004.12.025
[12] Gray, R. M. (1990). Entropy and Information Theory. Springer, New York.
Mathematical Reviews (MathSciNet):
MR1070359
[13] Han, G. and Marcus, B. (2006). Analyticity of entropy rate of hidden Markov chains. IEEE Trans. Inform. Theory 52 5251–5266.
Mathematical Reviews (MathSciNet):
MR2300690
Digital Object Identifier: doi:10.1109/TIT.2006.885481
[14] Han, G. and Marcus, B. (2007). Derivatives of entropy rate in special families of hidden Markov chains. IEEE Trans. Inform. Theory 53 2642–2652.
Mathematical Reviews (MathSciNet):
MR2319402
Digital Object Identifier: doi:10.1109/TIT.2007.899467
[15] Holliday, T., Goldsmith, A. and Glynn, P. (2003). Capacity of finite state Markov channels with general inputs. In Proceedings of the 2003 IEEE International Symposium on Information Theory 289.
[16] Holliday, T., Goldsmith, A. and Glynn, P. (2006). Capacity of finite state channels based on Lyapunov exponents of random matrices. IEEE Trans. Inform. Theory 52 3509–3532.
Mathematical Reviews (MathSciNet):
MR2242362
Digital Object Identifier: doi:10.1109/TIT.2006.878230
[17] Jacquet, P., Seroussi, G. and Szpankowski, W. (2004). On the entropy of a hidden Markov process (extended abstract). In Proceedings of Data Compression Conference 362–371.
[18] Jacquet, P., Seroussi, G. and Szpankowski, W. (2007). On the entropy of a hidden Markov process. Theoret. Comput. Sci. 395 203–219.
Mathematical Reviews (MathSciNet):
MR2424508
Digital Object Identifier: doi:10.1016/j.tcs.2008.01.012
[19] Jacquet, P., Seroussi, G. and Szpankowski, W. (2007). Noisy constrained capacity. In Proceeding of the 2007 IEEE International Symposium on Information Theory (Nice, France) 986–990.
[20] Lind, D. and Marcus, B. (1995). An Introduction to Symbolic Dynamics and Coding. Cambridge Univ. Press, Cambridge.
Mathematical Reviews (MathSciNet):
MR1369092
[21] Marcus, B. H., Roth, R. M. and Siegel, P. H. (1998). Constrained systems and coding for recording channels. In Handbook of Coding Theory (V. S. Pless and W. C. Hofman, eds.) II 1635–1764. North-Holland, Amsterdam.
Mathematical Reviews (MathSciNet):
MR1667956
[22] Ordentlich, E. and Weissman, T. (2006). On the optimality of symbol-by-symbol filtering and denoising. IEEE Trans. Inform. Theory 52 19–40.
Mathematical Reviews (MathSciNet):
MR2237333
[23] Ordentlich, E. and Weissman, T. (2004). New bounds on the entropy rate of hidden Markov process. In Proceedings of IEEE Information Theory Workshop (San Antonio, TX) 24–29.
[24] Parry, W. (1964). Intrinsic Markov chains. Trans. Amer. Math. Soc. 112 55–66.
Mathematical Reviews (MathSciNet):
MR161372
Digital Object Identifier: doi:10.2307/1994009
JSTOR: links.jstor.org
[25] Peres, Y. (1990). Analytic dependence of Lyapunov exponents on transition probabilities. In Lyapunov Exponents (Oberwolfach, 1990). Lecture Notes in Mathematics (L. Aruold, H. Crauel and J.-P. Eckman, eds.) 1486 64–80. Springer, Berlin.
Mathematical Reviews (MathSciNet):
MR1178947
Digital Object Identifier: doi:10.1007/BFb0086658
[26] Peres, Y. (1992). Domains of analytic continuation for the top Lyapunov exponent. Ann. Inst. H. Poincaré Probab. Statist. 28 131–148.
Mathematical Reviews (MathSciNet):
MR1158741
[27] Pfister, H., Soriaga, J. and Siegel, P. (2001). The achievable information rates of finite-state ISI channels. In Proceedings of IEEE GLOBECOM (San Antonio, TX) 2992–2996.
[28] Ruelle, D. (1979). Analycity properties of the characteristic exponents of random matrix products. Adv. in Math. 32 68–80.
Mathematical Reviews (MathSciNet):
MR534172
Digital Object Identifier: doi:10.1016/0001-8708(79)90029-X
[29] Shamai (Shitz), S. and Kofman, Y. (1990). On the capacity of binary and Gaussian channels with run-length limited inputs. IEEE Trans. Commun. 38 584–594.
[30] Shannon, C. E. (1948). A mathematical theory of communication. Bell System Tech. J. 27 379–423, 623–656.
Mathematical Reviews (MathSciNet):
MR26286
[31] Sharma, V. and Singh, S. (2001). Entropy and channel capacity in the regenerative setup with applications to Markov channels. In Proceedings of IEEE International Symposium on Information Theory (Washington, DC) 283.
[32] Vontobel, P. O., Kavčić, A., Arnold, D. M. and Loeliger, H.-A. (2008). A generalization of the Blahut-Arimoto algorithm to finite-state channels. IEEE Trans. Inform. Theory 54 1887–1918.
[33] Zehavi, E. and Wolf, J. (1988). On runlength codes. IEEE Trans. Inform. Theory 34 45–54.
Mathematical Reviews (MathSciNet):
MR880161
Digital Object Identifier: doi:10.1109/TIT.1987.1057292
[34] Zuk, O., Kanter, I. and Domany, E. (2005). The entropy of a binary hidden Markov process. J. Stat. Phys. 121 343–360.
Mathematical Reviews (MathSciNet):
MR2185333
Digital Object Identifier: doi:10.1007/s10955-005-7576-y
[35] Zuk, O., Domany, E., Kanter, I. and Aizenman, M. (2006). From finite-system entropy to entropy rate for a Hidden Markov process. IEEE Signal Processing Letters 13 517–520.
The Annals of Applied Probability
