Uniform-in-bandwidth consistency for kernel-type estimators of Shannon’s entropy

Uniform-in-bandwidth consistency for kernel-type estimators of Shannon’s entropy

Salim Bouzebda and Issam Elhattab

Source: Electron. J. Statist. Volume 5 (2011), 440-459.

Abstract

We establish uniform-in-bandwidth consistency for kernel-type estimators of the differential entropy. We consider two kernel-type estimators of Shannon’s entropy. As a consequence, an asymptotic 100% confidence interval of entropy is provided.

First Page: Show

Primary Subjects: 62F12, 62F03, 62G30, 60F17, 62E20

Keywords: Entropy; kernel estimation; uniform in bandwidth; consistency

Full-text: Open access

Screen Optimized PDF File (256 KB)

Links and Identifiers

Permanent link to this document: http://projecteuclid.org/euclid.ejs/1305034910
Digital Object Identifier: doi:10.1214/11-EJS614
Mathematical Reviews number (MathSciNet): MR2802051

back to Table of Contents

References

[1] Ahmad, I. A. and Lin, P. E. (1976). A nonparametric estimation of the entropy for absolutely continuous distributions., IEEE Trans. Information Theory, IT-22(3), 372–375.

Mathematical Reviews (MathSciNet): MR408987

Digital Object Identifier: doi:10.1109/TIT.1976.1055550

[2] Akaike, H. (1954). An approximation to the density function., Ann. Inst. Statist. Math., Tokyo, 6, 127–132.

Mathematical Reviews (MathSciNet): MR67412

Zentralblatt MATH: 0058.12302

Digital Object Identifier: doi:10.1007/BF02900741

[3] Ash, R. (1965)., Information theory. Interscience Tracts in Pure and Applied Mathematics, No. 19. Interscience Publishers John Wiley & Sons, New York-London-Sydney.

Mathematical Reviews (MathSciNet): MR229475

[4] Beirlant, J., Dudewicz, E. J., Györfi, L., and van der Meulen, E. C. (1997). Nonparametric entropy estimation: an overview., Int. J. Math. Stat. Sci., 6(1), 17–39.

Mathematical Reviews (MathSciNet): MR1471870

Zentralblatt MATH: 0882.62003

[5] Berger, T. (1971)., Rate distortion theory. Prentice-Hall Inc., Englewood Cliffs, N. J. A mathematical basis for data compression, Prentice-Hall Series in Information and System Sciences.

Mathematical Reviews (MathSciNet): MR408988

[6] Bosq, D. and Lecoutre, J.-P. (1987)., Théorie de l’estimation fonctionnelle. Économie et Statistiques Avancées. Economica, Paris.

[7] Bouzebda, S. and Elhattab, I. (2009). A strong consistency of a nonparametric estimate of entropy under random censorship., C. R. Math. Acad. Sci. Paris, 347(13-14), 821–826.

Mathematical Reviews (MathSciNet): MR2543991

Zentralblatt MATH: 1167.62410

Digital Object Identifier: doi:10.1016/j.crma.2009.04.021

[8] Bouzebda, S. and Elhattab, I. (2010). Uniform in bandwidth consistency of the kernel-type estimator of the shannon’s entropy., C. R. Math. Acad. Sci. Paris, 348(5-6), 317–321.

Mathematical Reviews (MathSciNet): MR2600131

Zentralblatt MATH: 1185.62072

Digital Object Identifier: doi:10.1016/j.crma.2009.12.007

[9] Clarkson, J. A. and Adams, C. R. (1933). On definitions of bounded variation for functions of two variables., Trans. Amer. Math. Soc., 35(4), 824–854.

Mathematical Reviews (MathSciNet): MR1501718

Digital Object Identifier: doi:10.1090/S0002-9947-1933-1501718-2

[10] Cover, T. M. and Thomas, J. A. (2006)., Elements of information theory. Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, second edition.

Mathematical Reviews (MathSciNet): MR2239987

[11] Csiszár, I. (1962). Informationstheoretische Konvergenzbegriffe im Raum der Wahrscheinlichkeitsverteilungen., Magyar Tud. Akad. Mat. Kutató Int. Közl., 7, 137–158.

Mathematical Reviews (MathSciNet): MR191733

[12] Deheuvels, P. (2000). Uniform limit laws for kernel density estimators on possibly unbounded intervals. In, Recent advances in reliability theory (Bordeaux, 2000), Stat. Ind. Technol., pages 477–492. Birkhäuser Boston, Boston, MA.

Mathematical Reviews (MathSciNet): MR1783500

Zentralblatt MATH: 0997.62038

Digital Object Identifier: doi:10.1007/978-1-4612-1384-0_30

[13] Deheuvels, P. and Mason, D. M. (2004). General asymptotic confidence bands based on kernel-type function estimators., Stat. Inference Stoch. Process., 7(3), 225–277.

Mathematical Reviews (MathSciNet): MR2111291

Digital Object Identifier: doi:10.1023/B:SISP.0000049092.55534.af

[14] Devroye, L. (1987)., A Course in density estimation. Birkhäuser, Boston-Basel-Stuttgart.

Mathematical Reviews (MathSciNet): MR891874

Zentralblatt MATH: 0617.62043

[15] Devroye, L. and Györfi, L. (1985)., Nonparametric density estimation. Wiley Series in Probability and Mathematical Statistics: Tracts on Probability and Statistics. John Wiley & Sons Inc., New York. The L₁ view.

Mathematical Reviews (MathSciNet): MR780746

Zentralblatt MATH: 0546.62015

[16] Devroye, L. and Lugosi, G. (2001)., Combinatorial methods in density estimation. Springer Series in Statistics. Springer-Verlag, New York.

Mathematical Reviews (MathSciNet): MR1843146

Zentralblatt MATH: 0964.62025

[17] Devroye, L. and Wise, G. L. (1980). Detection of abnormal behavior via nonparametric estimation of the support., SIAM J. Appl. Math., 38(3), 480–488.

Mathematical Reviews (MathSciNet): MR579432

Zentralblatt MATH: 0479.62028

Digital Object Identifier: doi:10.1137/0138038

[18] Dmitriev, J. G. and Tarasenko, F. P. (1973). The estimation of functionals of a probability density and its derivatives., Teor. Verojatnost. i Primenen., 18, 662–668.

Mathematical Reviews (MathSciNet): MR359157

[19] Dudewicz, E. J. and van der Meulen, E. C. (1981). Entropy-based tests of uniformity., J. Amer. Statist. Assoc., 76(376), 967–974.

Mathematical Reviews (MathSciNet): MR650913

Zentralblatt MATH: 0484.62035

Digital Object Identifier: doi:10.2307/2287597

[20] Ebrahimi, N., Habibullah, M., and Soofi, E. (1992). Testing exponentiality based on Kullback-Leibler information., J. Roy. Statist. Soc. Ser. B, 54(3), 739–748.

Mathematical Reviews (MathSciNet): MR1185219

[21] Eggermont, P. P. B. and LaRiccia, V. N. (1999). Best asymptotic normality of the kernel density entropy estimator for smooth densities., IEEE Trans. Inform. Theory, 45(4), 1321–1326.

Mathematical Reviews (MathSciNet): MR1686275

Digital Object Identifier: doi:10.1109/18.761291

[22] Einmahl, U. and Mason, D. M. (2000). An empirical process approach to the uniform consistency of kernel-type function estimators., J. Theoret. Probab., 13(1), 1–37.

Mathematical Reviews (MathSciNet): MR1744994

Zentralblatt MATH: 0995.62042

Digital Object Identifier: doi:10.1023/A:1007769924157

[23] Einmahl, U. and Mason, D. M. (2005). Uniform in bandwidth consistency of kernel-type function estimators., Ann. Statist., 33(3), 1380–1403.

Mathematical Reviews (MathSciNet): MR2195639

Zentralblatt MATH: 1079.62040

Digital Object Identifier: doi:10.1214/009053605000000129

Project Euclid: euclid.aos/1120224106

[24] Esteban, M. D., Castellanos, M. E., Morales, D., and Vajda, I. (2001). Monte Carlo comparison of four normality tests using different entropy estimates., Comm. Statist. Simulation Comput., 30(4), 761–785.

Mathematical Reviews (MathSciNet): MR1878453

Zentralblatt MATH: 1008.62505

Digital Object Identifier: doi:10.1081/SAC-100107780

[25] Gallager, R. (1968)., Information theory and reliable communication. New York-London-Sydney-Toronto: John Wiley & Sons, Inc. XVI, 588 p.

[26] Giné, E. and Mason, D. M. (2008). Uniform in bandwidth estimation of integral functionals of the density function., Scand. J. Statist., 35(4), 739–761.

Mathematical Reviews (MathSciNet): MR2468873

Digital Object Identifier: doi:10.1111/j.1467-9469.2008.00600.x

[27] Gokhale, D. (1983). On entropy-based goodness-of-fit tests., Comput. Stat. Data Anal., 1, 157–165.

[28] Gokhale, D. V. and Kullback, S. (1978)., The information in contingency tables, volume 23 of Statistics: Textbooks and Monographs. Marcel Dekker Inc., New York.

Mathematical Reviews (MathSciNet): MR524727

[29] Györfi, L. et van der Meulen, E. C. (1990). An entropy estimate based on a kernel density estimation. In, Limit theorems in probability and statistics (Pécs, 1989), volume 57 of Colloq. Math. Soc. János Bolyai, pages 229–240. North-Holland, Amsterdam.

Mathematical Reviews (MathSciNet): MR1116790

[30] Györfi, L. and van der Meulen, E. C. (1991). On the nonparametric estimation of the entropy functional. In, Nonparametric functional estimation and related topics (Spetses, 1990), volume 335 of NATO Adv. Sci. Inst. Ser. C Math. Phys. Sci., pages 81–95. Kluwer Acad. Publ., Dordrecht.

Mathematical Reviews (MathSciNet): MR1154321

[31] Hobson, E. W. (1958)., The theory of functions of a real variable and the theory of Fourier’s series. Vol. I. Dover Publications Inc., New York, N.Y.

Mathematical Reviews (MathSciNet): MR92828

[32] Jaynes, E. T. (1957). Information theory and statistical mechanics., Phys. Rev. (2), 106, 620–630.

Mathematical Reviews (MathSciNet): MR87305

Digital Object Identifier: doi:10.1103/PhysRev.106.620

[33] Kullback, S. (1959)., Information theory and statistics. John Wiley and Sons, Inc., New York.

Mathematical Reviews (MathSciNet): MR103557

[34] Lazo, A. C. and Rathie, P. N. (1978). On the entropy of continuous probability distributions., IEEE Trans. Inf. Theory, 24, 120–122.

Mathematical Reviews (MathSciNet): MR479686

Digital Object Identifier: doi:10.1109/TIT.1978.1055832

[35] Louani, D. (2005). Uniform, L₁-distance large deviations in nonparametric density estimation. Test, 14(1), 75–98.

Mathematical Reviews (MathSciNet): MR2203423

Zentralblatt MATH: 1069.62032

Digital Object Identifier: doi:10.1007/BF02595398

[36] Parzen, E. (1962). On estimation of a probability density function and mode., Ann. Math. Statist., 33, 1065–1076.

Mathematical Reviews (MathSciNet): MR143282

Zentralblatt MATH: 0116.11302

Digital Object Identifier: doi:10.1214/aoms/1177704472

Project Euclid: euclid.aoms/1177704472

[37] Prakasa Rao, B. L. S. (1983)., Nonparametric functional estimation. Probability and Mathematical Statistics. Academic Press Inc. [Harcourt Brace Jovanovich Publishers], New York.

Mathematical Reviews (MathSciNet): MR740865

[38] Prescott, P. (1976). On a test for normality based on sample entropy., J. R. Stat. Soc., Ser. B, 38, 254–256.

Mathematical Reviews (MathSciNet): MR420958

[39] Rényi, A. (1959). On the dimension and entropy of probability distributions., Acta Math. Acad. Sci. Hungar., 10, 193–215 (unbound insert).

Mathematical Reviews (MathSciNet): MR107575

Digital Object Identifier: doi:10.1007/BF02063299

[40] Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function., Ann. Math. Statist., 27, 832–837.

Mathematical Reviews (MathSciNet): MR79873

Zentralblatt MATH: 0073.14602

Digital Object Identifier: doi:10.1214/aoms/1177728190

Project Euclid: euclid.aoms/1177728190

[41] Scott, D. W. (1992)., Multivariate density estimation. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. John Wiley & Sons Inc., New York. Theory, practice, and visualization, A Wiley-Interscience Publication.

Mathematical Reviews (MathSciNet): MR1191168

[42] Shannon, C. E. (1948). A mathematical theory of communication., Bell System Tech. J., 27, 379–423, 623–656.

Mathematical Reviews (MathSciNet): MR26286

[43] Song, K.-S. (2000). Limit theorems for nonparametric sample entropy estimators., Statist. Probab. Lett., 49(1), 9–18.

Mathematical Reviews (MathSciNet): MR1789659

[44] Talagrand, M. (1994). Sharper bounds for Gaussian and empirical processes., Ann. Probab., 22(1), 28–76.

Mathematical Reviews (MathSciNet): MR1258865

Zentralblatt MATH: 0798.60051

Digital Object Identifier: doi:10.1214/aop/1176988847

Project Euclid: euclid.aop/1176988847

[45] Vasicek, O. (1976). A test for normality based on sample entropy., J. Roy. Statist. Soc. Ser. B, 38(1), 54–59.

Mathematical Reviews (MathSciNet): MR420958

[46] Viallon, V. (2006)., Processus empiriques, estimation non paramétrique et données censurées. Ph.D. thesis, Université Paris 6.

[47] Vituškin, A. G. (1955)., O mnogomernyh variaciyah. Gosudarstv. Izdat. Tehn.-Teor. Lit., Moscow.

Mathematical Reviews (MathSciNet): MR75267

previous :: next

Electronic Journal of Statistics

IMS and IMS Related Journals