EXPECTED RELATIVE ENTROPY BETWEEN A FINITE DISTRIBUTION AND ITS EMPIRICAL DISTRIBUTION

Abstract

The expected relative entropy (or the expected divergence) between finite probability distribution $Q$ on $\{1,2,\dots ,\ell \}$ and its empirical one obtained from the sample of size $n$ drawn from $Q$ is computed and is found to be given asymptotically by $(\ell –1)(\log e)/2n$ which is independent of $Q$. A method to compute the entropy of the binomial distribution more accurately than before is also given.