Density estimation in high and ultra high dimensions, regularization, and the L1 asymptotics

Abstract

This article gives a theoretical treatment of the asymptotics of the L₁ error of a model-based estimate of a density f(x|θ) on a finite dimensional Euclidean space ℛ^k.

The dimension p of the parameter vector θ is considered arbitrary but fixed in Section 2. Two theorems in Section 2 lay out the weak limits of a suitably scaled L₁ error, with a general estimating sequence θ̂ and a general family of smooth densities f(x|θ) dominated by some σ-finite measure, the discrete case included. We show that the L₁ error converges at the coarsest rate corresponding to the different coordinates of the parameter vector θ. Four applications are detailed in Section 3, a special one being a new confidence interval for a Poisson mean.

Section 4 considers the high and the ultra high dimensional case, where p grows with n. The exact critical growth rate for p when maximum likelihood starts to falter is derived. Maximum likelihood is shown to exhibit a trichotomy of behavior; the desired behavior below the threshold, problematic behavior at the threshold, and disastrous behavior above the threshold.

It is then shown that regularization, if coupled with the right amount of sparsity, can return consistent density estimation, even at the best possible n^−1/2 rate. We give a complete description of the limiting behavior of the regularized density estimate under different sparsity conditions. Section 4 is specialized to the Gaussian case due to its special importance and well known links to function estimation.