Abstract
The Lasso is a popular regression method for high-dimensional problems in which the number of parameters , is larger than the number n of samples: . A useful heuristics relates the statistical properties of the Lasso estimator to that of a simple soft-thresholding denoiser, in a denoising problem in which the parameters are observed in Gaussian noise, with a carefully tuned variance. Earlier work confirmed this picture in the limit , pointwise in the parameters θ and in the value of the regularization parameter.
Here, we consider a standard random design model and prove exponential concentration of its empirical distribution around the prediction provided by the Gaussian denoising model. Crucially, our results are uniform with respect to θ belonging to balls, , and with respect to the regularization parameter. This allows us to derive sharp results for the performances of various data-driven procedures to tune the regularization.
Our proofs make use of Gaussian comparison inequalities, and in particular of a version of Gordon’s minimax theorem developed by Thrampoulidis, Oymak and Hassibi, which controls the optimum value of the Lasso optimization problem. Crucially, we prove a stability property of the minimizer in Wasserstein distance that allows one to characterize properties of the minimizer itself.
Funding Statement
This work was partially supported by grants NSF DMS-1613091, NSF CCF-1714305 and NSF IIS-1741162 and ONR N00014-18-1-2729.
Citation
Léo Miolane. Andrea Montanari. "The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning." Ann. Statist. 49 (4) 2313 - 2335, August 2021. https://doi.org/10.1214/20-AOS2038
Information