Sparse linear regression methods generally have a free hyperparameter which controls the amount of sparsity, and is subject to a bias-variance tradeoff. This article considers the use of Aggregated hold-out to aggregate over values of this hyperparameter, in the context of linear regression with the Huber loss function. Aggregated hold-out (Agghoo) is a procedure which averages estimators selected by hold-out (cross-validation with a single split). In the theoretical part of the article, it is proved that Agghoo satisfies a non-asymptotic oracle inequality when it is applied to sparse estimators which are parametrized by their zero-norm. In particular, this includes a variant of the Lasso introduced by Zou, Hastié and Tibshirani . Simulations are used to compare Agghoo with cross-validation. They show that Agghoo performs better than CV when the intrinsic dimension is high and when there are confounders correlated with the predictive covariates.
While finishing the writing of this article, the author (Guillaume Maillard) has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 811017.
"Aggregated hold out for sparse linear regression with a robust loss function." Electron. J. Statist. 16 (1) 935 - 997, 2022. https://doi.org/10.1214/21-EJS1952