Annals of Statistics
- Ann. Statist.
- Volume 48, Number 2 (2020), 906-931.
Robust machine learning by median-of-means: Theory and practice
Median-of-means (MOM) based procedures have been recently introduced in learning theory (Lugosi and Mendelson (2019); Lecué and Lerasle (2017)). These estimators outperform classical least-squares estimators when data are heavy-tailed and/or are corrupted. None of these procedures can be implemented, which is the major issue of current MOM procedures (Ann. Statist. 47 (2019) 783–794).
In this paper, we introduce minmax MOM estimators and show that they achieve the same sub-Gaussian deviation bounds as the alternatives (Lugosi and Mendelson (2019); Lecué and Lerasle (2017)), both in small and high-dimensional statistics. In particular, these estimators are efficient under moments assumptions on data that may have been corrupted by a few outliers.
Besides these theoretical guarantees, the definition of minmax MOM estimators suggests simple and systematic modifications of standard algorithms used to approximate least-squares estimators and their regularized versions. As a proof of concept, we perform an extensive simulation study of these algorithms for robust versions of the LASSO.
Ann. Statist., Volume 48, Number 2 (2020), 906-931.
Received: July 2018
Revised: February 2019
First available in Project Euclid: 26 May 2020
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Primary: 60K35: Interacting random processes; statistical mechanics type models; percolation theory [See also 82B43, 82C43] 62G08: Nonparametric regression
Secondary: 62C20: Minimax procedures 62G05: Estimation 62G20: Asymptotic properties
Lecué, Guillaume; Lerasle, Matthieu. Robust machine learning by median-of-means: Theory and practice. Ann. Statist. 48 (2020), no. 2, 906--931. doi:10.1214/19-AOS1828. https://projecteuclid.org/euclid.aos/1590480039
- Supplementary material to “Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions”. Section 6 gives the proof of the main results. These main results focus on the regularized version of the MOM estimates of the increments presented in this Introduction that are well suited for high-dimensional learning frameworks. We complete these results in Section 7, providing results for the basic estimators without regularization in small dimension. Finally, Section 8 provides minimax optimality results for our procedures.