Open Access
August 2019 Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions
Pierre Alquier, Vincent Cottet, Guillaume Lecué
Ann. Statist. 47(4): 2117-2144 (August 2019). DOI: 10.1214/18-AOS1742


We obtain estimation error rates and sharp oracle inequalities for regularization procedures of the form \begin{equation*}\hat{f}\in\mathop{\operatorname{argmin}}_{f\in F}\Bigg(\frac{1}{N}\sum_{i=1}^{N}\ell_{f}(X_{i},Y_{i})+\lambda \Vert f\Vert \Bigg)\end{equation*} when $\Vert \cdot \Vert $ is any norm, $F$ is a convex class of functions and $\ell$ is a Lipschitz loss function satisfying a Bernstein condition over $F$. We explore both the bounded and sub-Gaussian stochastic frameworks for the distribution of the $f(X_{i})$’s, with no assumption on the distribution of the $Y_{i}$’s. The general results rely on two main objects: a complexity function and a sparsity equation, that depend on the specific setting in hand (loss $\ell$ and norm $\Vert \cdot \Vert $).

As a proof of concept, we obtain minimax rates of convergence in the following problems: (1) matrix completion with any Lipschitz loss function, including the hinge and logistic loss for the so-called 1-bit matrix completion instance of the problem, and quantile losses for the general case, which enables to estimate any quantile on the entries of the matrix; (2) logistic LASSO and variants such as the logistic SLOPE, and also shape constrained logistic regression; (3) kernel methods, where the loss is the hinge loss, and the regularization function is the RKHS norm.


Download Citation

Pierre Alquier. Vincent Cottet. Guillaume Lecué. "Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions." Ann. Statist. 47 (4) 2117 - 2144, August 2019.


Received: 1 January 2018; Revised: 1 June 2018; Published: August 2019
First available in Project Euclid: 21 May 2019

zbMATH: 07082281
MathSciNet: MR3953446
Digital Object Identifier: 10.1214/18-AOS1742

Primary: 60K35 , 62G08
Secondary: 62C20 , 62G05 , 62G20

Keywords: Empirical processes , High-dimensional statistics

Rights: Copyright © 2019 Institute of Mathematical Statistics

Vol.47 • No. 4 • August 2019
Back to Top