Open Access
Translator Disclaimer
December 2018 The landscape of empirical risk for nonconvex losses
Song Mei, Yu Bai, Andrea Montanari
Ann. Statist. 46(6A): 2747-2774 (December 2018). DOI: 10.1214/17-AOS1637

Abstract

Most high-dimensional estimation methods propose to minimize a cost function (empirical risk) that is a sum of losses associated to each data point (each example). In this paper, we focus on the case of nonconvex losses. Classical empirical process theory implies uniform convergence of the empirical (or sample) risk to the population risk. While under additional assumptions, uniform convergence implies consistency of the resulting M-estimator, it does not ensure that the latter can be computed efficiently.

In order to capture the complexity of computing M-estimators, we study the landscape of the empirical risk, namely its stationary points and their properties. We establish uniform convergence of the gradient and Hessian of the empirical risk to their population counterparts, as soon as the number of samples becomes larger than the number of unknown parameters (modulo logarithmic factors). Consequently, good properties of the population risk can be carried to the empirical risk, and we are able to establish one-to-one correspondence of their stationary points. We demonstrate that in several problems such as nonconvex binary classification, robust regression and Gaussian mixture model, this result implies a complete characterization of the landscape of the empirical risk, and of the convergence properties of descent algorithms.

We extend our analysis to the very high-dimensional setting in which the number of parameters exceeds the number of samples, and provides a characterization of the empirical risk landscape under a nearly information-theoretically minimal condition. Namely, if the number of samples exceeds the sparsity of the parameters vector (modulo logarithmic factors), then a suitable uniform convergence result holds. We apply this result to nonconvex binary classification and robust regression in very high-dimension.

Citation

Download Citation

Song Mei. Yu Bai. Andrea Montanari. "The landscape of empirical risk for nonconvex losses." Ann. Statist. 46 (6A) 2747 - 2774, December 2018. https://doi.org/10.1214/17-AOS1637

Information

Received: 1 January 2017; Revised: 1 August 2017; Published: December 2018
First available in Project Euclid: 7 September 2018

zbMATH: 06968598
MathSciNet: MR3851754
Digital Object Identifier: 10.1214/17-AOS1637

Subjects:
Primary: 62F10 , 62J02
Secondary: 62H30

Keywords: empirical risk minimization , landscape , nonconvex optimization , Uniform convergence

Rights: Copyright © 2018 Institute of Mathematical Statistics

JOURNAL ARTICLE
28 PAGES


SHARE
Vol.46 • No. 6A • December 2018
Back to Top