Open Access
December 2006 Local Rademacher complexities and oracle inequalities in risk minimization
Vladimir Koltchinskii
Ann. Statist. 34(6): 2593-2656 (December 2006). DOI: 10.1214/009053606000001019

Abstract

Let ℱ be a class of measurable functions f:S↦[0, 1] defined on a probability space (S, $\mathcal{A}$, P). Given a sample (X1, …, Xn) of i.i.d. random variables taking values in S with common distribution P, let Pn denote the empirical measure based on (X1, …, Xn). We study an empirical risk minimization problem Pnf→min , f∈ℱ. Given a solution n of this problem, the goal is to obtain very general upper bounds on its excess risk $$\mathcal{E}_{P}(\hat{f}_{n}):=P\hat{f}_{n}-\inf_{f\in \mathcal{F}}Pf,$$ expressed in terms of relevant geometric parameters of the class ℱ. Using concentration inequalities and other empirical processes tools, we obtain both distribution-dependent and data-dependent upper bounds on the excess risk that are of asymptotically correct order in many examples. The bounds involve localized sup-norms of empirical and Rademacher processes indexed by functions from the class. We use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification.

Citation

Download Citation

Vladimir Koltchinskii. "Local Rademacher complexities and oracle inequalities in risk minimization." Ann. Statist. 34 (6) 2593 - 2656, December 2006. https://doi.org/10.1214/009053606000001019

Information

Published: December 2006
First available in Project Euclid: 23 May 2007

zbMATH: 1118.62065
MathSciNet: MR2329442
Digital Object Identifier: 10.1214/009053606000001019

Subjects:
Primary: 60B99 , 62H30 , 68Q32
Secondary: 62G08 , 68T05 , 68T10

Keywords: ‎classification‎ , Concentration inequalities , empirical risk minimization , Model selection , Oracle inequalities , Rademacher complexities

Rights: Copyright © 2006 Institute of Mathematical Statistics

Vol.34 • No. 6 • December 2006
Back to Top