Translator Disclaimer
October 2019 Exact lower bounds for the agnostic probably-approximately-correct (PAC) machine learning model
Aryeh Kontorovich, Iosif Pinelis
Ann. Statist. 47(5): 2822-2854 (October 2019). DOI: 10.1214/18-AOS1766


We provide an exact nonasymptotic lower bound on the minimax expected excess risk (EER) in the agnostic probably-approximately-correct (PAC) machine learning classification model and identify minimax learning algorithms as certain maximally symmetric and minimally randomized “voting” procedures. Based on this result, an exact asymptotic lower bound on the minimax EER is provided. This bound is of the simple form $c_{\infty}/\sqrt{\nu}$ as $\nu\to\infty$, where $c_{\infty}=0.16997\dots$ is a universal constant, $\nu=m/d$, $m$ is the size of the training sample and $d$ is the Vapnik–Chervonenkis dimension of the hypothesis class. It is shown that the differences between these asymptotic and nonasymptotic bounds, as well as the differences between these two bounds and the maximum EER of any learning algorithms that minimize the empirical risk, are asymptotically negligible, and all these differences are due to ties in the mentioned “voting” procedures. A few easy to compute nonasymptotic lower bounds on the minimax EER are also obtained, which are shown to be close to the exact asymptotic lower bound $c_{\infty}/\sqrt{\nu}$ even for rather small values of the ratio $\nu=m/d$. As an application of these results, we substantially improve existing lower bounds on the tail probability of the excess risk. Among the tools used are Bayes estimation and apparently new identities and inequalities for binomial distributions.


Download Citation

Aryeh Kontorovich. Iosif Pinelis. "Exact lower bounds for the agnostic probably-approximately-correct (PAC) machine learning model." Ann. Statist. 47 (5) 2822 - 2854, October 2019.


Received: 1 June 2016; Revised: 1 December 2017; Published: October 2019
First available in Project Euclid: 3 August 2019

zbMATH: 07114930
MathSciNet: MR3988774
Digital Object Identifier: 10.1214/18-AOS1766

Primary: 62C10, 62C12, 62C20, 62G20, 62H30, 68T05
Secondary: 60C05, 62C20, 62G10, 91A35

Rights: Copyright © 2019 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.47 • No. 5 • October 2019
Back to Top