Open Access
Translator Disclaimer
August 2005 Boosting with early stopping: Convergence and consistency
Tong Zhang, Bin Yu
Ann. Statist. 33(4): 1538-1579 (August 2005). DOI: 10.1214/009053605000000255


Boosting is one of the most significant advances in machine learning for classification and regression. In its original and computationally flexible version, boosting seeks to minimize empirically a loss function in a greedy fashion. The resulting estimator takes an additive function form and is built iteratively by applying a base estimator (or learner) to updated samples depending on the previous iterations. An unusual regularization technique, early stopping, is employed based on CV or a test set.

This paper studies numerical convergence, consistency and statistical rates of convergence of boosting with early stopping, when it is carried out over the linear span of a family of basis functions. For general loss functions, we prove the convergence of boosting’s greedy optimization to the infinimum of the loss function over the linear span. Using the numerical convergence result, we find early-stopping strategies under which boosting is shown to be consistent based on i.i.d. samples, and we obtain bounds on the rates of convergence for boosting estimators. Simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting.

As a side product, these results also reveal the importance of restricting the greedy search step-sizes, as known in practice through the work of Friedman and others. Moreover, our results lead to a rigorous proof that for a linearly separable problem, AdaBoost with ɛ→0 step-size becomes an L1-margin maximizer when left to run to convergence.


Download Citation

Tong Zhang. Bin Yu. "Boosting with early stopping: Convergence and consistency." Ann. Statist. 33 (4) 1538 - 1579, August 2005.


Published: August 2005
First available in Project Euclid: 5 August 2005

zbMATH: 1078.62038
MathSciNet: MR2166555
Digital Object Identifier: 10.1214/009053605000000255

Primary: 62G05 , 62G08

Keywords: boosting , consistency , early stopping , greedy optimization , matching pursuit

Rights: Copyright © 2005 Institute of Mathematical Statistics


Vol.33 • No. 4 • August 2005
Back to Top