The Annals of Statistics
- Ann. Statist.
- Volume 30, Number 1 (2002), 51-73.
On weak base Hypotheses and their implications for boosting regression and classification
When studying the training error and the prediction error for boosting, it is often assumed that the hypotheses returned by the base learner are weakly accurate, or are able to beat a random guesser by a certain amount of difference. It has been an open question how much this difference can be, whether it will eventually disappear in the boosting process or be bounded by a positive amount. This question is crucial for the behavior of both the training error and the prediction error. In this paper we study this problem and show affirmatively that the amount of improvement over the random guesser will be at least a positive amount for almost all possible sample realizations and for most of the commonly used base hypotheses. This has a number of implications for the prediction error, including, for example, that boosting forever may not be good and regularization may be necessary. The problem is studied by first considering an analog of AdaBoost in regression, where we study similar properties and find that, for good performance, one cannot hope to avoid regularization by just adopting the boosting device to regression.
Ann. Statist., Volume 30, Number 1 (2002), 51-73.
First available in Project Euclid: 5 March 2002
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Jiang, Wenxin. On weak base Hypotheses and their implications for boosting regression and classification. Ann. Statist. 30 (2002), no. 1, 51--73. doi:10.1214/aos/1015362184. https://projecteuclid.org/euclid.aos/1015362184