Open Access
October 2001 Greedy function approximation: A gradient boosting machine.
Jerome H. Friedman
Ann. Statist. 29(5): 1189-1232 (October 2001). DOI: 10.1214/aos/1013203451


Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest-descent minimization. A general gradient descent “boosting” paradigm is developed for additive expansions based on any fitting criterion.Specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such “TreeBoost” models are presented. Gradient boosting of regression trees produces competitive, highly robust, interpretable procedures for both regression and classification, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire and Friedman, Hastie and Tibshirani are discussed.


Download Citation

Jerome H. Friedman. "Greedy function approximation: A gradient boosting machine.." Ann. Statist. 29 (5) 1189 - 1232, October 2001.


Published: October 2001
First available in Project Euclid: 8 February 2002

zbMATH: 1043.62034
MathSciNet: MR1873328
Digital Object Identifier: 10.1214/aos/1013203451

Primary: 62-02 , 62-07 , 62-08 , 62G08 , 62H30 , 68T10

Keywords: boosting , decision trees , Function estimation , robust nonparametric regression

Rights: Copyright © 2001 Institute of Mathematical Statistics

Vol.29 • No. 5 • October 2001
Back to Top