Open Access
April 2012 Characterizing $L_{2}$Boosting
John Ehrlinger, Hemant Ishwaran
Ann. Statist. 40(2): 1074-1101 (April 2012). DOI: 10.1214/12-AOS997


We consider $L_{2}$Boosting, a special case of Friedman’s generic boosting algorithm applied to linear regression under $L_{2}$-loss. We study $L_{2}$Boosting for an arbitrary regularization parameter and derive an exact closed form expression for the number of steps taken along a fixed coordinate direction. This relationship is used to describe $L_{2}$Boosting’s solution path, to describe new tools for studying its path, and to characterize some of the algorithm’s unique properties, including active set cycling, a property where the algorithm spends lengthy periods of time cycling between the same coordinates when the regularization parameter is arbitrarily small. Our fixed descent analysis also reveals a repressible condition that limits the effectiveness of $L_{2}$Boosting in correlated problems by preventing desirable variables from entering the solution path. As a simple remedy, a data augmentation method similar to that used for the elastic net is used to introduce $L_{2}$-penalization and is shown, in combination with decorrelation, to reverse the repressible condition and circumvents $L_{2}$Boosting’s deficiencies in correlated problems. In itself, this presents a new explanation for why the elastic net is successful in correlated problems and why methods like LAR and lasso can perform poorly in such settings.


Download Citation

John Ehrlinger. Hemant Ishwaran. "Characterizing $L_{2}$Boosting." Ann. Statist. 40 (2) 1074 - 1101, April 2012.


Published: April 2012
First available in Project Euclid: 18 July 2012

zbMATH: 1274.62420
MathSciNet: MR2985944
Digital Object Identifier: 10.1214/12-AOS997

Primary: 62J05
Secondary: 62J99

Keywords: Critical direction , gradient-correlation , regularization , repressibility , solution path

Rights: Copyright © 2012 Institute of Mathematical Statistics

Vol.40 • No. 2 • April 2012
Back to Top