## The Annals of Statistics

- Ann. Statist.
- Volume 45, Number 6 (2017), 2328-2364.

### A new perspective on boosting in linear regression via subgradient optimization and relatives

Robert M. Freund, Paul Grigas, and Rahul Mazumder

#### Abstract

We analyze boosting algorithms [*Ann. Statist.* **29** (2001) 1189–1232; *Ann. Statist.* **28** (2000) 337–407; *Ann. Statist.* **32** (2004) 407–499] in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm ($\text{FS}_{\varepsilon}$) and least squares boosting [LS-BOOST$(\varepsilon)$], can be viewed as subgradient descent to minimize the loss function defined as the maximum absolute correlation between the features and residuals. We also propose a minor modification of $\text{FS}_{\varepsilon}$ that yields an algorithm for the LASSO, and that may be easily extended to an algorithm that computes the LASSO path for different values of the regularization parameter. Furthermore, we show that these new algorithms for the LASSO may also be interpreted as the same master algorithm (subgradient descent), applied to a regularized version of the maximum absolute correlation loss function. We derive novel, comprehensive computational guarantees for several boosting algorithms in linear regression (including LS-BOOST$(\varepsilon)$ and $\text{FS}_{\varepsilon}$) by using techniques of first-order methods in convex optimization. Our computational guarantees inform us about the statistical properties of boosting algorithms. In particular, they provide, for the first time, a precise theoretical description of the amount of data-fidelity and regularization imparted by running a boosting algorithm with a prespecified learning rate for a fixed but arbitrary number of iterations, for *any* dataset.

#### Article information

**Source**

Ann. Statist., Volume 45, Number 6 (2017), 2328-2364.

**Dates**

Received: December 2015

Revised: August 2016

First available in Project Euclid: 15 December 2017

**Permanent link to this document**

https://projecteuclid.org/euclid.aos/1513328575

**Digital Object Identifier**

doi:10.1214/16-AOS1505

**Mathematical Reviews number (MathSciNet)**

MR3737894

**Zentralblatt MATH identifier**

06838135

**Subjects**

Primary: 62J05: Linear regression 62J07k

Secondary: 90C25: Convex programming

**Keywords**

Boosting forward stagewise regression linear regression bias-variance tradeoff shrinkage convex optimization computational guarantees

#### Citation

M. Freund, Robert; Grigas, Paul; Mazumder, Rahul. A new perspective on boosting in linear regression via subgradient optimization and relatives. Ann. Statist. 45 (2017), no. 6, 2328--2364. doi:10.1214/16-AOS1505. https://projecteuclid.org/euclid.aos/1513328575

#### Supplemental materials

- Supplement to “A new perspective on boosting in linear regression via subgradient optimization and relatives”. Additional proofs, technical details, figures and tables are provided in the Supplementary Section.Digital Object Identifier: doi:10.1214/16-AOS1505SUPPSupplemental files are immediately available to subscribers. Non-subscribers gain access to supplemental files with the purchase of the article.