Translator Disclaimer
November 2019 On Lasso refitting strategies
Evgenii Chzhen, Mohamed Hebiri, Joseph Salmon
Bernoulli 25(4A): 3175-3200 (November 2019). DOI: 10.3150/18-BEJ1085

Abstract

A well-known drawback of $\ell_{1}$-penalized estimators is the systematic shrinkage of the large coefficients towards zero. A simple remedy is to treat Lasso as a model-selection procedure and to perform a second refitting step on the selected support. In this work, we formalize the notion of refitting and provide oracle bounds for arbitrary refitting procedures of the Lasso solution. One of the most widely used refitting techniques which is based on Least-Squares may bring a problem of interpretability, since the signs of the refitted estimator might be flipped with respect to the original estimator. This problem arises from the fact that the Least-Squares refitting considers only the support of the Lasso solution, avoiding any information about signs or amplitudes. To this end, we define a sign consistent refitting as an arbitrary refitting procedure, preserving the signs of the first step Lasso solution and provide Oracle inequalities for such estimators. Finally, we consider special refitting strategies: Bregman Lasso and Boosted Lasso. Bregman Lasso has a fruitful property to converge to the Sign-Least-Squares refitting (Least-Squares with sign constraints), which provides with greater interpretability. We additionally study the Bregman Lasso refitting in the case of orthogonal design, providing with simple intuition behind the proposed method. Boosted Lasso, in contrast, considers information about magnitudes of the first Lasso step and allows to develop better oracle rates for prediction. Finally, we conduct an extensive numerical study to show advantages of one approach over others in different synthetic and semi-real scenarios.

Citation

Download Citation

Evgenii Chzhen. Mohamed Hebiri. Joseph Salmon. "On Lasso refitting strategies." Bernoulli 25 (4A) 3175 - 3200, November 2019. https://doi.org/10.3150/18-BEJ1085

Information

Received: 1 July 2017; Revised: 1 June 2018; Published: November 2019
First available in Project Euclid: 13 September 2019

zbMATH: 07110125
MathSciNet: MR4003578
Digital Object Identifier: 10.3150/18-BEJ1085

Rights: Copyright © 2019 Bernoulli Society for Mathematical Statistics and Probability

JOURNAL ARTICLE
26 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.25 • No. 4A • November 2019
Back to Top