Open Access
Translator Disclaimer
December 2018 Overcoming the limitations of phase transition by higher order analysis of regularization techniques
Haolei Weng, Arian Maleki, Le Zheng
Ann. Statist. 46(6A): 3099-3129 (December 2018). DOI: 10.1214/17-AOS1651

Abstract

We study the problem of estimating a sparse vector $\beta\in\mathbb{R}^{p}$ from the response variables $y=X\beta+w$, where $w\sim N(0,\sigma_{w}^{2}I_{n\times n})$, under the following high-dimensional asymptotic regime: given a fixed number $\delta$, $p\rightarrow\infty$, while $n/p\rightarrow\delta$. We consider the popular class of $\ell_{q}$-regularized least squares (LQLS), a.k.a. bridge estimators, given by the optimization problem \begin{equation*}\hat{\beta}(\lambda,q)\in\arg\min_{\beta}\frac{1}{2}\|y-X\beta\|_{2}^{2}+\lambda\|\beta\|_{q}^{q},\end{equation*} and characterize the almost sure limit of $\frac{1}{p}\|\hat{\beta}(\lambda,q)-\beta\|_{2}^{2}$, and call it asymptotic mean square error (AMSE). The expression we derive for this limit does not have explicit forms, and hence is not useful in comparing LQLS for different values of $q$, or providing information in evaluating the effect of $\delta$ or sparsity level of $\beta$. To simplify the expression, researchers have considered the ideal “error-free” regime, that is, $w=0$, and have characterized the values of $\delta$ for which AMSE is zero. This is known as the phase transition analysis.

In this paper, we first perform the phase transition analysis of LQLS. Our results reveal some of the limitations and misleading features of the phase transition analysis. To overcome these limitations, we propose the small error analysis of LQLS. Our new analysis framework not only sheds light on the results of the phase transition analysis, but also describes when phase transition analysis is reliable, and presents a more accurate comparison among different regularizers.

Citation

Download Citation

Haolei Weng. Arian Maleki. Le Zheng. "Overcoming the limitations of phase transition by higher order analysis of regularization techniques." Ann. Statist. 46 (6A) 3099 - 3129, December 2018. https://doi.org/10.1214/17-AOS1651

Information

Received: 1 March 2017; Revised: 1 September 2017; Published: December 2018
First available in Project Euclid: 7 September 2018

zbMATH: 06968610
MathSciNet: MR3851766
Digital Object Identifier: 10.1214/17-AOS1651

Subjects:
Primary: 62J05 , 62J07

Keywords: asymptotic mean square error , Bridge regression , comparison of estimators , optimal tuning , phase transition , second-order term , small error regime

Rights: Copyright © 2018 Institute of Mathematical Statistics

JOURNAL ARTICLE
31 PAGES


SHARE
Vol.46 • No. 6A • December 2018
Back to Top