Sparse high-dimensional linear regression. Estimating squared error and a phase transition

Abstract

We consider a sparse high-dimensional regression model where the goal is to recover a k-sparse unknown binary vector ${\mathit{\beta }}^{\ast }$ from n noisy linear observations of the form $\mathit{Y}=\mathit{X}{\mathit{\beta }}^{\ast }+\mathit{W}\in {\mathbb{R}}^{\mathit{n}}$ where $\mathit{X}\in {\mathbb{R}}^{\mathit{n}\times \mathit{p}}$ has i.i.d. $\mathit{N}(0,1)$ entries and $\mathit{W}\in {\mathbb{R}}^{\mathit{n}}$ has i.i.d. $\mathit{N}(0,{\mathit{\sigma }}^2)$ entries. In the high signal-to-noise ratio regime and sublinear sparsity regime, while the order of the sample size needed to recover the unknown vector information-theoretically is known to be ${\mathit{n}}^{\ast }:=2\mathit{k}log\mathit{p}/log(\mathit{k}/{\mathit{\sigma }}^2+1)$, no polynomial-time algorithm is known to succeed unless $\mathit{n}>{\mathit{n}}_{\mathrm{alg}}:=(2\mathit{k}+{\mathit{\sigma }}^2)log\mathit{p}$.

In this work, we offer a series of results investigating multiple computational and statistical aspects of the recovery task in the regime $\mathit{n}\in [{\mathit{n}}^{\ast },{\mathit{n}}_{\mathrm{alg}}]$. First, we establish a novel information-theoretic property of the MLE of the problem happening around $\mathit{n}={\mathit{n}}^{\ast }$ samples, which we coin as an “all-or-nothing behavior”: when $\mathit{n}>{\mathit{n}}^{\ast }$ it recovers almost perfectly the support of ${\mathit{\beta }}^{\ast }$, while if $\mathit{n}<{\mathit{n}}^{\ast }$ it fails to recover any fraction of it correctly. Second, at an attempt to understand the computational hardness in the regime $\mathit{n}\in [{\mathit{n}}^{\ast },{\mathit{n}}_{\mathrm{alg}}]$, we prove that at order ${\mathit{n}}_{\mathrm{alg}}$ samples there is an Overlap Gap Property (OGP) phase transition occurring at the landscape of the MLE: for constants $\mathit{c},\mathit{C}>0$ when $\mathit{n}<\mathit{c}{\mathit{n}}_{\mathrm{alg}}$ OGP appears in the landscape of MLE while if $\mathit{n}>\mathit{C}{\mathit{n}}_{\mathrm{alg}}$ OGP disappears. OGP is a geometric “disconnectivity” property, which initially appeared in the theory of spin glasses and is known to suggest algorithmic hardness when it occurs. Finally, using certain technical results obtained to establish the OGP phase transition, we additionally establish various novel positive and negative algorithmic results for the recovery task of interest, including the failure of LASSO with access to $\mathit{n}<\mathit{c}{\mathit{n}}_{\mathrm{alg}}$ samples and the success of a simple local search method with access to $\mathit{n}>\mathit{C}{\mathit{n}}_{\mathrm{alg}}$ samples.