Surprises in high-dimensional ridgeless least squares interpolation

Abstract

Interpolators—estimators that achieve zero training error—have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum ${\ell }_2$ norm (“ridgeless”) interpolation least squares regression, focusing on the high-dimensional regime in which the number of unknown parameters p is of the same order as the number of samples n. We consider two different models for the feature distribution: a linear model, where the feature vectors ${\mathit{x}}_{\mathit{i}}\in {\mathbb{R}}^{\mathit{p}}$ are obtained by applying a linear transform to a vector of i.i.d. entries, ${\mathit{x}}_{\mathit{i}}={\mathrm{\Sigma }}^{1/2}{\mathit{z}}_{\mathit{i}}$ (with ${\mathit{z}}_{\mathit{i}}\in {\mathbb{R}}^{\mathit{p}}$); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, ${\mathit{x}}_{\mathit{i}}=\mathit{\phi }(\mathit{W}{\mathit{z}}_{\mathit{i}})$ (with ${\mathit{z}}_{\mathit{i}}\in {\mathbb{R}}^{\mathit{d}}$, $\mathit{W}\in {\mathbb{R}}^{\mathit{p}\times \mathit{d}}$ a matrix of i.i.d. entries, and φ an activation function acting componentwise on $\mathit{W}{\mathit{z}}_{\mathit{i}}$). We recover—in a precise quantitative way—several phenomena that have been observed in large-scale neural networks and kernel machines, including the “double descent” behavior of the prediction risk, and the potential benefits of overparametrization.