February 2023 Suboptimality of constrained least squares and improvements via non-linear predictors
Tomas Vaškevičius, Nikita Zhivotovskiy
Author Affiliations +
Bernoulli 29(1): 473-495 (February 2023). DOI: 10.3150/22-BEJ1465

Abstract

We study the problem of predicting as well as the best linear predictor in a bounded Euclidean ball with respect to the squared loss. When only boundedness of the data generating distribution is assumed, we establish that the least squares estimator constrained to a bounded Euclidean ball does not attain the classical O(dn) excess risk rate, where d is the dimension of the covariates and n is the number of samples. In particular, we construct a bounded distribution such that the constrained least squares estimator incurs an excess risk of order Ω(d32n) hence refuting a recent conjecture of Ohad Shamir [JMLR 2015]. In contrast, we observe that non-linear predictors can achieve the optimal rate O(dn) with no assumptions on the distribution of the covariates. We discuss additional distributional assumptions sufficient to guarantee an O(dn) excess risk rate for the least squares estimator. Among them are certain moment equivalence assumptions often used in the robust statistics literature. While such assumptions are central in the analysis of unbounded and heavy-tailed settings, our work indicates that in some cases, they also rule out unfavorable bounded distributions.

Funding Statement

Tomas Vaškevičius is supported by the EPSRC and MRC through the OxWaSP CDT programme (EP/L016710/1). This work was conducted when Nikita Zhivotovskiy was at Google Research, Zürich.

Acknowledgements

We are indebted to Shahar Mendelson for fruitful discussions and valuable feedback: in particular, for suggesting us the technique to analyze the quadratic process in Theorem 2.1 and for motivating us to work on the lower bounds. We are also grateful to Manfred Warmuth for providing a reference to the predictor that removes the excess logarithmic factor appearing in one of our results. Finally, we thank Jaouad Mourtada for many related discussions.

Citation

Download Citation

Tomas Vaškevičius. Nikita Zhivotovskiy. "Suboptimality of constrained least squares and improvements via non-linear predictors." Bernoulli 29 (1) 473 - 495, February 2023. https://doi.org/10.3150/22-BEJ1465

Information

Received: 1 March 2021; Published: February 2023
First available in Project Euclid: 13 October 2022

MathSciNet: MR4497255
Digital Object Identifier: 10.3150/22-BEJ1465

Keywords: average stability , Constrained least squares , Empirical processes , Ridge regression , Vovk-Azoury-Warmuth forecaster

Vol.29 • No. 1 • February 2023
Back to Top