Open Access
September 2013 The geometry of least squares in the 21st century
Jonathan Taylor
Bernoulli 19(4): 1449-1464 (September 2013). DOI: 10.3150/12-BEJSP15

Abstract

It has been over 200 years since Gauss’s and Legendre’s famous priority dispute on who discovered the method of least squares. Nevertheless, we argue that the normal equations are still relevant in many facets of modern statistics, particularly in the domain of high-dimensional inference. Even today, we are still learning new things about the law of large numbers, first described in Bernoulli’s Ars Conjectandi 300 years ago, as it applies to high dimensional inference.

The other insight the normal equations provide is the asymptotic Gaussianity of the least squares estimators. The general form of the Gaussian distribution, Gaussian processes, are another tool used in modern high-dimensional inference. The Gaussian distribution also arises via the central limit theorem in describing weak convergence of the usual least squares estimators. In terms of high-dimensional inference, we are still missing the right notion of weak convergence.

In this mostly expository work, we try to describe how both the normal equations and the theory of Gaussian processes, what we refer to as the “geometry of least squares,” apply to many questions of current interest.

Citation

Download Citation

Jonathan Taylor. "The geometry of least squares in the 21st century." Bernoulli 19 (4) 1449 - 1464, September 2013. https://doi.org/10.3150/12-BEJSP15

Information

Published: September 2013
First available in Project Euclid: 27 August 2013

zbMATH: 06216085
MathSciNet: MR3102558
Digital Object Identifier: 10.3150/12-BEJSP15

Keywords: convex analysis , Gaussian processes , least squares , penalized regression

Rights: Copyright © 2013 Bernoulli Society for Mathematical Statistics and Probability

Vol.19 • No. 4 • September 2013
Back to Top