The Annals of Statistics
- Ann. Statist.
- Volume 36, Number 1 (2008), 64-94.
Approximation and learning by greedy algorithms
We consider the problem of approximating a given element f from a Hilbert space by means of greedy algorithms and the application of such procedures to the regression problem in statistical learning theory. We improve on the existing theory of convergence rates for both the orthogonal greedy algorithm and the relaxed greedy algorithm, as well as for the forward stepwise projection algorithm. For all these algorithms, we prove convergence results for a variety of function classes and not simply those that are related to the convex hull of the dictionary. We then show how these bounds for convergence rates lead to a new theory for the performance of greedy algorithms in learning. In particular, we build upon the results in [IEEE Trans. Inform. Theory 42 (1996) 2118–2132] to construct learning algorithms based on greedy approximations which are universally consistent and provide provable convergence rates for large classes of functions. The use of greedy algorithms in the context of learning is very appealing since it greatly reduces the computational burden when compared with standard model selection using general dictionaries.
Ann. Statist., Volume 36, Number 1 (2008), 64-94.
First available in Project Euclid: 1 February 2008
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Primary: 62G07: Density estimation 41A46: Approximation by arbitrary nonlinear expressions; widths and entropy 41A63: Multidimensional problems (should also be assigned at least one other classification number in this section) 46N30: Applications in probability theory and statistics
Barron, Andrew R.; Cohen, Albert; Dahmen, Wolfgang; DeVore, Ronald A. Approximation and learning by greedy algorithms. Ann. Statist. 36 (2008), no. 1, 64--94. doi:10.1214/009053607000000631. https://projecteuclid.org/euclid.aos/1201877294