The Annals of Statistics

Local greedy approximation for nonlinear regression and neural network training

L. K. Jones

Full-text: Open access


A criterion for local estimation and approximation in nonlinear regres- sion and neural network training is introduced and motivated. $N$th-order greedy approximation for the regression (or target) function based on the criterion is shown to converge at rate $O(1/N^{1/2})$ in the nonsampling case.

Article information

Ann. Statist., Volume 28, Number 5 (2000), 1379-1389.

First available in Project Euclid: 12 March 2002

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62H99: None of the above, but in this section

Greedy approximation local training


Jones, L. K. Local greedy approximation for nonlinear regression and neural network training. Ann. Statist. 28 (2000), no. 5, 1379--1389. doi:10.1214/aos/1015957398.

Export citation


  • [1] Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 40 930-945.
  • [2] Bottou, L. and Vapnik, V. (1992). Local learning algorithms. Neural Computation 4 888-900.
  • [3] Blum, A. L. and Rivest, R. L. (1992). Training a 3-Node Neural Network is NP-Complete. Neural Networks 5 117-127.
  • [4] DeVore, R. A. and Temlyakov, V. N. (1996). Some remarks on greedy algorithms. Adv. Comput. Math 5 173-187.
  • [5] Donohue, M. J., Gurvitz, L., Darken, C. and Sontag, E. (1994). Rates of convex approximation in non-Hilbert spaces. Constr. Approx. 13 187-220.
  • [6] Flick, T. E., Jones, L. K., Priest, R. and Herman, C.(1990). Projection pursuit classification. Pattern Recognition 23 1367-1376.
  • [7] Friedman, J. H. (1999). Greedy function approximation: a gradient boosting machine. Technical report, Stanford Univ.
  • [8] Friedman, J. H. and Stuetzle, W. (1981). Projection pursuit regression. J. Amer. Statist. Assoc. 76 817-823.
  • [9] Huber, P. J. (1985). Projection pursuit. Ann. Statist. 13 435-475.
  • [10] Jones, L. K. (1987). On a conjuecture of Huber concerning the convergence of projection pursuit regression. Ann. Statist. 15 880-882.
  • [11] Jones, L. K. (1992). A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20 608-613.
  • [12] Jones, L. K. (1994). Good weights and hyperbolic kernels for neural networks, projection pursuit, and pattern classification: Fourier strategies for extracting information from high-dimensional data. IEEE Trans. Inform. Theory 40 439-454.
  • [13] Jones, L. K. (1997). The Computational intractability of training sigmoidal neural networks. IEEE Trans. Inform. Theory 43 167-173.
  • [14] Lee, W. S. and Bartlett, P. L., and Williamson, R. C. (1996). Efficient agnostic learning of neural networks with bounded fan-in. IEEE Trans. Inform. Theory 42 2118-2132.
  • [15] Rejto, L. and Walter, G. G. (1992). Remarks on projection pursuit regression and density estimation. Stochastic Anal. Appl. 10 213-222.
  • [16] Vu, V. H. (1998). On the infeasibility of training neural networks with small mean-squared error. IEEE Trans. Inform. Theory 44 2892-2900.