Analysis of the rate of convergence of two regression estimates defined by neural features which are easy to implement

Abstract

Recent results in nonparametric regression have shown that neural network regression estimates with many hidden layers are able to achieve good rates of convergence even in case of high-dimensional predictor variables, provided suitable assumptions on the structure of the regression function are imposed. In those recent results, the estimates were defined by minimizing the empirical $L_2$ risk over a class of neural networks. In practice, however, it is not clear how this can be done exactly. In this article, motivated by some recent approximation results for neural networks, we introduce two new regression estimates defined by neural features where most of the neural network weights are chosen via random initialization and no training, thus sparing the costly data-dependent optimization. For the first estimate, which is defined by these neural features and an extra layer whose weights are set via least squares, we derive rates of convergence results in case the regression function is smooth. We then combine this estimate with the projection pursuit, where we choose the directions randomly, and we show that for sufficiently many repetitions we get a second regression estimate which achieves the one-dimensional rate of convergence (up to some logarithmic factor) in case that the regression function satisfies the assumptions of projection pursuit. Because the neural features are obtained by random initialization but not training of the weights, the two estimators thus defined are easy to implement.