Electronic Journal of Statistics

Early stopping for statistical inverse problems via truncated SVD estimation

Abstract

We consider truncated SVD (or spectral cut-off, projection) estimators for a prototypical statistical inverse problem in dimension $D$. Since calculating the singular value decomposition (SVD) only for the largest singular values is much less costly than the full SVD, our aim is to select a data-driven truncation level $\widehat{m}\in \{1,\ldots ,D\}$ only based on the knowledge of the first $\widehat{m}$ singular values and vectors.

We analyse in detail whether sequential early stopping rules of this type can preserve statistical optimality. Information-constrained lower bounds and matching upper bounds for a residual based stopping rule are provided, which give a clear picture in which situation optimal sequential adaptation is feasible. Finally, a hybrid two-step approach is proposed which allows for classical oracle inequalities while considerably reducing numerical complexity.

Article information

Source
Electron. J. Statist., Volume 12, Number 2 (2018), 3204-3231.

Dates
First available in Project Euclid: 28 September 2018

https://projecteuclid.org/euclid.ejs/1538121641

Digital Object Identifier
doi:10.1214/18-EJS1482

Mathematical Reviews number (MathSciNet)
MR3859376

Zentralblatt MATH identifier
06970002

Citation

Blanchard, Gilles; Hoffmann, Marc; Reiß, Markus. Early stopping for statistical inverse problems via truncated SVD estimation. Electron. J. Statist. 12 (2018), no. 2, 3204--3231. doi:10.1214/18-EJS1482. https://projecteuclid.org/euclid.ejs/1538121641

References

• [1] N. Bissantz, T. Hohage, A. Munk, and F. Ruymgaart. Convergence rates of general regularization methods for statistical inverse problems and applications., SIAM Journal on Numerical Analysis, 45 :2610–2636, 2007.
• [2] G. Blanchard, M. Hoffmann, and M. Reiß. Optimal adaptation for early stopping in statistical inverse problems., SIAM Journal of Uncertainty Quantification, 6 :1043–1075, 2018.
• [3] G. Blanchard and P. Mathé. Discrepancy principle for statistical inverse problems with application to conjugate gradient iteration., Inverse Problems, 28:pp. 115011, 2012.
• [4] P. Bühlmann and T. Hothorn. Boosting algorithms: Regularization, prediction and model fitting., Statistical Science, pages 477–505, 2007.
• [5] L. Cavalier. Inverse problems in statistics. In, Inverse problems and high-dimensional estimation, pages 3–96. Lecture Notes in Statistics 203, Springer, 2011.
• [6] L. Cavalier, G.K. Golubev D. Picard, and A.B. Tsybakov. Oracle inequalities for inverse problems., Annals of Statistics, 30:843–874, 2002.
• [7] L. Cavalier and Y. Golubev. Risk hull method and regularization by projections of ill-posed inverse problems., Annals of Statistics, 34 :1653–1677, 2006.
• [8] E. Chernousova, Y. Golubev, and E. Krymova. Ordered smoothers with exponential weighting., Electronic Journal of Statistics, 7 :2395–2419, 2013.
• [9] A. Cohen, M. Hoffmann, and M. Reiß. Adaptive wavelet Galerkin methods for linear inverse problems., SIAM Journal on Numerical Analysis, 42(4) :1479–1501, 2004.
• [10] H. Engl, M. Hanke, and A. Neubauer., Regularization of Inverse Problems. Kluwer Academic Publishers, London, 1996.
• [11] Y. Ingster and I. Suslina., Nonparametric goodness-of-fit testing under Gaussian models. Lecture Notes in Statistics 169, Springer, 2012.
• [12] B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection., The Annals of Statistics, 28 :1302–1338, 2000.
• [13] O. Lepski. Some new ideas in nonparametric estimation., arXiv :1603.03934, 2016.
• [14] F. Lucka, K. Proksch, C. Brune, N. Bissantz, M. Burger, H. Dette, and F. Wübbeling. Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations. e-print, arXiv :1701.04970, 2017.
• [15] P. Mathé and S. V. Pereverzev. Geometry of linear ill-posed problems in variable hilbert scales., Inverse problems, 19(3):789, 2003.
• [16] G. Raskutti, M. Wainwright, and B. Yu. Early stopping and non-parametric regression: An optimal data-dependent stopping rule., Journal of Machine Learning Research, 15:335–366, 2014.
• [17] Y. Saad., Numerical Methods for Large Eigenvalue Problems: Revised Edition. Society for Industrial and Applied Mathematics (SIAM), 2011.
• [18] A. B. Tsybakov., Introduction to nonparametric estimation. Springer Series in Statistics. Springer, New York, 2009.
• [19] G. Wahba. Practical approximate solutions to linear operator equations when the data are noisy., SIAM Journal on Numerical Analysis, 14(4):651–667, 1977.
• [20] Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning., Constructive Approximation, 26(2):289–315, 2007.