We consider truncated SVD (or spectral cut-off, projection) estimators for a prototypical statistical inverse problem in dimension $D$. Since calculating the singular value decomposition (SVD) only for the largest singular values is much less costly than the full SVD, our aim is to select a data-driven truncation level $\widehat{m}\in \{1,\ldots ,D\}$ only based on the knowledge of the first $\widehat{m}$ singular values and vectors.
We analyse in detail whether sequential early stopping rules of this type can preserve statistical optimality. Information-constrained lower bounds and matching upper bounds for a residual based stopping rule are provided, which give a clear picture in which situation optimal sequential adaptation is feasible. Finally, a hybrid two-step approach is proposed which allows for classical oracle inequalities while considerably reducing numerical complexity.
References
[1] N. Bissantz, T. Hohage, A. Munk, and F. Ruymgaart. Convergence rates of general regularization methods for statistical inverse problems and applications., SIAM Journal on Numerical Analysis, 45 :2610–2636, 2007. 1234.62062 10.1137/060651884[1] N. Bissantz, T. Hohage, A. Munk, and F. Ruymgaart. Convergence rates of general regularization methods for statistical inverse problems and applications., SIAM Journal on Numerical Analysis, 45 :2610–2636, 2007. 1234.62062 10.1137/060651884
[2] G. Blanchard, M. Hoffmann, and M. Reiß. Optimal adaptation for early stopping in statistical inverse problems., SIAM Journal of Uncertainty Quantification, 6 :1043–1075, 2018. 06954355 10.1137/17M1154096[2] G. Blanchard, M. Hoffmann, and M. Reiß. Optimal adaptation for early stopping in statistical inverse problems., SIAM Journal of Uncertainty Quantification, 6 :1043–1075, 2018. 06954355 10.1137/17M1154096
[3] G. Blanchard and P. Mathé. Discrepancy principle for statistical inverse problems with application to conjugate gradient iteration., Inverse Problems, 28:pp. 115011, 2012.[3] G. Blanchard and P. Mathé. Discrepancy principle for statistical inverse problems with application to conjugate gradient iteration., Inverse Problems, 28:pp. 115011, 2012.
[4] P. Bühlmann and T. Hothorn. Boosting algorithms: Regularization, prediction and model fitting., Statistical Science, pages 477–505, 2007. 1246.62163 10.1214/07-STS242 euclid.ss/1207580163[4] P. Bühlmann and T. Hothorn. Boosting algorithms: Regularization, prediction and model fitting., Statistical Science, pages 477–505, 2007. 1246.62163 10.1214/07-STS242 euclid.ss/1207580163
[5] L. Cavalier. Inverse problems in statistics. In, Inverse problems and high-dimensional estimation, pages 3–96. Lecture Notes in Statistics 203, Springer, 2011. MR2868199[5] L. Cavalier. Inverse problems in statistics. In, Inverse problems and high-dimensional estimation, pages 3–96. Lecture Notes in Statistics 203, Springer, 2011. MR2868199
[6] L. Cavalier, G.K. Golubev D. Picard, and A.B. Tsybakov. Oracle inequalities for inverse problems., Annals of Statistics, 30:843–874, 2002. 1029.62032 10.1214/aos/1028674843 euclid.aos/1028674843[6] L. Cavalier, G.K. Golubev D. Picard, and A.B. Tsybakov. Oracle inequalities for inverse problems., Annals of Statistics, 30:843–874, 2002. 1029.62032 10.1214/aos/1028674843 euclid.aos/1028674843
[7] L. Cavalier and Y. Golubev. Risk hull method and regularization by projections of ill-posed inverse problems., Annals of Statistics, 34 :1653–1677, 2006. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628[7] L. Cavalier and Y. Golubev. Risk hull method and regularization by projections of ill-posed inverse problems., Annals of Statistics, 34 :1653–1677, 2006. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628
[8] E. Chernousova, Y. Golubev, and E. Krymova. Ordered smoothers with exponential weighting., Electronic Journal of Statistics, 7 :2395–2419, 2013. 1349.62129 10.1214/13-EJS849[8] E. Chernousova, Y. Golubev, and E. Krymova. Ordered smoothers with exponential weighting., Electronic Journal of Statistics, 7 :2395–2419, 2013. 1349.62129 10.1214/13-EJS849
[9] A. Cohen, M. Hoffmann, and M. Reiß. Adaptive wavelet Galerkin methods for linear inverse problems., SIAM Journal on Numerical Analysis, 42(4) :1479–1501, 2004. 1077.65054 10.1137/S0036142902411793[9] A. Cohen, M. Hoffmann, and M. Reiß. Adaptive wavelet Galerkin methods for linear inverse problems., SIAM Journal on Numerical Analysis, 42(4) :1479–1501, 2004. 1077.65054 10.1137/S0036142902411793
[10] H. Engl, M. Hanke, and A. Neubauer., Regularization of Inverse Problems. Kluwer Academic Publishers, London, 1996. 0859.65054[10] H. Engl, M. Hanke, and A. Neubauer., Regularization of Inverse Problems. Kluwer Academic Publishers, London, 1996. 0859.65054
[12] B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection., The Annals of Statistics, 28 :1302–1338, 2000. 1105.62328 10.1214/aos/1015957395 euclid.aos/1015957395[12] B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection., The Annals of Statistics, 28 :1302–1338, 2000. 1105.62328 10.1214/aos/1015957395 euclid.aos/1015957395
[14] F. Lucka, K. Proksch, C. Brune, N. Bissantz, M. Burger, H. Dette, and F. Wübbeling. Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations. e-print, arXiv :1701.04970, 2017. 06945044 10.3934/ipi.2018047[14] F. Lucka, K. Proksch, C. Brune, N. Bissantz, M. Burger, H. Dette, and F. Wübbeling. Risk estimators for choosing regularization parameters in ill-posed problems - properties and limitations. e-print, arXiv :1701.04970, 2017. 06945044 10.3934/ipi.2018047
[16] G. Raskutti, M. Wainwright, and B. Yu. Early stopping and non-parametric regression: An optimal data-dependent stopping rule., Journal of Machine Learning Research, 15:335–366, 2014. 1318.62136[16] G. Raskutti, M. Wainwright, and B. Yu. Early stopping and non-parametric regression: An optimal data-dependent stopping rule., Journal of Machine Learning Research, 15:335–366, 2014. 1318.62136
[17] Y. Saad., Numerical Methods for Large Eigenvalue Problems: Revised Edition. Society for Industrial and Applied Mathematics (SIAM), 2011. MR3396212 1242.65068[17] Y. Saad., Numerical Methods for Large Eigenvalue Problems: Revised Edition. Society for Industrial and Applied Mathematics (SIAM), 2011. MR3396212 1242.65068
[19] G. Wahba. Practical approximate solutions to linear operator equations when the data are noisy., SIAM Journal on Numerical Analysis, 14(4):651–667, 1977. 0402.65032 10.1137/0714044[19] G. Wahba. Practical approximate solutions to linear operator equations when the data are noisy., SIAM Journal on Numerical Analysis, 14(4):651–667, 1977. 0402.65032 10.1137/0714044
[20] Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning., Constructive Approximation, 26(2):289–315, 2007. 1125.62035 10.1007/s00365-006-0663-2[20] Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning., Constructive Approximation, 26(2):289–315, 2007. 1125.62035 10.1007/s00365-006-0663-2
