We consider a general statistical linear inverse problem, where the solution is represented via a known (possibly overcomplete) dictionary that allows its sparse representation. We propose two different approaches. A model selection estimator selects a single model by minimizing the penalized empirical risk over all possible models. By contrast with direct problems, the penalty depends on the model itself rather than on its size only as for complexity penalties. A Q-aggregate estimator averages over the entire collection of estimators with properly chosen weights. Under mild conditions on the dictionary, we establish oracle inequalities both with high probability and in expectation for the two estimators. Moreover, for the latter estimator these inequalities are sharp. The proposed procedures are implemented numerically and their performance is assessed by a simulation study.
References
[1] Abramovich, F. and Grinshtein, V. (2010). MAP model selection in Gaussian regression., Electr. J. Statist. 4, 932–949. 1329.62051 10.1214/10-EJS573[1] Abramovich, F. and Grinshtein, V. (2010). MAP model selection in Gaussian regression., Electr. J. Statist. 4, 932–949. 1329.62051 10.1214/10-EJS573
[2] Abramovich F. and Silverman B.W. (1998). Wavelet decomposition approaches to statistical inverse problems., Biometrika 85, 115–129. 0908.62095 10.1093/biomet/85.1.115[2] Abramovich F. and Silverman B.W. (1998). Wavelet decomposition approaches to statistical inverse problems., Biometrika 85, 115–129. 0908.62095 10.1093/biomet/85.1.115
[3] Bellec, P.C. (2018). Optimal bounds for aggregation of affine estimators., Ann. Statist. 46, 30–59. 06865104 10.1214/17-AOS1540 euclid.aos/1519268423[3] Bellec, P.C. (2018). Optimal bounds for aggregation of affine estimators., Ann. Statist. 46, 30–59. 06865104 10.1214/17-AOS1540 euclid.aos/1519268423
[4] Birgé, L. and Massart, P. (2001). Gaussian model selection., J. Eur. Math. Soc. 3, 203–268. 1037.62001 10.1007/s100970100031[4] Birgé, L. and Massart, P. (2001). Gaussian model selection., J. Eur. Math. Soc. 3, 203–268. 1037.62001 10.1007/s100970100031
[5] Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection., Probab. Theory Relat. Fields 138 33–73. 1112.62082 10.1007/s00440-006-0011-8[5] Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection., Probab. Theory Relat. Fields 138 33–73. 1112.62082 10.1007/s00440-006-0011-8
[7] Cavalier, L. (2011). Inverse problems in staistics., Inverse Problems and High Dimensional Estimation (Eds. Alquier, P., Gautier, E. and Gilles, G.), Lecture Notes in Statistics 203, Springer, 3–96.[7] Cavalier, L. (2011). Inverse problems in staistics., Inverse Problems and High Dimensional Estimation (Eds. Alquier, P., Gautier, E. and Gilles, G.), Lecture Notes in Statistics 203, Springer, 3–96.
[8] Cavalier, L. and Golubev, Yu. (2006) Risk hull method and regularization by projections of ill-posed inverse problems., Ann. Statist. 34, 1653–1677. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628[8] Cavalier, L. and Golubev, Yu. (2006) Risk hull method and regularization by projections of ill-posed inverse problems., Ann. Statist. 34, 1653–1677. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628
[9] Cavalier, L., Golubev, G.K., Picard, D. and Tsybakov, A.B. (2002). Oracle inequalities for inverse problems., Ann. Statist. 30, 843–874. 1029.62032 10.1214/aos/1028674843 euclid.aos/1028674843[9] Cavalier, L., Golubev, G.K., Picard, D. and Tsybakov, A.B. (2002). Oracle inequalities for inverse problems., Ann. Statist. 30, 843–874. 1029.62032 10.1214/aos/1028674843 euclid.aos/1028674843
[10] Cavalier, L. and Tsybakov, A. (2002). Sharp adaptation for inverse problems with random noise, Probab. Theory Related Fields 123, 323—354. 1039.62031 10.1007/s004400100169[10] Cavalier, L. and Tsybakov, A. (2002). Sharp adaptation for inverse problems with random noise, Probab. Theory Related Fields 123, 323—354. 1039.62031 10.1007/s004400100169
[11] Chen, S., Donoho, D.L. and Saunders, M. A. (1999). Atomic decomposition by basic pursuit., SIAM J. Scient. Comput. 20, 33–61. 0919.94002 10.1137/S1064827596304010[11] Chen, S., Donoho, D.L. and Saunders, M. A. (1999). Atomic decomposition by basic pursuit., SIAM J. Scient. Comput. 20, 33–61. 0919.94002 10.1137/S1064827596304010
[12] Dai, D., Rigolette, P. and Zhang, T. (2012) Deviation optimal leaning using greedy Q-aggregation., Ann. Statist. 40, 1878–1905. 1257.62037 10.1214/12-AOS1025 euclid.aos/1350394520[12] Dai, D., Rigolette, P. and Zhang, T. (2012) Deviation optimal leaning using greedy Q-aggregation., Ann. Statist. 40, 1878–1905. 1257.62037 10.1214/12-AOS1025 euclid.aos/1350394520
[13] Dai, D., Rigolette, P., Xia, L. and Zhang, T. (2014) Aggregation of affine estimators., Electr. J. Statist. 8, 302–327. 1348.62132 10.1214/14-EJS886[13] Dai, D., Rigolette, P., Xia, L. and Zhang, T. (2014) Aggregation of affine estimators., Electr. J. Statist. 8, 302–327. 1348.62132 10.1214/14-EJS886
[14] Dalalyan, A. and Salmon, J. (2012) Sharp oracle inequalities for aggregation of affine estimators., Ann. Statist. 40, 2327–2355. 1257.62038 10.1214/12-AOS1038 euclid.aos/1358951384[14] Dalalyan, A. and Salmon, J. (2012) Sharp oracle inequalities for aggregation of affine estimators., Ann. Statist. 40, 2327–2355. 1257.62038 10.1214/12-AOS1038 euclid.aos/1358951384
[15] Donoho, D.L. (1995). Nonlinear solutions of linear inverse problems by wavelet-vaguelette decomposition., Appl. Comput. Harmon. Anal. 2, 101–126. MR1325535 0826.65117 10.1006/acha.1995.1008[15] Donoho, D.L. (1995). Nonlinear solutions of linear inverse problems by wavelet-vaguelette decomposition., Appl. Comput. Harmon. Anal. 2, 101–126. MR1325535 0826.65117 10.1006/acha.1995.1008
[16] Donoho, D.L. and Elad, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via $l_1$ minimization., PNAS 100, 2197–2202. 1064.94011 10.1073/pnas.0437847100[16] Donoho, D.L. and Elad, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via $l_1$ minimization., PNAS 100, 2197–2202. 1064.94011 10.1073/pnas.0437847100
[17] Donoho, D.L. and Johnstone, I.M. (1994). Ideal spatial adaptation by wavelet shrinkage., Biometrika 81, 425–455. 0815.62019 10.1093/biomet/81.3.425[17] Donoho, D.L. and Johnstone, I.M. (1994). Ideal spatial adaptation by wavelet shrinkage., Biometrika 81, 425–455. 0815.62019 10.1093/biomet/81.3.425
[18] Foster, D.P. and George, E.I. (1994) The risk inflation criterion for multiple regression., Ann. Statist. 22, 1947–1975. 0829.62066 10.1214/aos/1176325766 euclid.aos/1176325766[18] Foster, D.P. and George, E.I. (1994) The risk inflation criterion for multiple regression., Ann. Statist. 22, 1947–1975. 0829.62066 10.1214/aos/1176325766 euclid.aos/1176325766
[19] Johnstone, I.M., Kerkyacharian, G., Picard, D. and Raimondo, M. (2004) Wavelet decomposition in a periodic setting., J. R. Statist. Soc. B 66, 547–573 (with discussion). 1046.62039 10.1111/j.1467-9868.2004.02056.x[19] Johnstone, I.M., Kerkyacharian, G., Picard, D. and Raimondo, M. (2004) Wavelet decomposition in a periodic setting., J. R. Statist. Soc. B 66, 547–573 (with discussion). 1046.62039 10.1111/j.1467-9868.2004.02056.x
[20] Johnstone, I.M. and Silverman, B.W. (1990) Speed of estimation in positron emission tomography and related inverse problems., Ann. Statist. 18, 251–280. 0699.62043 10.1214/aos/1176347500 euclid.aos/1176347500[20] Johnstone, I.M. and Silverman, B.W. (1990) Speed of estimation in positron emission tomography and related inverse problems., Ann. Statist. 18, 251–280. 0699.62043 10.1214/aos/1176347500 euclid.aos/1176347500
[21] Kalifa, J. and Mallat, S. (2003) Thresholding estimators for linear inverse problems and deconvolution., Ann. Statist. 31, 58–109. 1102.62318 10.1214/aos/1046294458 euclid.aos/1046294458[21] Kalifa, J. and Mallat, S. (2003) Thresholding estimators for linear inverse problems and deconvolution., Ann. Statist. 31, 58–109. 1102.62318 10.1214/aos/1046294458 euclid.aos/1046294458
[22] Leung, G. and Barron, A.R. (2006) Information theory and mixing least-squares regressions., IEEE Trans. Inf. Theory 52, 3396–3410. 1309.94051 10.1109/TIT.2006.878172[22] Leung, G. and Barron, A.R. (2006) Information theory and mixing least-squares regressions., IEEE Trans. Inf. Theory 52, 3396–3410. 1309.94051 10.1109/TIT.2006.878172
[23] Mallat, S. and Zhang, Z. (1993). Matching pursuit with time-frequency dictionaries., IEEE Trans. in Signal Proc. 41, 3397–3415. 0842.94004 10.1109/78.258082[23] Mallat, S. and Zhang, Z. (1993). Matching pursuit with time-frequency dictionaries., IEEE Trans. in Signal Proc. 41, 3397–3415. 0842.94004 10.1109/78.258082
[24] Pensky, M. (2016) Solution of linear ill-posed problems using overcomplete dictionaries., Ann. Statist. 44, 1739–1754. 1346.62061 10.1214/16-AOS1445 euclid.aos/1467894714[24] Pensky, M. (2016) Solution of linear ill-posed problems using overcomplete dictionaries., Ann. Statist. 44, 1739–1754. 1346.62061 10.1214/16-AOS1445 euclid.aos/1467894714
[25] Rigollet, P. and Tsybakov, A. (2011) Exponential screening and optimal rates of sparse estimation., Ann. Statist. 39, 731–771. 1215.62043 10.1214/10-AOS854 euclid.aos/1299680953[25] Rigollet, P. and Tsybakov, A. (2011) Exponential screening and optimal rates of sparse estimation., Ann. Statist. 39, 731–771. 1215.62043 10.1214/10-AOS854 euclid.aos/1299680953
[26] Rigollet, P. and Tsybakov, A. (2012) Sparse estimation by exponential weighting., Statist. Science 27, 558–575. 1331.62351 10.1214/12-STS393 euclid.ss/1356098556[26] Rigollet, P. and Tsybakov, A. (2012) Sparse estimation by exponential weighting., Statist. Science 27, 558–575. 1331.62351 10.1214/12-STS393 euclid.ss/1356098556
[27] Verzelen, N. (2012). Minimax risks for sparse regressions: Ultra-high dimensionals phenomenon., Electr. J. Statist. 6, 38–90. 1334.62120 10.1214/12-EJS666[27] Verzelen, N. (2012). Minimax risks for sparse regressions: Ultra-high dimensionals phenomenon., Electr. J. Statist. 6, 38–90. 1334.62120 10.1214/12-EJS666
