Electronic Journal of Statistics

Solution of linear ill-posed problems by model selection and aggregation

Felix Abramovich, Daniela De Canditiis, and Marianna Pensky

Full-text: Open access


We consider a general statistical linear inverse problem, where the solution is represented via a known (possibly overcomplete) dictionary that allows its sparse representation. We propose two different approaches. A model selection estimator selects a single model by minimizing the penalized empirical risk over all possible models. By contrast with direct problems, the penalty depends on the model itself rather than on its size only as for complexity penalties. A Q-aggregate estimator averages over the entire collection of estimators with properly chosen weights. Under mild conditions on the dictionary, we establish oracle inequalities both with high probability and in expectation for the two estimators. Moreover, for the latter estimator these inequalities are sharp. The proposed procedures are implemented numerically and their performance is assessed by a simulation study.

Article information

Electron. J. Statist., Volume 12, Number 1 (2018), 1822-1841.

Received: October 2017
First available in Project Euclid: 12 June 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G05: Estimation
Secondary: 62C10: Bayesian problems; characterization of Bayes procedures

Aggregation ill-posed linear inverse problem model selection oracle inequality overcomplete dictionary

Creative Commons Attribution 4.0 International License.


Abramovich, Felix; De Canditiis, Daniela; Pensky, Marianna. Solution of linear ill-posed problems by model selection and aggregation. Electron. J. Statist. 12 (2018), no. 1, 1822--1841. doi:10.1214/18-EJS1447. https://projecteuclid.org/euclid.ejs/1528769121

Export citation


  • [1] Abramovich, F. and Grinshtein, V. (2010). MAP model selection in Gaussian regression., Electr. J. Statist. 4, 932–949.
  • [2] Abramovich F. and Silverman B.W. (1998). Wavelet decomposition approaches to statistical inverse problems., Biometrika 85, 115–129.
  • [3] Bellec, P.C. (2018). Optimal bounds for aggregation of affine estimators., Ann. Statist. 46, 30–59.
  • [4] Birgé, L. and Massart, P. (2001). Gaussian model selection., J. Eur. Math. Soc. 3, 203–268.
  • [5] Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection., Probab. Theory Relat. Fields 138 33–73.
  • [6] Brémaud, P. (1999)., Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues, Springer, New York.
  • [7] Cavalier, L. (2011). Inverse problems in staistics., Inverse Problems and High Dimensional Estimation (Eds. Alquier, P., Gautier, E. and Gilles, G.), Lecture Notes in Statistics 203, Springer, 3–96.
  • [8] Cavalier, L. and Golubev, Yu. (2006) Risk hull method and regularization by projections of ill-posed inverse problems., Ann. Statist. 34, 1653–1677.
  • [9] Cavalier, L., Golubev, G.K., Picard, D. and Tsybakov, A.B. (2002). Oracle inequalities for inverse problems., Ann. Statist. 30, 843–874.
  • [10] Cavalier, L. and Tsybakov, A. (2002). Sharp adaptation for inverse problems with random noise, Probab. Theory Related Fields 123, 323—354.
  • [11] Chen, S., Donoho, D.L. and Saunders, M. A. (1999). Atomic decomposition by basic pursuit., SIAM J. Scient. Comput. 20, 33–61.
  • [12] Dai, D., Rigolette, P. and Zhang, T. (2012) Deviation optimal leaning using greedy Q-aggregation., Ann. Statist. 40, 1878–1905.
  • [13] Dai, D., Rigolette, P., Xia, L. and Zhang, T. (2014) Aggregation of affine estimators., Electr. J. Statist. 8, 302–327.
  • [14] Dalalyan, A. and Salmon, J. (2012) Sharp oracle inequalities for aggregation of affine estimators., Ann. Statist. 40, 2327–2355.
  • [15] Donoho, D.L. (1995). Nonlinear solutions of linear inverse problems by wavelet-vaguelette decomposition., Appl. Comput. Harmon. Anal. 2, 101–126.
  • [16] Donoho, D.L. and Elad, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via $l_1$ minimization., PNAS 100, 2197–2202.
  • [17] Donoho, D.L. and Johnstone, I.M. (1994). Ideal spatial adaptation by wavelet shrinkage., Biometrika 81, 425–455.
  • [18] Foster, D.P. and George, E.I. (1994) The risk inflation criterion for multiple regression., Ann. Statist. 22, 1947–1975.
  • [19] Johnstone, I.M., Kerkyacharian, G., Picard, D. and Raimondo, M. (2004) Wavelet decomposition in a periodic setting., J. R. Statist. Soc. B 66, 547–573 (with discussion).
  • [20] Johnstone, I.M. and Silverman, B.W. (1990) Speed of estimation in positron emission tomography and related inverse problems., Ann. Statist. 18, 251–280.
  • [21] Kalifa, J. and Mallat, S. (2003) Thresholding estimators for linear inverse problems and deconvolution., Ann. Statist. 31, 58–109.
  • [22] Leung, G. and Barron, A.R. (2006) Information theory and mixing least-squares regressions., IEEE Trans. Inf. Theory 52, 3396–3410.
  • [23] Mallat, S. and Zhang, Z. (1993). Matching pursuit with time-frequency dictionaries., IEEE Trans. in Signal Proc. 41, 3397–3415.
  • [24] Pensky, M. (2016) Solution of linear ill-posed problems using overcomplete dictionaries., Ann. Statist. 44, 1739–1754.
  • [25] Rigollet, P. and Tsybakov, A. (2011) Exponential screening and optimal rates of sparse estimation., Ann. Statist. 39, 731–771.
  • [26] Rigollet, P. and Tsybakov, A. (2012) Sparse estimation by exponential weighting., Statist. Science 27, 558–575.
  • [27] Verzelen, N. (2012). Minimax risks for sparse regressions: Ultra-high dimensionals phenomenon., Electr. J. Statist. 6, 38–90.