Electronic Journal of Statistics

Ordered smoothers with exponential weighting

Elena Chernousova, Yuri Golubev, and Ekaterina Krymova

Full-text: Open access


The main goal in this paper is to propose a new approach to deriving oracle inequalities related to the exponential weighting method. The paper focuses on recovering an unknown vector from noisy data with the help of the family of ordered smoothers [12]. The estimators withing this family are aggregated using the exponential weighting method and the aim is to control the risk of the aggregated estimate. Based on the natural probabilistic properties of the unbiased risk estimate, we derive new oracle inequalities for the mean square risk and show that the exponential weighting permits to improve Kneip’s oracle inequality.

Article information

Electron. J. Statist., Volume 7 (2013), 2395-2419.

First available in Project Euclid: 30 September 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62C99: None of the above, but in this section
Secondary: 62C10: Bayesian problems; characterization of Bayes procedures 62C20: Minimax procedures 62J05: Linear regression

Linear model ordered smoother exponential weighting


Chernousova, Elena; Golubev, Yuri; Krymova, Ekaterina. Ordered smoothers with exponential weighting. Electron. J. Statist. 7 (2013), 2395--2419. doi:10.1214/13-EJS849. https://projecteuclid.org/euclid.ejs/1380546591

Export citation


  • [1] Alquier, P. and Lounici, K. (2011). Pac-bayesian theorems for sparse regression estimation with exponential weights., Electronic Journal of Statistics 5 127–145.
  • [2] Arias-Castro, E. and Lounici, K. Variable Selection with Exponential Weights and $\ell_0$-Penalization. arXiv, :1208.2635.
  • [3] Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle., Proc. 2nd Intern. Symp. Inf. Theory 267–281.
  • [4] Catoni, O. (2004)., Statistical Learning Theory and Stochastic Optimization. Lectures Notes in Math. 1851. Springer-Verlag, Berlin.
  • [5] Dalayan, A. and Salmon, J. (2012). Sharp oracle inequalities for aggregation of affine estimators., Ann. Statist. 40 2327–2355.
  • [6] Demmler, A. and Reinsch, C. (1975). Oscillation matrices with spline smoothing., Numerische Mathematik 24 375–382.
  • [7] Engl, H. W., Hanke, M., and Neubauer, A. (1996)., Regularization of Inverse Problems. Mathematics and its Applications 375. Kluwer Academic Publishers Group, Dordrecht.
  • [8] Golubev, Yu. (2010). On universal oracle inequalities related to high dimensional linear models., Ann. Statist. 38 2751–2780.
  • [9] Golubev, G. (2012). Exponential weighting and oracle inequalities for projection estimates., Problems of Information Transmission 48 269–280.
  • [10] Green, P. J. and Silverman, B. W. (1994)., Nonparametric Regression and Generalized Linear Models. A Roughness Penalty Approach. Chapman and Hall.
  • [11] Juditsky, A. and Nemirovski, A. (2000). Functional aggregation for nonparametric regression., Ann. Statist. 28 681–712.
  • [12] Kneip, A. (1994). Ordered linear smoothers., Ann. Statist. 22 835–866.
  • [13] Lecué, G. (2007). Simultaneous adaptation to the margin and to complexity in classification., Ann. Statist. 35 1698–1721.
  • [14] Leung, G. and Barron, A. (2006). Information theory and mixing least-squares regressions., IEEE Transactions on Information Theory 52 3396–3410.
  • [15] Mallows, C. L. (1973). Some comments on $C_p$., Technometrics 15 661–675.
  • [16] Nemirovski, A. (2000)., Topics in Non-Parametric Statistics. Lectures Notes in Math. 1738. Springer-Verlag, Berlin.
  • [17] Nussbaum, M. (1985). Spline smoothing in regression models and asymptotic efficiency in $L_2$., Ann. Statist. 13 984–997.
  • [18] Rigollet, Ph. and Tsybakov, A. (2007). Linear and convex aggregation of density estimators., Math. Methods Statist. 16 260–280.
  • [19] Rigollet, Ph. and Tsybakov, A. (2012). Sparse estimation by exponential weighting., Statist. Sci. 27 558–575.
  • [20] Speckman, P. (1985). Spline smoothing and optimal rates of convergence in nonparametric regression., Ann. Statist. 13 970–983.
  • [21] Stein, C. (1981). Estimation of the mean of a multivariate normal distribution., Ann. Statist. 9 1135–1151.
  • [22] Tikhonov, A. N. and Arsenin, V. A. (1977)., Solution of Ill-posed Problems. Translated from the Russian. Preface by translation editor Fritz John. Scripta Series in Mathematics. V. H. Winston & Sons, Washington, D.C.: John Wiley & Sons, New York.
  • [23] Wahba, G. (1990)., Spline Models for Observational Data. SIAM, Philadelphia.
  • [24] Yang, Y. (2000). Combining different procedures for adaptive regression., J. Multivariate Anal. 74 135–161.