Electronic Journal of Statistics

Ordered smoothers with exponential weighting

Elena Chernousova, Yuri Golubev, and Ekaterina Krymova

Full-text: Open access

Abstract

The main goal in this paper is to propose a new approach to deriving oracle inequalities related to the exponential weighting method. The paper focuses on recovering an unknown vector from noisy data with the help of the family of ordered smoothers [12]. The estimators withing this family are aggregated using the exponential weighting method and the aim is to control the risk of the aggregated estimate. Based on the natural probabilistic properties of the unbiased risk estimate, we derive new oracle inequalities for the mean square risk and show that the exponential weighting permits to improve Kneip’s oracle inequality.

Article information

Source
Electron. J. Statist., Volume 7 (2013), 2395-2419.

Dates
First available in Project Euclid: 30 September 2013

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1380546591

Digital Object Identifier
doi:10.1214/13-EJS849

Mathematical Reviews number (MathSciNet)
MR3108818

Zentralblatt MATH identifier
1349.62129

Subjects
Primary: 62C99: None of the above, but in this section
Secondary: 62C10: Bayesian problems; characterization of Bayes procedures 62C20: Minimax procedures 62J05: Linear regression

Keywords
Linear model ordered smoother exponential weighting

Citation

Chernousova, Elena; Golubev, Yuri; Krymova, Ekaterina. Ordered smoothers with exponential weighting. Electron. J. Statist. 7 (2013), 2395--2419. doi:10.1214/13-EJS849. https://projecteuclid.org/euclid.ejs/1380546591


Export citation

References

  • [1] Alquier, P. and Lounici, K. (2011). Pac-bayesian theorems for sparse regression estimation with exponential weights., Electronic Journal of Statistics 5 127–145.
  • [2] Arias-Castro, E. and Lounici, K. Variable Selection with Exponential Weights and $\ell_0$-Penalization. arXiv, :1208.2635.
  • [3] Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle., Proc. 2nd Intern. Symp. Inf. Theory 267–281.
  • [4] Catoni, O. (2004)., Statistical Learning Theory and Stochastic Optimization. Lectures Notes in Math. 1851. Springer-Verlag, Berlin.
  • [5] Dalayan, A. and Salmon, J. (2012). Sharp oracle inequalities for aggregation of affine estimators., Ann. Statist. 40 2327–2355.
  • [6] Demmler, A. and Reinsch, C. (1975). Oscillation matrices with spline smoothing., Numerische Mathematik 24 375–382.
  • [7] Engl, H. W., Hanke, M., and Neubauer, A. (1996)., Regularization of Inverse Problems. Mathematics and its Applications 375. Kluwer Academic Publishers Group, Dordrecht.
  • [8] Golubev, Yu. (2010). On universal oracle inequalities related to high dimensional linear models., Ann. Statist. 38 2751–2780.
  • [9] Golubev, G. (2012). Exponential weighting and oracle inequalities for projection estimates., Problems of Information Transmission 48 269–280.
  • [10] Green, P. J. and Silverman, B. W. (1994)., Nonparametric Regression and Generalized Linear Models. A Roughness Penalty Approach. Chapman and Hall.
  • [11] Juditsky, A. and Nemirovski, A. (2000). Functional aggregation for nonparametric regression., Ann. Statist. 28 681–712.
  • [12] Kneip, A. (1994). Ordered linear smoothers., Ann. Statist. 22 835–866.
  • [13] Lecué, G. (2007). Simultaneous adaptation to the margin and to complexity in classification., Ann. Statist. 35 1698–1721.
  • [14] Leung, G. and Barron, A. (2006). Information theory and mixing least-squares regressions., IEEE Transactions on Information Theory 52 3396–3410.
  • [15] Mallows, C. L. (1973). Some comments on $C_p$., Technometrics 15 661–675.
  • [16] Nemirovski, A. (2000)., Topics in Non-Parametric Statistics. Lectures Notes in Math. 1738. Springer-Verlag, Berlin.
  • [17] Nussbaum, M. (1985). Spline smoothing in regression models and asymptotic efficiency in $L_2$., Ann. Statist. 13 984–997.
  • [18] Rigollet, Ph. and Tsybakov, A. (2007). Linear and convex aggregation of density estimators., Math. Methods Statist. 16 260–280.
  • [19] Rigollet, Ph. and Tsybakov, A. (2012). Sparse estimation by exponential weighting., Statist. Sci. 27 558–575.
  • [20] Speckman, P. (1985). Spline smoothing and optimal rates of convergence in nonparametric regression., Ann. Statist. 13 970–983.
  • [21] Stein, C. (1981). Estimation of the mean of a multivariate normal distribution., Ann. Statist. 9 1135–1151.
  • [22] Tikhonov, A. N. and Arsenin, V. A. (1977)., Solution of Ill-posed Problems. Translated from the Russian. Preface by translation editor Fritz John. Scripta Series in Mathematics. V. H. Winston & Sons, Washington, D.C.: John Wiley & Sons, New York.
  • [23] Wahba, G. (1990)., Spline Models for Observational Data. SIAM, Philadelphia.
  • [24] Yang, Y. (2000). Combining different procedures for adaptive regression., J. Multivariate Anal. 74 135–161.