Electronic Journal of Statistics

Sparsity considerations for dependent variables

Pierre Alquier and Paul Doukhan

Full-text: Open access

Abstract

The aim of this paper is to provide a comprehensive introduction for the study of 1-penalized estimators in the context of dependent observations. We define a general 1-penalized estimator for solving problems of stochastic optimization. This estimator turns out to be the LASSO [Tib96] in the regression estimation setting. Powerful theoretical guarantees on the statistical performances of the LASSO were provided in recent papers, however, they usually only deal with the iid case. Here, we study this estimator under various dependence assumptions.

Article information

Source
Electron. J. Statist. Volume 5 (2011), 750-774.

Dates
First available in Project Euclid: 8 August 2011

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1312818917

Digital Object Identifier
doi:10.1214/11-EJS626

Mathematical Reviews number (MathSciNet)
MR2824815

Zentralblatt MATH identifier
1274.62462

Subjects
Primary: 62J07: Ridge regression; shrinkage estimators
Secondary: 62M10: Time series, auto-correlation, regression, etc. [See also 91B84] 62J05: Linear regression 62G07: Density estimation 62G08: Nonparametric regression

Keywords
Estimation in high dimension weak dependence sparsity deviation of empirical mean penalization LASSO regression estimation density estimation

Citation

Alquier, Pierre; Doukhan, Paul. Sparsity considerations for dependent variables. Electron. J. Statist. 5 (2011), 750--774. doi:10.1214/11-EJS626. https://projecteuclid.org/euclid.ejs/1312818917


Export citation

References

  • [Aka73] H. Akaike. Information theory and an extension of the maximum likelihood principle. In B. N. Petrov and F. Csaki, editors, 2nd International Symposium on Information Theory, pages 267–281. Budapest: Akademia Kiado, 1973.
  • [Alq08] P. Alquier. Density estimation with quadratic loss, a confidence intervals method., ESAIM: P&S, 12:438–463, 2008.
  • [BC11a] A. Belloni and V. Chernozhukov., 1-penalized quantile regression in high-dimensional sparse models. Annals of Statistics, 32(11) :2011–2055, 2011.
  • [BC11b] A. Belloni and V. Chernozhukov. High dimensional sparse econometric models: An introduction. In P. Alquier, E. Gautier, and G. Stoltz, editors, Inverse Problems and High-Dimensional Estimation. Springer Lecture Notes in Statistics, 2011.
  • [BGH09] Y. Baraud, C. Giraud, and S. Huet. Gaussian model selection with an unknown variance., Annals of Statistics, 37(2):630–672, 2009.
  • [BJMW10] K. Bartkiewicz, A. Jakubowskin, T. Mikosch, and O. Wintenberger. Infinite variances stable limits for sums of dependent random variables., Probability Theory and Related Fields, 2010.
  • [BM01] L. Birgé and P. Massart. Gaussian model selection., Journal of the European Mathematical Society, 3(3):203–268, 2001.
  • [BRT09] P. J. Bickel, Y. Ritov, and A. Tsybakov. Simultaneous analysis of lasso and Dantzig selector., Annals of Statistics, 37(4) :1705–1732, 2009.
  • [BTW07] F. Bunea, A.B. Tsybakov, and M.H. Wegkamp. Aggregation for Gaussian regression., Annals of Statistics, 35 :1674–1697, 2007.
  • [BTWB10] F. Bunea, A. Tsybakov, M. Wegkamp, and A. Barbu. SPADES and mixture models., Annals of Statistics, 38(4) :2525–2558, 2010.
  • [BWT07] F. Bunea, M. Wegkamp, and A. Tsybakov. Sparse density estimation with, 1 penalties. Proceedings of 20th Annual Conference on Learning Theory (COLT 2007) - Springer, pages 530–543, 2007.
  • [CDS01] S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit., SIAM Review, 43(1):129–159, 2001.
  • [Che95] S. S. Chen. Basis pursuit, 1995. PhD Thesis, Stanford, University.
  • [CT07] E. Candes and T. Tao. The Dantzig selector: statistical estimation when, p is much larger than n. Annals of Statistics, 35, 2007.
  • [DDL+07] J. Dedecker, P. Doukhan, G. Lang, J. R. León R., S. Louhichi, and C. Prieur., Weak dependence: with examples and applications, volume 190 of Lecture Notes in Statistics. Springer, New York, 2007.
  • [DL99] P. Doukhan and S. Louhichi. A new weak dependence condition and applications to moment inequalities., Stochastic Processes and their Applications, 84(2):313–342, 1999.
  • [DN07] P. Doukhan and M. H. Neumann. Probability and moment inequalities for sums of weakly dependent random variables, with applications., Stochastic Processes and their Applications, 117(7):878–903, 2007.
  • [Dou94] P. Doukhan., Mixing, volume 85 of Lecture Notes in Statistics. Springer-Verlag, New York, 1994. Properties and examples.
  • [EHJT04] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression., Annals of Statistics, 32(2):407–499, 2004.
  • [FHHT07] J. Friedman, T. Hastie, H. Höfling, and R. Tibshirani. Pathwise coordinate optimization., Annals of Applied Statist., 1(2):302–332, 2007.
  • [Heb09] M. Hebiri. Quelques questions de selection de variables autour de l’estimateur lasso, 2009. PhD Thesis, Université Paris 7 (in, english).
  • [Hoe63] W. Hoeffding. Probability inequalities for sums of bounded random variables., Journal of the American Statistical Association, 58:13–30, 1963.
  • [Kol] V. Koltchinskii. Sparsity in empirical risk minimization., Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 45(1):7–57, 2009.
  • [LP05] H. Leeb and B. M. Pötscher. Sparse estimators and the oracle property, or the return of hodges’ estimator. Cowles Foundation Discussion Papers 1500, Cowles Foundation, Yale University, 2005.
  • [RBV08] F. Rapaport, E. Barillot, and J.-P. Vert. Classification of array-CGH data using fused SVM., Bioinformatics, 24(13) :1375, 1382, 2008.
  • [Rio00] E. Rio., Théorie asymptotique pour des processus aléatoire faiblement dépendants. SMAI, Mathématiques et Applications 31, Springer, 2000.
  • [Ros56] M. Rosenblatt. A central limit theorem and a strong mixing condition., Proc. Nat. Ac. Sc. U.S.A., 42:43–47, 1956.
  • [Ros61] M. Rosenblatt. Independence and dependence., Proceeding 4th. Berkeley Symp. Math. Stat. Prob. Berkeley University Press, pages 411–443, 1961.
  • [Sch78] G. Schwarz. Estimating the dimension of a model., Annals of Statistics, 6:461–464, 1978.
  • [Tib96] R. Tibshirani. Regression shrinkage and selection via the lasso., Journal of the Royal Statistical Society B, 58(1):267–288, 1996.
  • [vdGB09] S. A. van de Geer and P. Bühlmann. On the conditions used to prove oracle results for the lasso., Electronic Journal of Statistics, 3 :1360–1392, 2009.
  • [Win10] O. Wintenberger. Deviation inequalities for sums of weakly dependent time series., Electronic Communications in Probability, 15:489–503, 2010.
  • [ZH05] H. Zou and T. Hastie. Regularization and variable selection via the elastic net., Journal of the Royal Statistical Society B, 67(2):301–320, 2005.