The Annals of Statistics

Adaptive robust variable selection

Jianqing Fan, Yingying Fan, and Emre Barut

Full-text: Open access

Enhanced PDF (352 KB)

Abstract

Heavy-tailed high-dimensional data are commonly encountered in various scientific fields and pose great challenges to modern statistical analysis. A natural procedure to address this problem is to use penalized quantile regression with weighted $L_{1}$-penalty, called weighted robust Lasso (WR-Lasso), in which weights are introduced to ameliorate the bias problem induced by the $L_{1}$-penalty. In the ultra-high dimensional setting, where the dimensionality can grow exponentially with the sample size, we investigate the model selection oracle property and establish the asymptotic normality of the WR-Lasso. We show that only mild conditions on the model error distribution are needed. Our theoretical results also reveal that adaptive choice of the weight vector is essential for the WR-Lasso to enjoy these nice asymptotic properties. To make the WR-Lasso practically feasible, we propose a two-step procedure, called adaptive robust Lasso (AR-Lasso), in which the weight vector in the second step is constructed based on the $L_{1}$-penalized quantile regression estimate from the first step. This two-step procedure is justified theoretically to possess the oracle property and the asymptotic normality. Numerical studies demonstrate the favorable finite-sample performance of the AR-Lasso.

Article information

Source
Ann. Statist. Volume 42, Number 1 (2014), 324-351.

Dates
First available in Project Euclid: 19 March 2014

Permanent link to this document
http://projecteuclid.org/euclid.aos/1395234980

Digital Object Identifier
doi:10.1214/13-AOS1191

Mathematical Reviews number (MathSciNet)
MR3189488

Zentralblatt MATH identifier
1296.62144

Subjects
Primary: 62J07: Ridge regression; shrinkage estimators
Secondary: 62H12: Estimation

Keywords
Adaptive weighted $L_{1}$ high dimensions oracle properties robust regularization

Citation

Fan, Jianqing; Fan, Yingying; Barut, Emre. Adaptive robust variable selection. Ann. Statist. 42 (2014), no. 1, 324--351. doi:10.1214/13-AOS1191. http://projecteuclid.org/euclid.aos/1395234980.


Export citation

References

  • Belloni, A. and Chernozhukov, V. (2011). $\ell_1$-penalized quantile regression in high-dimensional sparse models. Ann. Statist. 39 82–130.
    Mathematical Reviews (MathSciNet): MR2797841
    Digital Object Identifier: doi:10.1214/10-AOS827
    Project Euclid: euclid.aos/1291388370
  • Bickel, P. J. and Li, B. (2006). Regularization in statistics. TEST 15 271–344. With comments and a rejoinder by the authors.
    Mathematical Reviews (MathSciNet): MR2273731
    Digital Object Identifier: doi:10.1007/BF02607055
  • Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
    Mathematical Reviews (MathSciNet): MR2533469
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • Bradic, J., Fan, J. and Wang, W. (2011). Penalized composite quasi-likelihood for ultrahigh dimensional variable selection. J. R. Stat. Soc. Ser. B Stat. Methodol. 73 325–349.
    Mathematical Reviews (MathSciNet): MR2815779
    Digital Object Identifier: doi:10.1111/j.1467-9868.2010.00764.x
  • Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Heidelberg.
    Mathematical Reviews (MathSciNet): MR2807761
  • Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351.
    Mathematical Reviews (MathSciNet): MR2382644
    Digital Object Identifier: doi:10.1214/009053606000001523
    Project Euclid: euclid.aos/1201012958
  • Fan, J., Fan, Y. and Barut, E. (2014). Supplement to “Adaptive robust variable selection.” DOI:10.1214/13-AOS1191SUPP.
  • Fan, J., Fan, Y. and Lv, J. (2008). High dimensional covariance matrix estimation using a factor model. J. Econometrics 147 186–197.
    Mathematical Reviews (MathSciNet): MR2472991
    Digital Object Identifier: doi:10.1016/j.jeconom.2008.09.017
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Digital Object Identifier: doi:10.1198/016214501753382273
  • Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 849–911.
    Mathematical Reviews (MathSciNet): MR2530322
    Digital Object Identifier: doi:10.1111/j.1467-9868.2008.00674.x
  • Fan, J. and Lv, J. (2011). Nonconcave penalized likelihood with $np$-dimensionality. IEEE Trans. Inform. Theory 57 5467–5484.
    Mathematical Reviews (MathSciNet): MR2849368
    Digital Object Identifier: doi:10.1109/TIT.2011.2158486
  • Fan, J. and Peng, H. (2004). Nonconcave penalized likelihood with a diverging number of parameters. Ann. Statist. 32 928–961.
    Mathematical Reviews (MathSciNet): MR2065194
    Digital Object Identifier: doi:10.1214/009053604000000256
    Project Euclid: euclid.aos/1085408491
  • Li, Y. and Zhu, J. (2008). $L_1$-norm quantile regression. J. Comput. Graph. Statist. 17 163–185.
    Mathematical Reviews (MathSciNet): MR2424800
    Digital Object Identifier: doi:10.1198/106186008X289155
  • Lv, J. and Fan, Y. (2009). A unified approach to model selection and sparse recovery using regularized least squares. Ann. Statist. 37 3498–3528.
    Mathematical Reviews (MathSciNet): MR2549567
    Digital Object Identifier: doi:10.1214/09-AOS683
    Project Euclid: euclid.aos/1250515394
  • Meinshausen, N. and Bühlmann, P. (2010). Stability selection. J. R. Stat. Soc. Ser. B Stat. Methodol. 72 417–473.
    Mathematical Reviews (MathSciNet): MR2758523
    Digital Object Identifier: doi:10.1111/j.1467-9868.2010.00740.x
  • Newey, W. K. and Powell, J. L. (1990). Efficient estimation of linear and type I censored regression models under conditional quantile restrictions. Econometric Theory 6 295–317.
    Mathematical Reviews (MathSciNet): MR1085576
    Digital Object Identifier: doi:10.1017/S0266466600005284
  • Nolan, J. P. (2012). Stable Distributions—Models for Heavy-Tailed Data. Birkhauser, Cambridge. (In progress, Chapter 1 online at academic2.american.edu/~jpnolan).
  • Pollard, D. (1991). Asymptotics for least absolute deviation regression estimators. Econometric Theory 7 186–199.
    Mathematical Reviews (MathSciNet): MR1128411
    Digital Object Identifier: doi:10.1017/S0266466600004394
  • Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • van de Geer, S. and Müller, P. (2012). Quasi-likelihood and/or robust estimation in high dimensions. Statist. Sci. 27 469–480.
    Mathematical Reviews (MathSciNet): MR3025129
    Digital Object Identifier: doi:10.1214/12-STS397
    Project Euclid: euclid.ss/1356098551
  • Wang, L. (2013). $L_1$ penalized LAD estimator for high dimensional linear regression. J. Multivariate Anal. 120 135–151.
    Mathematical Reviews (MathSciNet): MR3072722
    Digital Object Identifier: doi:10.1016/j.jmva.2013.04.001
  • Wang, H., Li, G. and Jiang, G. (2007). Robust regression shrinkage and consistent variable selection through the LAD-Lasso. J. Bus. Econom. Statist. 25 347–355.
    Mathematical Reviews (MathSciNet): MR2380753
  • Wang, L., Wu, Y. and Li, R. (2012). Quantile regression for analyzing heterogeneity in ultra-high dimension. J. Amer. Statist. Assoc. 107 214–222.
    Mathematical Reviews (MathSciNet): MR2949353
    Digital Object Identifier: doi:10.1080/01621459.2012.656014
  • Wu, Y. and Liu, Y. (2009). Variable selection in quantile regression. Statist. Sinica 19 801–817.
    Mathematical Reviews (MathSciNet): MR2514189
  • Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894–942.
    Mathematical Reviews (MathSciNet): MR2604701
    Digital Object Identifier: doi:10.1214/09-AOS729
    Project Euclid: euclid.aos/1266586618
  • Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541–2563.
    Mathematical Reviews (MathSciNet): MR2274449
  • Zou, H. (2006). The adaptive Lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418–1429.
    Mathematical Reviews (MathSciNet): MR2279469
    Digital Object Identifier: doi:10.1198/016214506000000735
  • Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models. Ann. Statist. 36 1509–1533.
    Mathematical Reviews (MathSciNet): MR2435443
    Digital Object Identifier: doi:10.1214/009053607000000802
    Project Euclid: euclid.aos/1216237287
  • Zou, H. and Yuan, M. (2008). Composite quantile regression and the oracle model selection theory. Ann. Statist. 36 1108–1126.
    Mathematical Reviews (MathSciNet): MR2418651
    Digital Object Identifier: doi:10.1214/07-AOS507
    Project Euclid: euclid.aos/1211819558

Supplemental materials

  • Supplementary material: Supplementary material for: Adaptive robust variable selection. Due to space constraints, the proofs of Theorems 3 and 5 and the results of the real life data-set study are relegated to the supplement [Fan, Fan and Barut (2014)].
    Digital Object Identifier: doi:10.1214/13-AOS1191SUPP
    Supplemental PDF (175 KB)

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more