The Annals of Statistics

Marginal empirical likelihood and sure independence feature screening

Jinyuan Chang, Cheng Yong Tang, and Yichao Wu

Full-text: Open access


We study a marginal empirical likelihood approach in scenarios when the number of variables grows exponentially with the sample size. The marginal empirical likelihood ratios as functions of the parameters of interest are systematically examined, and we find that the marginal empirical likelihood ratio evaluated at zero can be used to differentiate whether an explanatory variable is contributing to a response variable or not. Based on this finding, we propose a unified feature screening procedure for linear models and the generalized linear models. Different from most existing feature screening approaches that rely on the magnitudes of some marginal estimators to identify true signals, the proposed screening approach is capable of further incorporating the level of uncertainties of such estimators. Such a merit inherits the self-studentization property of the empirical likelihood approach, and extends the insights of existing feature screening methods. Moreover, we show that our screening approach is less restrictive to distributional assumptions, and can be conveniently adapted to be applied in a broad range of scenarios such as models specified using general moment conditions. Our theoretical results and extensive numerical examples by simulations and data analysis demonstrate the merits of the marginal empirical likelihood approach.

Article information

Ann. Statist., Volume 41, Number 4 (2013), 2123-2148.

First available in Project Euclid: 23 October 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G09: Resampling methods
Secondary: 62H99: None of the above, but in this section

Empirical likelihood high-dimensional data analysis sure independence screening large deviation


Chang, Jinyuan; Tang, Cheng Yong; Wu, Yichao. Marginal empirical likelihood and sure independence feature screening. Ann. Statist. 41 (2013), no. 4, 2123--2148. doi:10.1214/13-AOS1139.

Export citation


  • Bühlmann, P. and van de Geer, S. (2011). Statistics for High-dimensional Data: Methods, Theory and Applications. Springer, Heidelberg.
  • Chang, J., Chen, S. X. and Chen, X. (2013). High dimensional generalized empirical likelihood for moment restrictions with dependent data. Available at arXiv:1308.5732.
  • Chang, J., Tang, C. Y. and Wu, Y. (2013). Supplement to “Marginal empirical likelihood and sure independence feature screening.” DOI:10.1214/13-AOS1139SUPP.
  • Chen, S. X. and Cui, H. (2003). An extended empirical likelihood for generalized linear models. Statist. Sinica 13 69–81.
  • Chen, S. X., Gao, J. and Tang, C. Y. (2008). A test for model specification of diffusion processes. Ann. Statist. 36 167–198.
  • Chen, S. X., Peng, L. and Qin, Y.-L. (2009). Effects of data dimension on empirical likelihood. Biometrika 96 711–722.
  • Chen, S. X. and Van Keilegom, I. (2009). A review on empirical likelihood methods for regression. TEST 18 415–447.
  • Diggle, P. J., Heagerty, P. J., Liang, K.-Y. and Zeger, S. L. (2002). Analysis of Longitudinal Data, 2nd ed. Oxford Statistical Science Series 25. Oxford Univ. Press, Oxford.
  • Fan, J., Feng, Y. and Song, R. (2011). Nonparametric independence screening in sparse ultra-high-dimensional additive models. J. Amer. Statist. Assoc. 106 544–557.
  • Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 849–911.
  • Fan, J. and Lv, J. (2010). A selective overview of variable selection in high dimensional feature space. Statist. Sinica 20 101–148.
  • Fan, J. and Song, R. (2010). Sure independence screening in generalized linear models with NP-dimensionality. Ann. Statist. 38 3567–3604.
  • Hansen, L. P. (1982). Large sample properties of generalized method of moments estimators. Econometrica 50 1029–1054.
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.
  • Hjort, N. L., McKeague, I. W. and Van Keilegom, I. (2009). Extending the scope of empirical likelihood. Ann. Statist. 37 1079–1111.
  • Horvath, S., Zhang, B., Carlson, M., Lu, K. V., Zhu, S., Felciano, R. M., Laurance, M. F., Zhao, W., Shu, Q., Lee, Y., Scheck, A. C., Liau, L. M., Wu, H., Geschwind, D. H., Febbo, P. G., Kornblum, H. I., Cloughesy, T. F., Nelson, S. F. and Mischel, P. S. (2006). Analysis of oncogenic signaling networks in glioblastoma identifies ASPM as a molecular target. Proc. Natl. Acad. Sci. USA 103 17402–17407.
  • Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36 587–613.
  • Kolaczyk, E. D. (1994). Empirical likelihood for generalized linear models. Statist. Sinica 4 199–218.
  • Leng, C. and Tang, C. Y. (2012). Penalized empirical likelihood and growing dimensional general estimating equations. Biometrika 99 703–716.
  • Li, C. and Li, H. (2008). Network-constrained regularization and variable selection for analysis of genomic data. Bioinformatics 24 1175–1182.
  • Li, R., Zhong, W. and Zhu, L. (2012). Feature screening via distance correlation learning. J. Amer. Statist. Assoc. 107 1129–1139.
  • Li, G., Peng, H., Zhang, J. and Zhu, L. (2012). Robust rank correlation based screening. Ann. Statist. 40 1846–1877.
  • Liang, K. Y. and Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73 13–22.
  • McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models. Chapman & Hall/CRC, New York.
  • Newey, W. K. and Smith, R. J. (2004). Higher order properties of GMM and generalized empirical likelihood estimators. Econometrica 72 219–255.
  • Owen, A. B. (1988). Empirical likelihood ratio confidence intervals for a single functional. Biometrika 75 237–249.
  • Owen, A. B. (2001). Empirical Likelihood. Chapman & Hall/CRC, New York.
  • Pan, W., Xie, B. and Shen, X. (2010). Incorporating predictor network in penalized regression with application to microarray data. Biometrics 66 474–484.
  • Petrov, V. V. (1995). Limit Theorems of Probability Theory: Sequences of Independent Random Variables. Oxford Studies in Probability 4. Oxford Univ. Press, New York.
  • Qin, J. and Lawless, J. (1994). Empirical likelihood and general estimating equations. Ann. Statist. 22 300–325.
  • Qu, A., Lindsay, B. G. and Li, B. (2000). Improving generalised estimating equations using quadratic inference functions. Biometrika 87 823–836.
  • Saulis, L. and Statulevičius, V. A. (1991). Limit Theorems for Large Deviations. Mathematics and Its Applications (Soviet Series) 73. Kluwer Academic, Dordrecht. Translated and revised from the 1989 Russian original.
  • Tang, C. Y. and Leng, C. (2010). Penalized high-dimensional empirical likelihood. Biometrika 97 905–919. With supplementary material available online.
  • Wang, H. (2012). Factor profiled sure independence screening. Biometrika 99 15–28.
  • Xue, L. and Zou, H. (2011). Sure independence screening and compressed random sensing. Biometrika 98 371–380.
  • Zhao, S. D. and Li, Y. (2012). Sure screening for estimating equations in ultra-high dimensions. Unpublished manuscript.
  • Zhu, L.-P., Li, L., Li, R. and Zhu, L.-X. (2011). Model-free feature screening for ultrahigh-dimensional data. J. Amer. Statist. Assoc. 106 1464–1475.

Supplemental materials