The Annals of Statistics

Estimating sufficient reductions of the predictors in abundant high-dimensional regressions

R. Dennis Cook, Liliana Forzani, and Adam J. Rothman

Full-text: Open access

Abstract

We study the asymptotic behavior of a class of methods for sufficient dimension reduction in high-dimension regressions, as the sample size and number of predictors grow in various alignments. It is demonstrated that these methods are consistent in a variety of settings, particularly in abundant regressions where most predictors contribute some information on the response, and oracle rates are possible. Simulation results are presented to support the theoretical conclusion.

Article information

Source
Ann. Statist. Volume 40, Number 1 (2012), 353-384.

Dates
First available in Project Euclid: 4 April 2012

Permanent link to this document
http://projecteuclid.org/euclid.aos/1333567193

Digital Object Identifier
doi:10.1214/11-AOS962

Mathematical Reviews number (MathSciNet)
MR3014310

Zentralblatt MATH identifier
06075618

Subjects
Primary: 62H20: Measures of association (correlation, canonical correlation, etc.)
Secondary: 62J07: Ridge regression; shrinkage estimators

Keywords
Central subspace oracle property SPICE sparsity sufficient dimension reduction principal fitted components

Citation

Cook, R. Dennis; Forzani, Liliana; Rothman, Adam J. Estimating sufficient reductions of the predictors in abundant high-dimensional regressions. Ann. Statist. 40 (2012), no. 1, 353--384. doi:10.1214/11-AOS962. http://projecteuclid.org/euclid.aos/1333567193.


Export citation

References

  • Adragni, K. P. and Cook, R. D. (2009). Sufficient dimension reduction and prediction in regression. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4385–4405.
  • Bai, Z. D. (1999). Methodologies in spectral analysis of large-dimensional random matrices, a review. Statist. Sinica 9 611–677.
  • Bickel, P. J. and Levina, E. (2008a). Regularized estimation of large covariance matrices. Ann. Statist. 36 199–227.
  • Bickel, P. J. and Levina, E. (2008b). Covariance regularization by thresholding. Ann. Statist. 36 2577–2604.
  • Bondell, H. D. and Li, L. (2009). Shrinkage inverse regression estimation for model-free variable selection. J. R. Stat. Soc. Ser. B Stat. Methodol. 71 287–299.
  • Bura, E. and Cook, R. D. (2001). Extending sliced inverse regression: The weighted chi-squared test. J. Amer. Statist. Assoc. 96 996–1003.
  • Chiaromonte, F. and Martinelli, J. (2002). Dimension reduction strategies for analyzing global gene expression data with a response. Math. Biosci. 176 123–144.
  • Chun, H. and Keleş, S. (2010). Sparse partial least squares regression for simultaneous dimension reduction and variable selection. J. R. Stat. Soc. Ser. B Stat. Methodol. 72 3–25.
  • Cook, R. D. (1994). Using dimension-reduction subspaces to identify important inputs in models of physical systems. In Proceedings of the Section on Physical and Engineering Sciences 18–25. Amer. Statist. Assoc., Alexandria, VA.
  • Cook, R. D. (1998). Regression Graphics: Ideas for Studying Regressions Through Graphics. Wiley, New York.
  • Cook, R. D. (2004). Testing predictor contributions in sufficient dimension reduction. Ann. Statist. 32 1062–1092.
  • Cook, R. D. (2007). Fisher lecture: Dimension reduction in regression (with discussion). Statist. Sci. 22 1–26.
  • Cook, R. D. and Forzani, L. (2008). Principal fitted components for dimension reduction in regression. Statist. Sci. 23 485–501.
  • Cook, R. D. and Forzani, L. (2009). Likelihood-based sufficient dimension reduction. J. Amer. Statist. Assoc. 104 197–208.
  • Cook, R. D., Forzani, L. and Rothman, A. J. (2012). Supplement to “Estimating sufficient reductions of the predictors in abundant high-dimensional regressions.” DOI:10.1214/11-AOS962SUPP.
  • Cook, R. D., Li, B. and Chiaromonte, F. (2007). Dimension reduction in regression without matrix inversion. Biometrika 94 569–584.
  • Cook, R. D. and Ni, L. (2005). Sufficient dimension reduction via inverse regression: A minimum discrepancy approach. J. Amer. Statist. Assoc. 100 410–428.
  • Cook, R. D. and Weisberg, S. (1991). Discussion of “Sliced inverse regression for dimension reduction” by K.-C. Li. J. Amer. Statist. Assoc. 86 382–332.
  • Cook, R. D. and Yin, X. (2001). Dimension reduction and visualization in discriminant analysis. Aust. N. Z. J. Stat. 43 901–999.
  • d’Aspremont, A., Banerjee, O. and El Ghaoui, L. (2008). First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30 56–66.
  • Dong, Y. and Li, B. (2010). Dimension reduction for non-elliptically distributed predictors: Second-order methods. Biometrika 97 279–294.
  • Friedman, J., Hastie, T. and Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9 432–441.
  • Friedman, J., Hastie, T., Rosset, R., Tibshirani, R. and Zhu, J. (2004). Consistency in boosting: Discussion. Ann. Statist. 32 102–107.
  • Fukumizu, K., Bach, F. R. and Jordan, M. I. (2009). Kernel dimension reduction in regression. Ann. Statist. 37 1871–1905.
  • Guan, Y. and Wang, H. (2010). Sufficient dimension reduction for spatial point processes directed by Gaussian random fields. J. R. Stat. Soc. Ser. B Stat. Methodol. 72 367–387.
  • Hall, P. and Li, K.-C. (1993). On almost linearity of low-dimensional projections from high-dimensional data. Ann. Statist. 21 867–889.
  • Johnstone, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29 295–327.
  • Johnstone, I. M. and Lu, A. Y. (2009). On consistency and sparsity for principal components analysis in high dimensions. J. Amer. Statist. Assoc. 104 682–693.
  • Lam, C. and Fan, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Statist. 37 4254–4278.
  • Li, K.-C. (1991). Sliced inverse regression for dimension reduction (with discussion). J. Amer. Statist. Assoc. 86 316–342.
  • Li, B. and Dong, Y. (2009). Dimension reduction for nonelliptically distributed predictors. Ann. Statist. 37 1272–1298.
  • Li, L. and Li, H. (2004). Dimension reduction methods for microarrays with application to censored survival data. Bioinformatics 20 3406–3412.
  • Li, L. and Yin, X. (2008). Sliced inverse regression with regularizations. Biometrics 64 124–131.
  • Paul, D. (2005). Nonparametric estimation of principal components. Ph.D. thesis, Dept. Statistics, Stanford Univ.
  • Ravikumar, P., Wainwright, M. J., Raskutti, G. and Yu, B. (2011). High-dimensional covariance estimation by minimizing 1-penalized log-determinant divergence. Electron. J. Stat. 5 935–980.
  • Rothman, A. J., Bickel, P. J., Levina, E. and Zhu, J. (2008). Sparse permutation invariant covariance estimation. Electron. J. Stat. 2 494–515.
  • Sæbø, S., Almøy, T., Aarøe, J. and Aastveit, A. H. (2007). ST-PLS: A multi-directional nearest shrunken centroid type classifier via PLS. Journal of Chemometrics 20 54–62.
  • von Rosen, D. (1988). The inverted Wishart distribution. Scand. J. Stat. 15 97–109.
  • Witten, D. M. and Tibshirani, R. (2009). Covariance-regularized regression and classification for high dimensional problems. J. R. Stat. Soc. Ser. B Stat. Methodol. 71 615–636.
  • Wu, Y. and Li, L. (2011). Asymptotic properties of sufficient dimension reduction with a diverging number of predictors. Statist. Sinica 21 707–730.
  • Xia, Y., Zhang, D. and Xu, J. (2010). Dimension reduction and semiparametric estimation of survival models. J. Amer. Statist. Assoc. 105 278–290.
  • Yuan, M. and Lin, Y. (2007). Model selection and estimation in the Gaussian graphical model. Biometrika 94 19–35.
  • Zhu, L.-P., Zhu, L.-X. and Feng, Z.-H. (2010). Dimension reduction in regressions through cumulative slicing estimation. J. Amer. Statist. Assoc. 105 1455–1466.
  • Zyskind, G. (1967). On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. Ann. Math. Statist. 38 1092–1109.

Supplemental materials

  • Supplementary material: Supplement to “Estimating sufficient reductions of the predictors in abundant high-dimensional regressions”. Owing to space constraints, we have placed the technical proofs in a supplemental article [Cook, Forzani and Rothman (2012)]. The supplement also contains several preparatory technical results that may be of interest in their own right and additional simulations. For instance, we gave in Section 7 simulation results from models with exponentially decreasing error correlations. In the supplemental article we give parallel results based on the same models but with constant error correlations.