Open Access
February 2021 On the optimality of sliced inverse regression in high dimensions
Qian Lin, Xinran Li, Dongming Huang, Jun S. Liu
Ann. Statist. 49(1): 1-20 (February 2021). DOI: 10.1214/19-AOS1813
Abstract

The central subspace of a pair of random variables $(y,\boldsymbol{x})\in \mathbb{R}^{p+1}$ is the minimal subspace $\mathcal{S}$ such that $y\perp\!\!\!\!\!\perp \boldsymbol{x}|P_{\mathcal{S}}\boldsymbol{x}$. In this paper, we consider the minimax rate of estimating the central space under the multiple index model $y=f(\boldsymbol{\beta }_{1}^{\tau }\boldsymbol{x},\boldsymbol{\beta }_{2}^{\tau }\boldsymbol{x},\ldots,\boldsymbol{\beta }_{d}^{\tau }\boldsymbol{x},\epsilon )$ with at most $s$ active predictors, where $\boldsymbol{x}\sim N(0,\boldsymbol{\Sigma })$ for some class of $\boldsymbol{\Sigma }$. We first introduce a large class of models depending on the smallest nonzero eigenvalue $\lambda $ of $\operatorname{var}(\mathbb{E}[\boldsymbol{x}|y])$, over which we show that an aggregated estimator based on the SIR procedure converges at rate $d\wedge ((sd+s\log (ep/s))/(n\lambda ))$. We then show that this rate is optimal in two scenarios, the single index models and the multiple index models with fixed central dimension $d$ and fixed $\lambda $. By assuming a technical conjecture, we can show that this rate is also optimal for multiple index models with bounded dimension of the central space.

References

1.

Abramovich, F., Benjamini, Y., Donoho, D. L. and Johnstone, I. M. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584–653. MR2281879 1092.62005 10.1214/009053606000000074 euclid.aos/1151418235Abramovich, F., Benjamini, Y., Donoho, D. L. and Johnstone, I. M. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584–653. MR2281879 1092.62005 10.1214/009053606000000074 euclid.aos/1151418235

2.

Amini, A. A. and Wainwright, M. J. (2008). High-dimensional analysis of semidefinite relaxations for sparse principal components. In Information Theory, 2008. ISIT 2008. IEEE International Symposium on 2454–2458. IEEE. MR2541450 10.1214/08-AOS664 euclid.aos/1247836672Amini, A. A. and Wainwright, M. J. (2008). High-dimensional analysis of semidefinite relaxations for sparse principal components. In Information Theory, 2008. ISIT 2008. IEEE International Symposium on 2454–2458. IEEE. MR2541450 10.1214/08-AOS664 euclid.aos/1247836672

3.

Berthet, Q. and Rigollet, P. (2013). Computational lower bounds for sparse pca. Preprint. Available at  arXiv:1304.08281304.0828 1277.62155 euclid.aos/1378386239Berthet, Q. and Rigollet, P. (2013). Computational lower bounds for sparse pca. Preprint. Available at  arXiv:1304.08281304.0828 1277.62155 euclid.aos/1378386239

4.

Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732. 1173.62022 10.1214/08-AOS620 euclid.aos/1245332830Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732. 1173.62022 10.1214/08-AOS620 euclid.aos/1245332830

5.

Birnbaum, A., Johnstone, I. M., Nadler, B. and Paul, D. (2013). Minimax bounds for sparse PCA with noisy high-dimensional data. Ann. Statist. 41 1055–1084. 1292.62071 10.1214/12-AOS1014 euclid.aos/1371150893Birnbaum, A., Johnstone, I. M., Nadler, B. and Paul, D. (2013). Minimax bounds for sparse PCA with noisy high-dimensional data. Ann. Statist. 41 1055–1084. 1292.62071 10.1214/12-AOS1014 euclid.aos/1371150893

6.

Cai, T. T., Ma, Z. and Wu, Y. (2013). Sparse PCA: Optimal rates and adaptive estimation. Ann. Statist. 41 3074–3110. 1288.62099 euclid.aos/1388545679Cai, T. T., Ma, Z. and Wu, Y. (2013). Sparse PCA: Optimal rates and adaptive estimation. Ann. Statist. 41 3074–3110. 1288.62099 euclid.aos/1388545679

7.

Cai, T. T. and Zhou, H. H. (2012). Optimal rates of convergence for sparse covariance matrix estimation. Ann. Statist. 40 2389–2420. 1373.62247 10.1214/12-AOS998 euclid.aos/1359987525Cai, T. T. and Zhou, H. H. (2012). Optimal rates of convergence for sparse covariance matrix estimation. Ann. Statist. 40 2389–2420. 1373.62247 10.1214/12-AOS998 euclid.aos/1359987525

8.

Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351. 1139.62019 10.1214/009053606000001523 euclid.aos/1201012958Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351. 1139.62019 10.1214/009053606000001523 euclid.aos/1201012958

9.

Chen, C.-H. and Li, K.-C. (1998). Can SIR be as popular as multiple linear regression? Statist. Sinica 8 289–316. 0897.62069Chen, C.-H. and Li, K.-C. (1998). Can SIR be as popular as multiple linear regression? Statist. Sinica 8 289–316. 0897.62069

10.

Cook, R. D., Forzani, L. and Rothman, A. J. (2012). Estimating sufficient reductions of the predictors in abundant high-dimensional regressions. Ann. Statist. 40 353–384. 1246.62150 10.1214/11-AOS962 euclid.aos/1333567193Cook, R. D., Forzani, L. and Rothman, A. J. (2012). Estimating sufficient reductions of the predictors in abundant high-dimensional regressions. Ann. Statist. 40 353–384. 1246.62150 10.1214/11-AOS962 euclid.aos/1333567193

11.

Dennis Cook, R. (1998). Regression Graphics: Ideas for Studying Regressions Through Graphics. Wiley Series in Probability and Statistics: Probability and Statistics. Wiley, New York. 0903.62001Dennis Cook, R. (1998). Regression Graphics: Ideas for Studying Regressions Through Graphics. Wiley Series in Probability and Statistics: Probability and Statistics. Wiley, New York. 0903.62001

12.

Dennis Cook, R. (2000). Save: A method for dimension reduction and graphics in regression. Comm. Statist. Theory Methods 29 2109–2121. 1061.62503 10.1080/03610920008832598Dennis Cook, R. (2000). Save: A method for dimension reduction and graphics in regression. Comm. Statist. Theory Methods 29 2109–2121. 1061.62503 10.1080/03610920008832598

13.

Dennis Cook, R. and Weisberg, S. (1991). Comment. J. Amer. Statist. Assoc. 86 328–332. 1353.62037Dennis Cook, R. and Weisberg, S. (1991). Comment. J. Amer. Statist. Assoc. 86 328–332. 1353.62037

14.

Duan, N. and Li, K.-C. (1991). Slicing regression: A link-free regression method. Ann. Statist. 19 505–530. 0738.62070 10.1214/aos/1176348109 euclid.aos/1176348109Duan, N. and Li, K.-C. (1991). Slicing regression: A link-free regression method. Ann. Statist. 19 505–530. 0738.62070 10.1214/aos/1176348109 euclid.aos/1176348109

15.

Ferré, L. (1998). Determining the dimension in sliced inverse regression and related methods. J. Amer. Statist. Assoc. 93 132–140.Ferré, L. (1998). Determining the dimension in sliced inverse regression and related methods. J. Amer. Statist. Assoc. 93 132–140.

16.

Hsing, T. and Carroll, R. J. (1992). An asymptotic theory for sliced inverse regression. Ann. Statist. 20 1040–1061. 0821.62019 10.1214/aos/1176348669 euclid.aos/1176348669Hsing, T. and Carroll, R. J. (1992). An asymptotic theory for sliced inverse regression. Ann. Statist. 20 1040–1061. 0821.62019 10.1214/aos/1176348669 euclid.aos/1176348669

17.

Johnstone, I. M. and Lu, A. Y. (2004). Sparse Principal Components Analysis.Johnstone, I. M. and Lu, A. Y. (2004). Sparse Principal Components Analysis.

18.

Jung, S. and Marron, J. S. (2009). PCA consistency in high dimension, low sample size context. Ann. Statist. 37 4104–4130. 1191.62108 10.1214/09-AOS709 euclid.aos/1256303538Jung, S. and Marron, J. S. (2009). PCA consistency in high dimension, low sample size context. Ann. Statist. 37 4104–4130. 1191.62108 10.1214/09-AOS709 euclid.aos/1256303538

19.

Li, K.-C. (1991). Sliced inverse regression for dimension reduction. J. Amer. Statist. Assoc. 86 316–342. 0742.62044 10.1080/01621459.1991.10475035Li, K.-C. (1991). Sliced inverse regression for dimension reduction. J. Amer. Statist. Assoc. 86 316–342. 0742.62044 10.1080/01621459.1991.10475035

20.

Li, K.-C. (2000). High dimensional data analysis via the SIR/PHD approach.Li, K.-C. (2000). High dimensional data analysis via the SIR/PHD approach.

21.

Li, L. (2007). Sparse sufficient dimension reduction. Biometrika 94 603–613. 1135.62062 10.1093/biomet/asm044Li, L. (2007). Sparse sufficient dimension reduction. Biometrika 94 603–613. 1135.62062 10.1093/biomet/asm044

22.

Li, L. and Nachtsheim, C. J. (2006). Sparse sliced inverse regression. Technometrics 48 503–510.Li, L. and Nachtsheim, C. J. (2006). Sparse sliced inverse regression. Technometrics 48 503–510.

23.

Li, B. and Wang, S. (2007). On directional regression for dimension reduction. J. Amer. Statist. Assoc. 102 997–1008. 05564427 10.1198/016214507000000536Li, B. and Wang, S. (2007). On directional regression for dimension reduction. J. Amer. Statist. Assoc. 102 997–1008. 05564427 10.1198/016214507000000536

24.

Lin, Q., Zhao, Z. and Liu, J. S. (2018). On consistency and sparsity for sliced inverse regression in high dimensions. Ann. Statist. 46 580–610. 1395.62196 10.1214/17-AOS1561 euclid.aos/1522742430Lin, Q., Zhao, Z. and Liu, J. S. (2018). On consistency and sparsity for sliced inverse regression in high dimensions. Ann. Statist. 46 580–610. 1395.62196 10.1214/17-AOS1561 euclid.aos/1522742430

25.

Lin, Q., Zhao, Z. and Liu, J. S. (2019). Sparse sliced inverse regression via Lasso. J. Amer. Statist. Assoc. 114 1726–1739. 1428.62320 10.1080/01621459.2018.1520115Lin, Q., Zhao, Z. and Liu, J. S. (2019). Sparse sliced inverse regression via Lasso. J. Amer. Statist. Assoc. 114 1726–1739. 1428.62320 10.1080/01621459.2018.1520115

26.

Lin, Q., Li, X., Huang, D. and Liu, J. S. (2021). Supplement to “On the optimality of sliced inverse regression in high dimensions.”  https://doi.org/10.1214/19-AOS1813SUPPLin, Q., Li, X., Huang, D. and Liu, J. S. (2021). Supplement to “On the optimality of sliced inverse regression in high dimensions.”  https://doi.org/10.1214/19-AOS1813SUPP

27.

Neykov, M., Lin, Q. and Liu, J. S. (2016). Signed support recovery for single index models in high-dimensions. Ann. Math. Sci. Appl. 1 379–426. MR3876481 1392.62175 10.4310/AMSA.2016.v1.n2.a5Neykov, M., Lin, Q. and Liu, J. S. (2016). Signed support recovery for single index models in high-dimensions. Ann. Math. Sci. Appl. 1 379–426. MR3876481 1392.62175 10.4310/AMSA.2016.v1.n2.a5

28.

Raskutti, G., Wainwright, M. J. and Yu, B. (2011). Minimax rates of estimation for high-dimensional linear regression over $\ell_{q}$-balls. IEEE Trans. Inf. Theory 57 6976–6994. 1365.62276 10.1109/TIT.2011.2165799Raskutti, G., Wainwright, M. J. and Yu, B. (2011). Minimax rates of estimation for high-dimensional linear regression over $\ell_{q}$-balls. IEEE Trans. Inf. Theory 57 6976–6994. 1365.62276 10.1109/TIT.2011.2165799

29.

Schott, J. R. (1994). Determining the dimensionality in sliced inverse regression. J. Amer. Statist. Assoc. 89 141–148. 0791.62069 10.1080/01621459.1994.10476455Schott, J. R. (1994). Determining the dimensionality in sliced inverse regression. J. Amer. Statist. Assoc. 89 141–148. 0791.62069 10.1080/01621459.1994.10476455

30.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288. 0850.62538 10.1111/j.2517-6161.1996.tb02080.xTibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288. 0850.62538 10.1111/j.2517-6161.1996.tb02080.x

31.

Vu, V. Q. and Lei, J. (2012). Minimax rates of estimation for sparse pca in high dimensions. Preprint. Available at  arXiv:1202.07861202.0786Vu, V. Q. and Lei, J. (2012). Minimax rates of estimation for sparse pca in high dimensions. Preprint. Available at  arXiv:1202.07861202.0786

32.

Yang, S. S. (1977). General distribution theory of the concomitants of order statistics. Ann. Statist. 5 996–1002. 0367.62017 10.1214/aos/1176343954 euclid.aos/1176343954Yang, S. S. (1977). General distribution theory of the concomitants of order statistics. Ann. Statist. 5 996–1002. 0367.62017 10.1214/aos/1176343954 euclid.aos/1176343954

33.

Zhu, L., Miao, B. and Peng, H. (2006). On sliced inverse regression with high-dimensional covariates. J. Amer. Statist. Assoc. 101 630–643. 1119.62331 10.1198/016214505000001285Zhu, L., Miao, B. and Peng, H. (2006). On sliced inverse regression with high-dimensional covariates. J. Amer. Statist. Assoc. 101 630–643. 1119.62331 10.1198/016214505000001285

34.

Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 301–320. 1069.62054 10.1111/j.1467-9868.2005.00503.xZou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 301–320. 1069.62054 10.1111/j.1467-9868.2005.00503.x
Copyright © 2021 Institute of Mathematical Statistics
Qian Lin, Xinran Li, Dongming Huang, and Jun S. Liu "On the optimality of sliced inverse regression in high dimensions," The Annals of Statistics 49(1), 1-20, (February 2021). https://doi.org/10.1214/19-AOS1813
Received: 1 April 2018; Published: February 2021
Vol.49 • No. 1 • February 2021
Back to Top