We consider the multivariate response regression problem with a regression coefficient matrix of low, unknown rank. In this setting, we analyze a new criterion for selecting the optimal reduced rank. This criterion differs notably from the one proposed in Bunea, She and Wegkamp (Ann. Statist. 39 (2011) 1282–1309) in that it does not require estimation of the unknown variance of the noise, nor does it depend on a delicate choice of a tuning parameter. We develop an iterative, fully data-driven procedure, that adapts to the optimal signal-to-noise ratio. This procedure finds the true rank in a few steps with overwhelming probability. At each step, our estimate increases, while at the same time it does not exceed the true rank. Our finite sample results hold for any sample size and any dimension, even when the number of responses and of covariates grow much faster than the number of observations. We perform an extensive simulation study that confirms our theoretical findings. The new method performs better and is more stable than the procedure of Bunea, She and Wegkamp (Ann. Statist. 39 (2011) 1282–1309) in both low- and high-dimensional settings.
References
Anderson, T. W. (1951). Estimating linear restrictions on regression coefficients for multivariate normal distributions. Ann. Math. Stat. 22 327–351. MR42664 0043.13902 10.1214/aoms/1177729580 euclid.aoms/1177729580Anderson, T. W. (1951). Estimating linear restrictions on regression coefficients for multivariate normal distributions. Ann. Math. Stat. 22 327–351. MR42664 0043.13902 10.1214/aoms/1177729580 euclid.aoms/1177729580
Anderson, T. W. (1999). Asymptotic distribution of the reduced rank regression estimator under general conditions. Ann. Statist. 27 1141–1154. 0961.62011 10.1214/aos/1017938918 euclid.aos/1017938918Anderson, T. W. (1999). Asymptotic distribution of the reduced rank regression estimator under general conditions. Ann. Statist. 27 1141–1154. 0961.62011 10.1214/aos/1017938918 euclid.aos/1017938918
Anderson, T. W. (2002). Specification and misspecification in reduced rank regression. Sankhyā Ser. A 64 193–205. MR1981753 1192.62145Anderson, T. W. (2002). Specification and misspecification in reduced rank regression. Sankhyā Ser. A 64 193–205. MR1981753 1192.62145
Bai, Z. D. and Yin, Y. Q. (1993). Limit of the smallest eigenvalue of a large-dimensional sample covariance matrix. Ann. Probab. 21 1275–1294. 0779.60026 10.1214/aop/1176989118 euclid.aop/1176989118Bai, Z. D. and Yin, Y. Q. (1993). Limit of the smallest eigenvalue of a large-dimensional sample covariance matrix. Ann. Probab. 21 1275–1294. 0779.60026 10.1214/aop/1176989118 euclid.aop/1176989118
Bing, X. and Wegkamp, M. H (2019). Supplement to “Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models.” DOI:10.1214/18-AOS1774SUPP. 07151057 10.1214/18-AOS1774 euclid.aos/1572487389Bing, X. and Wegkamp, M. H (2019). Supplement to “Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models.” DOI:10.1214/18-AOS1774SUPP. 07151057 10.1214/18-AOS1774 euclid.aos/1572487389
Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Series in Statistics. Springer, Heidelberg.Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Series in Statistics. Springer, Heidelberg.
Bunea, F., She, Y. and Wegkamp, M. H. (2011). Optimal selection of reduced rank estimators of high-dimensional matrices. Ann. Statist. 39 1282–1309. MR2816355 1216.62086 10.1214/11-AOS876 euclid.aos/1304947051Bunea, F., She, Y. and Wegkamp, M. H. (2011). Optimal selection of reduced rank estimators of high-dimensional matrices. Ann. Statist. 39 1282–1309. MR2816355 1216.62086 10.1214/11-AOS876 euclid.aos/1304947051
Bunea, F., She, Y. and Wegkamp, M. H. (2012). Joint variable and rank selection for parsimonious estimation of high-dimensional matrices. Ann. Statist. 40 2359–2388. 1373.62246 10.1214/12-AOS1039 euclid.aos/1359987524Bunea, F., She, Y. and Wegkamp, M. H. (2012). Joint variable and rank selection for parsimonious estimation of high-dimensional matrices. Ann. Statist. 40 2359–2388. 1373.62246 10.1214/12-AOS1039 euclid.aos/1359987524
Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika 1 211–218. 62.1075.02 10.1007/BF02288367Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika 1 211–218. 62.1075.02 10.1007/BF02288367
Giraud, C. (2011). Low rank multivariate regression. Electron. J. Stat. 5 775–799. 1274.62434 10.1214/11-EJS625Giraud, C. (2011). Low rank multivariate regression. Electron. J. Stat. 5 775–799. 1274.62434 10.1214/11-EJS625
Giraud, C. (2015). Introduction to High-Dimensional Statistics. Monographs on Statistics and Applied Probability 139. CRC Press, Boca Raton, FL. MR3307991Giraud, C. (2015). Introduction to High-Dimensional Statistics. Monographs on Statistics and Applied Probability 139. CRC Press, Boca Raton, FL. MR3307991
Izenman, A. J. (1975). Reduced-rank regression for the multivariate linear model. J. Multivariate Anal. 5 248–264. 0313.62042 10.1016/0047-259X(75)90042-1Izenman, A. J. (1975). Reduced-rank regression for the multivariate linear model. J. Multivariate Anal. 5 248–264. 0313.62042 10.1016/0047-259X(75)90042-1
Izenman, A. J. (2008). Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning. Springer Texts in Statistics. Springer, New York. MR2445017 1155.62040Izenman, A. J. (2008). Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning. Springer Texts in Statistics. Springer, New York. MR2445017 1155.62040
Johnstone, I. M. (2001). Chi-square oracle inequalities. In State of the Art in Probability and Statistics: Festschrift for Willem R. van Zwet (M. de Gunst, C. Klaasen and A. van der Vaart, eds.) 399–418. IMS, Beachwood, OH. 1373.62062Johnstone, I. M. (2001). Chi-square oracle inequalities. In State of the Art in Probability and Statistics: Festschrift for Willem R. van Zwet (M. de Gunst, C. Klaasen and A. van der Vaart, eds.) 399–418. IMS, Beachwood, OH. 1373.62062
Lounici, K., Pontil, M., van de Geer, S. and Tsybakov, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39 2164–2204. 1306.62156 10.1214/11-AOS896 euclid.aos/1319595462Lounici, K., Pontil, M., van de Geer, S. and Tsybakov, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39 2164–2204. 1306.62156 10.1214/11-AOS896 euclid.aos/1319595462
Negahban, S. and Wainwright, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069–1097. 1216.62090 10.1214/10-AOS850 euclid.aos/1304947044Negahban, S. and Wainwright, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069–1097. 1216.62090 10.1214/10-AOS850 euclid.aos/1304947044
Obozinski, G., Wainwright, M. J. and Jordan, M. I. (2011). Support union recovery in high-dimensional multivariate regression. Ann. Statist. 39 1–47. 1373.62372 10.1214/09-AOS776 euclid.aos/1291388368Obozinski, G., Wainwright, M. J. and Jordan, M. I. (2011). Support union recovery in high-dimensional multivariate regression. Ann. Statist. 39 1–47. 1373.62372 10.1214/09-AOS776 euclid.aos/1291388368
Rao, C. R. (1978). Matrix approximations and reduction of dimensionality in multivariate statistical analysis. In Multivariate Analysis V 3–22. North-Holland, Amsterdam.Rao, C. R. (1978). Matrix approximations and reduction of dimensionality in multivariate statistical analysis. In Multivariate Analysis V 3–22. North-Holland, Amsterdam.
Reinsel, G. C. and Velu, R. P. (1998). Multivariate Reduced-Rank Regression: Theory and Applications. Lecture Notes in Statistics 136. Springer, New York. 0909.62066Reinsel, G. C. and Velu, R. P. (1998). Multivariate Reduced-Rank Regression: Theory and Applications. Lecture Notes in Statistics 136. Springer, New York. 0909.62066
Robinson, P. M. (1973). Generalized canonical analysis for time series. J. Multivariate Anal. 3 141–160. 0263.62036 10.1016/0047-259X(73)90019-5Robinson, P. M. (1973). Generalized canonical analysis for time series. J. Multivariate Anal. 3 141–160. 0263.62036 10.1016/0047-259X(73)90019-5
Robinson, P. M. (1974). Identification, estimation and large-sample theory for regressions containing unobservable variables. Internat. Econom. Rev. 15 680–692. MR356376 0298.62013 10.2307/2525734Robinson, P. M. (1974). Identification, estimation and large-sample theory for regressions containing unobservable variables. Internat. Econom. Rev. 15 680–692. MR356376 0298.62013 10.2307/2525734
Rohde, A. and Tsybakov, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887–930. 1215.62056 10.1214/10-AOS860 euclid.aos/1299680958Rohde, A. and Tsybakov, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887–930. 1215.62056 10.1214/10-AOS860 euclid.aos/1299680958
Rudelson, M. and Vershynin, R. (2010). Non-asymptotic theory of random matrices: Extreme singular values. In Proceedings of the International Congress of Mathematicians. Volume III 1576–1602. Hindustan Book Agency, New Delhi. MR2827856 1227.60011Rudelson, M. and Vershynin, R. (2010). Non-asymptotic theory of random matrices: Extreme singular values. In Proceedings of the International Congress of Mathematicians. Volume III 1576–1602. Hindustan Book Agency, New Delhi. MR2827856 1227.60011
Schmidt, E. (1907). Zur Theorie der linearen und nichtlinearen Integralgleichungen. I. Teil: Entwicklung willkürlicher Funktionen nach Systemen vorgeschriebener. Math. Ann. 63 433–476.Schmidt, E. (1907). Zur Theorie der linearen und nichtlinearen Integralgleichungen. I. Teil: Entwicklung willkürlicher Funktionen nach Systemen vorgeschriebener. Math. Ann. 63 433–476.
Stewart, G. W. (1993). On the early history of the singular value decomposition. SIAM Rev. 35 551–566. 0799.01016 10.1137/1035134Stewart, G. W. (1993). On the early history of the singular value decomposition. SIAM Rev. 35 551–566. 0799.01016 10.1137/1035134
Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing 210–268. Cambridge Univ. Press, Cambridge.Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing 210–268. Cambridge Univ. Press, Cambridge.
Weyl, H. (1912). Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung). Math. Ann. 71 441–479. 43.0436.01 10.1007/BF01456804Weyl, H. (1912). Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung). Math. Ann. 71 441–479. 43.0436.01 10.1007/BF01456804
Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B. Stat. Methodol. 68 49–67. 1141.62030 10.1111/j.1467-9868.2005.00532.xYuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B. Stat. Methodol. 68 49–67. 1141.62030 10.1111/j.1467-9868.2005.00532.x
