The Annals of Statistics

Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models

Xin Bing and Marten H. Wegkamp

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

We consider the multivariate response regression problem with a regression coefficient matrix of low, unknown rank. In this setting, we analyze a new criterion for selecting the optimal reduced rank. This criterion differs notably from the one proposed in Bunea, She and Wegkamp (Ann. Statist. 39 (2011) 1282–1309) in that it does not require estimation of the unknown variance of the noise, nor does it depend on a delicate choice of a tuning parameter. We develop an iterative, fully data-driven procedure, that adapts to the optimal signal-to-noise ratio. This procedure finds the true rank in a few steps with overwhelming probability. At each step, our estimate increases, while at the same time it does not exceed the true rank. Our finite sample results hold for any sample size and any dimension, even when the number of responses and of covariates grow much faster than the number of observations. We perform an extensive simulation study that confirms our theoretical findings. The new method performs better and is more stable than the procedure of Bunea, She and Wegkamp (Ann. Statist. 39 (2011) 1282–1309) in both low- and high-dimensional settings.

Article information

Source
Ann. Statist., Volume 47, Number 6 (2019), 3157-3184.

Dates
Received: February 2018
Revised: August 2018
First available in Project Euclid: 31 October 2019

Permanent link to this document
https://projecteuclid.org/euclid.aos/1572487389

Digital Object Identifier
doi:10.1214/18-AOS1774

Mathematical Reviews number (MathSciNet)
MR4025738

Subjects
Primary: 62H15: Hypothesis testing 62J07: Ridge regression; shrinkage estimators

Keywords
Multivariate response regression reduced rank estimator self-tuning adaptive rank estimation rank consistency dimension reduction oracle inequalities

Citation

Bing, Xin; Wegkamp, Marten H. Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models. Ann. Statist. 47 (2019), no. 6, 3157--3184. doi:10.1214/18-AOS1774. https://projecteuclid.org/euclid.aos/1572487389


Export citation

References

  • Anderson, T. W. (1951). Estimating linear restrictions on regression coefficients for multivariate normal distributions. Ann. Math. Stat. 22 327–351.
  • Anderson, T. W. (1999). Asymptotic distribution of the reduced rank regression estimator under general conditions. Ann. Statist. 27 1141–1154.
  • Anderson, T. W. (2002). Specification and misspecification in reduced rank regression. Sankhyā Ser. A 64 193–205.
  • Bai, Z. D. and Yin, Y. Q. (1993). Limit of the smallest eigenvalue of a large-dimensional sample covariance matrix. Ann. Probab. 21 1275–1294.
  • Bing, X. and Wegkamp, M. H (2019). Supplement to “Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models.” DOI:10.1214/18-AOS1774SUPP.
  • Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Series in Statistics. Springer, Heidelberg.
  • Bunea, F., She, Y. and Wegkamp, M. H. (2011). Optimal selection of reduced rank estimators of high-dimensional matrices. Ann. Statist. 39 1282–1309.
  • Bunea, F., She, Y. and Wegkamp, M. H. (2012). Joint variable and rank selection for parsimonious estimation of high-dimensional matrices. Ann. Statist. 40 2359–2388.
  • Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika 1 211–218.
  • Giraud, C. (2011). Low rank multivariate regression. Electron. J. Stat. 5 775–799.
  • Giraud, C. (2015). Introduction to High-Dimensional Statistics. Monographs on Statistics and Applied Probability 139. CRC Press, Boca Raton, FL.
  • Izenman, A. J. (1975). Reduced-rank regression for the multivariate linear model. J. Multivariate Anal. 5 248–264.
  • Izenman, A. J. (2008). Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning. Springer Texts in Statistics. Springer, New York.
  • Johnstone, I. M. (2001). Chi-square oracle inequalities. In State of the Art in Probability and Statistics: Festschrift for Willem R. van Zwet (M. de Gunst, C. Klaasen and A. van der Vaart, eds.) 399–418. IMS, Beachwood, OH.
  • Lounici, K., Pontil, M., van de Geer, S. and Tsybakov, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39 2164–2204.
  • Negahban, S. and Wainwright, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069–1097.
  • Obozinski, G., Wainwright, M. J. and Jordan, M. I. (2011). Support union recovery in high-dimensional multivariate regression. Ann. Statist. 39 1–47.
  • Rao, C. R. (1978). Matrix approximations and reduction of dimensionality in multivariate statistical analysis. In Multivariate Analysis V 3–22. North-Holland, Amsterdam.
  • Reinsel, G. C. and Velu, R. P. (1998). Multivariate Reduced-Rank Regression: Theory and Applications. Lecture Notes in Statistics 136. Springer, New York.
  • Robinson, P. M. (1973). Generalized canonical analysis for time series. J. Multivariate Anal. 3 141–160.
  • Robinson, P. M. (1974). Identification, estimation and large-sample theory for regressions containing unobservable variables. Internat. Econom. Rev. 15 680–692.
  • Rohde, A. and Tsybakov, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887–930.
  • Rudelson, M. and Vershynin, R. (2010). Non-asymptotic theory of random matrices: Extreme singular values. In Proceedings of the International Congress of Mathematicians. Volume III 1576–1602. Hindustan Book Agency, New Delhi.
  • Schmidt, E. (1907). Zur Theorie der linearen und nichtlinearen Integralgleichungen. I. Teil: Entwicklung willkürlicher Funktionen nach Systemen vorgeschriebener. Math. Ann. 63 433–476.
  • Stewart, G. W. (1993). On the early history of the singular value decomposition. SIAM Rev. 35 551–566.
  • Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing 210–268. Cambridge Univ. Press, Cambridge.
  • Weyl, H. (1912). Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung). Math. Ann. 71 441–479.
  • Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B. Stat. Methodol. 68 49–67.

Supplemental materials

  • Supplement to “Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models”. The supplementary document includes the oracle inequality for the fit, additional simulation results and all proofs.