The Annals of Statistics

On the distribution of the largest eigenvalue in principal components analysis

Iain M. Johnstone

Full-text: Open access

Abstract

Let x(1) denote the square of the largest singular value of an n × p matrix X, all of whose entries are independent standard Gaussian variates. Equivalently, x(1) is the largest principal component variance of the covariance matrix $X'X$, or the largest eigenvalue of a p­variate Wishart distribution on n degrees of freedom with identity covariance.

Consider the limit of large p and n with $n/p = \gamma \ge 1$. When centered by $\mu_p = (\sqrt{n-1} + \sqrt{p})^2$ and scaled by $\sigma_p = (\sqrt{n-1} + \sqrt{p})(1/\sqrt{n-1} + 1/\sqrt{p}^{1/3}$, the distribution of x(1) approaches the Tracey-Widom law of order 1, which is defined in terms of the Painlevé II differential equation and can be numerically evaluated and tabulated in software. Simulations show the approximation to be informative for n and p as small as 5.

The limit is derived via a corresponding result for complex Wishart matrices using methods from random matrix theory. The result suggests that some aspects of large p multivariate distribution theory may be easier to apply in practice than their fixed p counterparts.

Article information

Source
Ann. Statist. Volume 29, Number 2 (2001), 295-327.

Dates
First available in Project Euclid: 24 December 2001

Permanent link to this document
http://projecteuclid.org/euclid.aos/1009210544

Digital Object Identifier
doi:10.1214/aos/1009210544

Mathematical Reviews number (MathSciNet)
MR1863961

Zentralblatt MATH identifier
1016.62078

Subjects
Primary: 62H25: Factor analysis and principal components; correspondence analysis 62F20
Secondary: 33C45: Orthogonal polynomials and functions of hypergeometric type (Jacobi, Laguerre, Hermite, Askey scheme, etc.) [See also 42C05 for general orthogonal polynomials and functions] 60H25: Random operators and equations [See also 47B80]

Keywords
Karhunen–Loève transform Laguerre ensemble empirical orthogonal functions largest eigenvalue largest singular value Laguerre polynomial Wishart distribution Plancherel–Rotach asymptotics Painlevé equation Tracy–Widom distribution random matrix theory Fredholm determinant Liouville–Green method

Citation

Johnstone, Iain M. On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29 (2001), no. 2, 295--327. doi:10.1214/aos/1009210544. http://projecteuclid.org/euclid.aos/1009210544.


Export citation

References

  • Aldous, D. and Diaconis, P. (1999). Longest increasing subsequences: from patience sorting to the Baik-Deift-Johansson theorem. Bull. Amer. Math. Soc. 36 413-432.
  • Anderson, T. W. (1963). Asymptotic theory for principal component analysis. Ann Math. Statist. 34 122-148
  • Anderson, T. W. (1996). R. A. Fisher and multivariate analysis. Statist. Sci. 11 20-34.
  • Bai, Z. D. (1999). Methodologies in spectral analysis of large dimensional random matrices: a review. Statist. Sinica 9 611-677.
  • Baik, J., Deift, P. and Johansson, K. (1999). On the distribution of the length of the longest increasing subsequence of random permutations. J. Amer. Math. Soc. 12 1119-1178.
  • Baker, T. H., Forrester, P. J. and Pearce, P. A. (1998). Random matrix ensembles with an effective extensive external charge. J. Phys. A 31 6087-6101.
  • Basor, E. L. (1997). Distribution functions for random variables for ensembles of positive Hermitian matrices, Comm. Math. Phys. 188 327-350.
  • Buja, A., Hastie, T. and Tibshirani, R. (1995). Penalized discriminant analysis. Ann. Statist. 23 73-102.
  • Constantine, A. G. (1963). Some non-central distribution problems in multivariate analysis. Ann. Math. Statist. 34 1270-1285. Deift, P. (1999a). Integrable systems and combinatorial theory. Notices Amer. Math. Soc. 47 631-640. Deift, P. (1999b). Orthogonal Polynomials and Random Matrices: A Riemann-Hilbert Approach. Amer. Math. Soc., Providence, RI.
  • Dunster, T. M. (1989). Uniform asymptotic expansions for Whittaker's confluent hypergeometric functions. SIAM J. Math. Anal. 20 744-760.
  • Dyson, F. J. (1970). Correlations between eigenvalues of a random matrix. Comm. Math. Phys. 19 235-250.
  • Eaton, M. L. (1983). Multivariate Statistics: A Vector Space Approach. Wiley, NewYork.
  • Edelman, A. (1988). Eigenvalues and condition numbers of random matrices. SIAM J. Matrix Anal. Appl. 9 543-560.
  • Edelman, A. (1991). The distribution and moments of the smallest eigenvalue of a random matrix of Wishart type. Linear Algebra Appl. 159 55-80.
  • Erd´elyi, A. (1960). Asymptotic forms for Laguerre polynomials. J. Indian Math. Soc. 24 235-250.
  • Forrester, P. J. (1993). The spectrum edge of random matrix ensembles. Nuclear Phys. B 402 709-728.
  • Forrester, P. J. (2000). Painlev´e transcendent evaluation of the scaled distribution of the smallest eigenvalue in the Laguerre orthogonal and symplectic ensembles. Technical report. www.lanl.gov arXiv:nlin.SI/0005064.
  • Geman, S. (1980). A limit theorem for the norm of random matrices. Ann. Probab. 8 252-261.
  • Gohberg, I. C. and Krein, M. G. (1969). Introduction to the Theory of Linear Non-selfadjoint Operators. Amer. Math. Soc., Providence, RI.
  • Hastings, S. P. and McLeod, J. B. (1980). A boundary value problem associated with the second Painlev´e transcendent and the Korteweg-de Vries equation. Arch. Rational Mech. Anal. 73 31-51.
  • Horn, R. A. and Johnson, C. R. (1985). Matrix Analysis. Cambridge Univ. Press.
  • James, A. T. (1964). Distributions of matrix variates and latent roots derived from normal samples. Ann. Math. Statist. 35 475-501.
  • Johansson, K. (1998). On fluctations of eigenvalues of random Hermitian matrices. Duke Math. J. 91 151-204.
  • Johansson, K. (2000). Shape fluctuations and random matrices. Comm. Math. Phys. 209 437-476.
  • Mardia, K. V., Kent, J. T. and Bibby, J. M. (1979). Multivariate Analysis. Academic Press, NewYork.
  • Mar cenko, V. A. and Pastur, L. A. (1967). Distributions of eigenvalues of some sets of random matrices. Math. USSR-Sb. 1 507-536.
  • Mehta, M. L. (1991). Random Matrices, 2nd ed. Academic Press, NewYork.
  • Muirhead, R. J. (1974). Powers of the largest latent root test of = I. Comm. Statist. 3 513-524.
  • Muirhead, R. J. (1982). Aspects of Multivariate Statistical Theory. Wiley, NewYork.
  • Olver, F. W. J. (1974). Asymptotics and Special Functions. Academic Press, NewYork.
  • Preisendorfer, R. W. (1988). Principal Component Analysis in Meteorology and Oceanogaphy. North-Holland, Amsterdam.
  • Riesz, F. and Sz.-Nagy, B. (1955). Functional Analysis. Ungar, NewYork.
  • Soshnikov, A. (1999). Universality at the edge of the spectrum in Wigner random matrices. Comm. Math. Phys. 207 697-733.
  • Soshnikov, A. (2001). A note on universality of the distribution of the largest eigenvalues in certain classes of sample covariance matrices, Technical report, www.lanl.gov arXiv:math:PR/0104113.
  • Stein, C. (1969). Multivariate analysis I. Technical report, Dept. Statistics Stanford Univ., pages 79-81. (Notes prepared by M. L. Eaton in 1966.)
  • Szeg ¨o, G. (1967). Orthogonal Polynomials, 3rd ed. Amer. Math. Soc. Providence.
  • Temme, N. M. (1990). Asymptotic estimates for Laguerre polynomials. J. Appl. Math. Phys. (ZAMP) 41 114-126.
  • Tracy, C. A. and Widom, H. (1994). Level-spacing distributions and the Airy kernel. Comm. Math. Phys. 159 151-174.
  • Tracy, C. A. and Widom, H. (1996). On orthogonal and symplectic matrix ensembles. Comm. Math. Phys. 177 727-754.
  • Tracy, C. A. and Widom, H. (1998). Correlation functions, cluster functions, and spacing distributions for random matrices. J. Statis. Phys. 92 809-835.
  • Tracy, C. A. and Widom, H. (1999). Airy kernel and Painlev´e II. Technical report. www.lanl.gov solv-int/9901004. To appear in CRM Proceedings and Lecture Notes: "Isomonodromic Deformations and Applications in Physics," J. Harnad, ed.
  • Tracy, C. A. and Widom, H. (2000). The distribution of the largest eigenvalue in the Gaussian ensembles. In Calogero-Moser-Sutherland Models (J. van Diejen and L. Vinet, eds.) 461-472. Springer, NewYork.
  • Wachter, K. W. (1976). Probability plotting points for principal components. In Ninth Interface Symposium Computer Science and Statistics (D. Hoaglin and R. Welsch, eds.) 299-308. Prindle, Weber and Schmidt, Boston.
  • Widom, H. (1999). On the relation between orthogonal, symplectic and unitary ensembles. J. Statist. Phys. 94 347-363.
  • Wigner, E. P. (1955). Characteristic vectors of bordered matrices of infinite dimensions. Ann. Math. 62 548-564.
  • Wigner, E. P. (1958). On the distribution of the roots of certain symmetric matrices. Ann. Math. 67 325-328.
  • Wilks, S. S. (1962). Mathematical Statistics. Wiley, NewYork.