## The Annals of Statistics

### Extreme eigenvalues of large-dimensional spiked Fisher matrices with application

#### Abstract

Consider two $p$-variate populations, not necessarily Gaussian, with covariance matrices $\Sigma_{1}$ and $\Sigma_{2}$, respectively. Let $S_{1}$ and $S_{2}$ be the corresponding sample covariance matrices with degrees of freedom $m$ and $n$. When the difference $\Delta$ between $\Sigma_{1}$ and $\Sigma_{2}$ is of small rank compared to $p,m$ and $n$, the Fisher matrix $S:=S_{2}^{-1}S_{1}$ is called a spiked Fisher matrix. When $p,m$ and $n$ grow to infinity proportionally, we establish a phase transition for the extreme eigenvalues of the Fisher matrix: a displacement formula showing that when the eigenvalues of $\Delta$ (spikes) are above (or under) a critical value, the associated extreme eigenvalues of $S$ will converge to some point outside the support of the global limit (LSD) of other eigenvalues (become outliers); otherwise, they will converge to the edge points of the LSD. Furthermore, we derive central limit theorems for those outlier eigenvalues of $S$. The limiting distributions are found to be Gaussian if and only if the corresponding population spike eigenvalues in $\Delta$ are simple. Two applications are introduced. The first application uses the largest eigenvalue of the Fisher matrix to test the equality between two high-dimensional covariance matrices, and explicit power function is found under the spiked alternative. The second application is in the field of signal detection, where an estimator for the number of signals is proposed while the covariance structure of the noise is arbitrary.

#### Article information

Source
Ann. Statist., Volume 45, Number 1 (2017), 415-460.

Dates
Received: April 2015
Revised: March 2016
First available in Project Euclid: 21 February 2017

Permanent link to this document
https://projecteuclid.org/euclid.aos/1487667628

Digital Object Identifier
doi:10.1214/16-AOS1463

Mathematical Reviews number (MathSciNet)
MR3611497

Zentralblatt MATH identifier
1360.62283

Subjects
Primary: 62H12: Estimation
Secondary: 60F05: Central limit and other weak theorems

#### Citation

Wang, Qinwen; Yao, Jianfeng. Extreme eigenvalues of large-dimensional spiked Fisher matrices with application. Ann. Statist. 45 (2017), no. 1, 415--460. doi:10.1214/16-AOS1463. https://projecteuclid.org/euclid.aos/1487667628