## The Annals of Statistics

- Ann. Statist.
- Volume 45, Number 1 (2017), 415-460.

### Extreme eigenvalues of large-dimensional spiked Fisher matrices with application

#### Abstract

Consider two $p$-variate populations, not necessarily Gaussian, with covariance matrices $\Sigma_{1}$ and $\Sigma_{2}$, respectively. Let $S_{1}$ and $S_{2}$ be the corresponding sample covariance matrices with degrees of freedom $m$ and $n$. When the difference $\Delta$ between $\Sigma_{1}$ and $\Sigma_{2}$ is of small rank compared to $p,m$ and $n$, the Fisher matrix $S:=S_{2}^{-1}S_{1}$ is called a *spiked Fisher matrix*. When $p,m$ and $n$ grow to infinity proportionally, we establish a phase transition for the extreme eigenvalues of the Fisher matrix: a displacement formula showing that when the eigenvalues of $\Delta$ (*spikes*) are above (or under) a critical value, the associated extreme eigenvalues of $S$ will converge to some point outside the support of the global limit (LSD) of other eigenvalues (become outliers); otherwise, they will converge to the edge points of the LSD. Furthermore, we derive central limit theorems for those outlier eigenvalues of $S$. The limiting distributions are found to be Gaussian if and only if the corresponding population spike eigenvalues in $\Delta$ are *simple*. Two applications are introduced. The first application uses the largest eigenvalue of the Fisher matrix to test the equality between two high-dimensional covariance matrices, and explicit power function is found under the spiked alternative. The second application is in the field of signal detection, where an estimator for the number of signals is proposed while the covariance structure of the noise is arbitrary.

#### Article information

**Source**

Ann. Statist., Volume 45, Number 1 (2017), 415-460.

**Dates**

Received: April 2015

Revised: March 2016

First available in Project Euclid: 21 February 2017

**Permanent link to this document**

https://projecteuclid.org/euclid.aos/1487667628

**Digital Object Identifier**

doi:10.1214/16-AOS1463

**Mathematical Reviews number (MathSciNet)**

MR3611497

**Zentralblatt MATH identifier**

1360.62283

**Subjects**

Primary: 62H12: Estimation

Secondary: 60F05: Central limit and other weak theorems

**Keywords**

Large-dimensional Fisher matrices spiked Fisher matrix spiked population model extreme eigenvalue phase transition central limit theorem signal detection high-dimensional data analysis

#### Citation

Wang, Qinwen; Yao, Jianfeng. Extreme eigenvalues of large-dimensional spiked Fisher matrices with application. Ann. Statist. 45 (2017), no. 1, 415--460. doi:10.1214/16-AOS1463. https://projecteuclid.org/euclid.aos/1487667628