Ratio-consistent estimation for long range dependent Toeplitz covariance with application to matrix data whitening

Abstract

We consider a data matrix $\mathbf{X}:={\mathbf{C}}_N^{1∕2}\mathbf{Z}{\mathbf{R}}_M^{1∕2}$ from a multivariate stationary process with a separable covariance function, where ${\mathbf{C}}_N$ is a $N\times N$ positive semi-definite matrix, Z a $N\times M$ random matrix of uncorrelated standardized white noise, and ${\mathbf{R}}_M$ a $M\times M$ Toeplitz matrix. Under the assumption of long range dependence (LRD), we re-examine the consistency of two toeplitzifized estimators ${\hat{\mathbf{R}}}_M$ (unbiased) and ${\hat{\mathbf{R}}}_M^b$ (biased) for ${\mathbf{R}}_M$, which are known to be norm consistent with ${\mathbf{R}}_M$ when the process is short range dependent (SRD). However in the LRD case, some simulations suggest that the norm consistency does not hold in general for both estimators. Instead, a weaker ratio consistency is established for the unbiased estimator ${\hat{\mathbf{R}}}_M$, and a further weaker ratio LSD consistency is established for the biased estimator ${\hat{\mathbf{R}}}_M^b$. The main result leads to a consistent whitening procedure on the original data matrix X, which is further applied to two real world questions, one is a signal detection problem, and the other is PCA on the space covariance ${\mathbf{C}}_N$ to achieve a noise reduction and data compression.