The Annals of Statistics
- Ann. Statist.
- Volume 38, Number 3 (2010), 1686-1732.
Innovated higher criticism for detecting sparse signals in correlated noise
Higher criticism is a method for detecting signals that are both sparse and weak. Although first proposed in cases where the noise variables are independent, higher criticism also has reasonable performance in settings where those variables are correlated. In this paper we show that, by exploiting the nature of the correlation, performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer. Indeed, it turns out that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection (for a given level of signal sparsity and strength) can be obtained when correlation is present. We characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary. The boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is Toeplitz.
Ann. Statist., Volume 38, Number 3 (2010), 1686-1732.
First available in Project Euclid: 24 March 2010
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Primary: 62G10: Hypothesis testing 62M10: Time series, auto-correlation, regression, etc. [See also 91B84]
Secondary: 62G32: Statistics of extreme values; tail inference 62H15: Hypothesis testing
Hall, Peter; Jin, Jiashun. Innovated higher criticism for detecting sparse signals in correlated noise. Ann. Statist. 38 (2010), no. 3, 1686--1732. doi:10.1214/09-AOS764. https://projecteuclid.org/euclid.aos/1269452652