Open Access
June 2010 Innovated higher criticism for detecting sparse signals in correlated noise
Peter Hall, Jiashun Jin
Ann. Statist. 38(3): 1686-1732 (June 2010). DOI: 10.1214/09-AOS764


Higher criticism is a method for detecting signals that are both sparse and weak. Although first proposed in cases where the noise variables are independent, higher criticism also has reasonable performance in settings where those variables are correlated. In this paper we show that, by exploiting the nature of the correlation, performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer. Indeed, it turns out that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection (for a given level of signal sparsity and strength) can be obtained when correlation is present. We characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary. The boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is Toeplitz.


Download Citation

Peter Hall. Jiashun Jin. "Innovated higher criticism for detecting sparse signals in correlated noise." Ann. Statist. 38 (3) 1686 - 1732, June 2010.


Published: June 2010
First available in Project Euclid: 24 March 2010

zbMATH: 1189.62080
MathSciNet: MR2662357
Digital Object Identifier: 10.1214/09-AOS764

Primary: 62G10 , 62M10
Secondary: 62G32 , 62H15

Keywords: Adding noise , Cholesky factorization , empirical process , innovation , multiple hypothesis testing , sparse normal means , Spectral density , Toeplitz matrix

Rights: Copyright © 2010 Institute of Mathematical Statistics

Vol.38 • No. 3 • June 2010
Back to Top