The Annals of Statistics

Global testing against sparse alternatives in time-frequency analysis

T. Tony Cai, Yonina C. Eldar, and Xiaodong Li

Full-text: Open access


In this paper, an over-sampled periodogram higher criticism (OPHC) test is proposed for the global detection of sparse periodic effects in a complex-valued time series. An explicit minimax detection boundary is established between the rareness and weakness of the complex sinusoids hidden in the series. The OPHC test is shown to be asymptotically powerful in the detectable region. Numerical simulations illustrate and verify the effectiveness of the proposed test. Furthermore, the periodogram over-sampled by $O(\log N)$ is proven universally optimal in global testing for periodicities under a mild minimum separation condition.

Article information

Ann. Statist., Volume 44, Number 4 (2016), 1438-1466.

Received: June 2015
Revised: October 2015
First available in Project Euclid: 7 July 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62C20: Minimax procedures 62F03: Hypothesis testing 62F05: Asymptotic properties of tests 62G30: Order statistics; empirical distribution functions 62G32: Statistics of extreme values; tail inference

Testing for periodicity sparsity over-sampled periodogram higher criticism detection boundary empirical processes


Cai, T. Tony; Eldar, Yonina C.; Li, Xiaodong. Global testing against sparse alternatives in time-frequency analysis. Ann. Statist. 44 (2016), no. 4, 1438--1466. doi:10.1214/15-AOS1412.

Export citation


  • [1] Ahdesmaki, M., Lahdesmaki, H., Pearson, R., Huttunen, H. and Yli-Harja, O. (2005). Robust detection of periodic time series measured from biological systems. BMC Bioinformatics 6 117.
  • [2] Ahdesmaki, M., Lahdesmaki, H. and Yli-Harja, O. (2007). Robust Fisher’s test for periodicity detection in noisy biological time series. In GENSIPS, IEEE International Workshop on 1–4. Tuusula, Finland.
  • [3] Alexander, K. S. (1987). Rates of growth and sample moduli for weighted empirical processes indexed by sets. Probab. Theory Related Fields 75 379–423.
  • [4] Arias-Castro, E., Candès, E. J., Helgason, H. and Zeitouni, O. (2008). Searching for a trail of evidence in a maze. Ann. Statist. 36 1726–1757.
  • [5] Arias-Castro, E., Candès, E. J. and Plan, Y. (2011). Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism. Ann. Statist. 39 2533–2556.
  • [6] Arias-Castro, E., Donoho, D. L. and Huo, X. (2005). Near-optimal detection of geometric objects by fast multiscale methods. IEEE Trans. Inform. Theory 51 2402–2425.
  • [7] Bhaskar, B. N., Tang, G. and Recht, B. (2013). Atomic norm denoising with applications to line spectral estimation. IEEE Trans. Signal Process. 61 5987–5999.
  • [8] Bølviken, E. (1983). New tests of significance in periodogram analysis. Scand. J. Stat. 10 1–9.
  • [9] Brockwell, P. J. and Davis, R. A. (2009). Times Series: Theory and Methods, 2nd ed. Springer, Berlin.
  • [10] Cadzow, J. (1988). Signal enhancement—a composite property mapping algorithm. IEEE Trans. Acoust. Speech Signal Process. 36 49–62.
  • [11] Cai, T., Eldar, Y. C. and Li, X. (2016). Supplement to “Global testing against sparse alternatives in time-frequency analysis.” DOI:10.1214/15-AOS1412SUPP.
  • [12] Cai, T. T., Jeng, X. J. and Jin, J. (2011). Optimal detection of heterogeneous and heteroscedastic mixtures. J. R. Stat. Soc. Ser. B Stat. Methodol. 73 629–662.
  • [13] Cai, T. T., Jin, J. and Low, M. G. (2007). Estimation and confidence sets for sparse normal mixtures. Ann. Statist. 35 2421–2449.
  • [14] Cai, T. T. and Wu, Y. (2014). Optimal detection of sparse mixtures against a given null distribution. IEEE Trans. Inform. Theory 60 2217–2232.
  • [15] Candès, E. J. and Fernandez-Granda, C. (2013). Super-resolution from noisy data. J. Fourier Anal. Appl. 19 1229–1254.
  • [16] Candès, E. J. and Fernandez-Granda, C. (2014). Towards a mathematical theory of super-resolution. Comm. Pure Appl. Math. 67 906–956.
  • [17] Chen, J. (2005). Identification of significant genes in microarray gene expression data. BMC Bioinformatics 6 286.
  • [18] Chen, S. S. and Donoho, D. L. (1998). Application of basis pursuit in spectrum estimation. In Conference on Acoustics, Speech and Signal Processing, Vol. 3. Seattle, WA.
  • [19] Chiu, S.-T. (1989). Detecting periodic components in a white Gaussian time series. J. Roy. Statist. Soc. Ser. B 51 249–259.
  • [20] Davies, R. B. (1987). Hypothesis testing when a nuisance parameter is present only under the alternative. Biometrika 74 33–43.
  • [21] Dicker, L. H. (2014). Variance estimation in high-dimensional linear models. Biometrika 101 269–284.
  • [22] Djurić, P. M. (1996). A model selection rule for sinusoids in white Gaussian noise. IEEE Trans. Signal Process. 44 1744–1751.
  • [23] Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • [24] Donoho, D. L. (1992). Superresolution via sparsity constraints. SIAM J. Math. Anal. 23 1309–1331.
  • [25] Duarte, M. F. and Baraniuk, R. G. (2013). Spectral compressive sensing. Appl. Comput. Harmon. Anal. 35 111–129.
  • [26] Fannjiang, A. and Liao, W. (2012). Coherence pattern-guided compressive sensing with unresolved grids. SIAM J. Imaging Sci. 5 179–202.
  • [27] Fisher, R. A. (1929). Tests of significance in harmonic analysis. Proceedings of Royal Society, Ser. A 125 54–59.
  • [28] Fuller, W. A. (1976). Introduction to Statistical Time Series. Wiley, New York.
  • [29] Giné, E. and Koltchinskii, V. (2006). Concentration inequalities and asymptotic results for ratio type empirical processes. Ann. Probab. 34 1143–1216.
  • [30] Glynn, E. F., Chen, J. and Mushegian, A. R. (2006). Detecting periodic patterns in unevenly spaced gene expression time series using Lomb–Scargle periodograms. Bioinformatics 22 310–316.
  • [31] Hall, P. and Jin, J. (2010). Innovated higher criticism for detecting sparse signals in correlated noise. Ann. Statist. 38 1686–1732.
  • [32] Hu, L., Shi, Z., Zhou, J. and Fu, Q. (2012). Compressed sensing of complex sinusoids: An approach based on dictionary refinement. IEEE Trans. Signal Process. 60 3809–3822.
  • [33] Hua, Y. and Sarkar, T. K. (1990). Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise. IEEE Trans. Acoust. Speech Signal Process. 38 814–824.
  • [34] Ingster, Yu. I. (1998). Minimax detection of a signal for $l^{n}$-balls. Math. Methods Statist. 7 401–428 (1999).
  • [35] Ingster, Y. I., Tsybakov, A. B. and Verzelen, N. (2010). Detection boundary in sparse regression. Electron. J. Stat. 4 1476–1526.
  • [36] Juditsky, A. and Nemirovski, A. (2015). On detecting harmonic oscillations. Bernoulli 21 1134–1165.
  • [37] Kay, S. M. (1998). Fundamentals of Statistical Signal Processing: Detection Theory. Prentice Hall, Upper Saddle River, NJ.
  • [38] Knight, W. C., Pridham, R. G. and Kay, S. M. (1981). Digital signal processing for sonar. Proc. IEEE 69 1451–1506.
  • [39] Leadbetter, M. R., Lindgren, G. and Rootzén, H. (1983). Extremes and Related Properties of Random Sequences and Processes. Springer, New York.
  • [40] Li, J. and Siegmund, D. (2015). Higher criticism: $p$-values and criticism. Ann. Statist. 43 1323–1350.
  • [41] Li, W. V. and Shao, Q.-M. (2002). A normal comparison inequality and its applications. Probab. Theory Related Fields 122 494–508.
  • [42] Nadler, B. and Kontorovich, A. (2011). Model selection for sinusoids in noise: Statistical analysis and a new penalty term. IEEE Trans. Signal Process. 59 1333–1345.
  • [43] Prony, R. (1795). Essai expérimental et analytique: Sur les lois de la dilatabilité de fluides élastique et sur celles de la force expansive de la vapeur de l’alkool, à différentes températures. J. Éc. Polytech. 1 24–76.
  • [44] Quinn, B. G. and Hannan, E. J. (2001). The Estimation and Tracking of Frequency. Cambridge Series in Statistical and Probabilistic Mathematics 9. Cambridge Univ. Press, Cambridge.
  • [45] Quinn, B. G. and Kootsookos, P. J. (1994). Threshold behavior of the maximum likelihood estimator of frequency. IEEE Trans. Signal Process. 42 3291–3294.
  • [46] Roy, R. and Kailath, T. (1990). ESPRIT—estimation of signal parameters via rotational invariance techniques. In Signal Processing, Part II. IMA Vol. Math. Appl. 23 369–411. Springer, New York.
  • [47] Rubin-Delanchy, P. and Walden, A. T. (2008). Kinematics of complex-valued time series. IEEE Trans. Signal Process. 56 4189–4198.
  • [48] Schmidt, R. (1986). Multiple emitter location and signal parameter estimation. IEEE Trans. Antennas and Propagation 34 276–280.
  • [49] Siegel, A. F. (1980). Testing for periodicity in a time series. J. Amer. Statist. Assoc. 75 345–348.
  • [50] Skolnik, M. I. (1980). Introduction to Radar Systems. McGraw-Hill, New York.
  • [51] Sykulski, A. M., Olhede, S. C., Lilly, J. M. and Early, J. J. (2013). The Whittle likelihood for complex-valued time series. Preprint.
  • [52] Tang, G., Bhaskar, B. N. and Recht, B. (2015). Near minimax line spectral estimation. IEEE Trans. Inform. Theory 61 499–512.
  • [53] van Zuijlen, M. C. A. (1978). Properties of the empirical distribution function for independent nonidentically distributed random variables. Ann. Probab. 6 250–266.
  • [54] Van Trees, H. L. (1992). Detection, Estimation, and Modulation Theory, Part 3: Radar-Sonar Signal Processing and Gaussian Signals in Noise. Wiley, New York.
  • [55] Whalen, A. D. (1995). Detection of Signals in Noise. Academic Press, New York.
  • [56] Wichert, S., Fokianos, K. and Strimmer, K. (2004). Identifying periodically expressed transcripts in microarray time series data. Bioinformatics 20 5–20.

Supplemental materials