The Annals of Statistics

Higher criticism: $p$-values and criticism

Jian Li and David Siegmund

Full-text: Open access


This paper compares the higher criticism statistic (Donoho and Jin [ Ann. Statist. 32 (2004) 962–994]), a modification of the higher criticism statistic also suggested by Donoho and Jin, and two statistics of the Berk–Jones [ Z. Wahrsch. Verw. Gebiete 47 (1979) 47–59] type. New approximations to the significance levels of the statistics are derived, and their accuracy is studied by simulations. By numerical examples it is shown that over a broad range of sample sizes the Berk–Jones statistics have a better power function than the higher criticism statistics to detect sparse mixtures. The applications suggested by Meinshausen and Rice [ Ann. Statist. 34 (2006) 373–393], to find lower confidence bounds for the number of false hypotheses, and by Jeng, Cai and Li [ Biometrika 100 (2013) 157–172], to detect copy number variants, are also studied.

Article information

Ann. Statist., Volume 43, Number 3 (2015), 1323-1350.

Received: January 2014
Revised: October 2014
First available in Project Euclid: 15 May 2015

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62F10: Point estimation
Secondary: 62G20: Asymptotic properties

Global $p$-value sparse mixture


Li, Jian; Siegmund, David. Higher criticism: $p$-values and criticism. Ann. Statist. 43 (2015), no. 3, 1323--1350. doi:10.1214/15-AOS1312.

Export citation


  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 289–300.
  • Berk, R. H. and Jones, D. H. (1979). Goodness-of-fit test statistics that dominate the Kolmogorov statistics. Z. Wahrsch. Verw. Gebiete 47 47–59.
  • Darling, D. A. and Erdös, P. (1956). A limit theorem for the maximum of normalized sums of independent random variables. Duke Math. J. 23 143–155.
  • Dembo, A. and Zeitouni, O. (2010). Large Deviations Techniques and Applications. Stochastic Modelling and Applied Probability 38. Springer, Berlin.
  • Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • Eiger, A. M., Nadler, B. and Spiegelman, C. (2013). The calibrated Kolmogorov–Smirnov test. Available at arXiv:1311.3190v1.
  • Jaeschke, D. (1979). The asymptotic distribution of the supremum of the standardized empirical distribution function on subintervals. Ann. Statist. 7 108–115.
  • Jager, L. and Wellner, J. A. (2007). Goodness-of-fit tests via phi-divergences. Ann. Statist. 35 2018–2053.
  • Jeng, X. J., Cai, T. T. and Li, H. (2013). Simultaneous discovery of rare and common segment variants. Biometrika 100 157–172.
  • Loader, C. R. (1992). Boundary crossing probabilities for locally Poisson processes. Ann. Appl. Probab. 2 199–228.
  • Meinshausen, N. and Rice, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 373–393.
  • Noé, M. (1972). The calculation of distributions of two-sided Kolmogorov–Smirnov type statistics. Ann. Math. Statist. 43 58–64.
  • Owen, A. B. (1995). Nonparametric likelihood confidence bands for a distribution function. J. Amer. Statist. Assoc. 90 516–521.
  • Siegmund, D. (1985). Sequential Analysis: Tests and Confidence Intervals. Springer, New York.
  • Siegmund, D., Yakir, B. and Zhang, N. R. (2011). Detecting simultaneous variant intervals in aligned sequences. Ann. Appl. Stat. 5 645–668.
  • Walther, G. (2013). The average likelihood ratio for large-scale multiple testing and detecting sparse mixtures. In From Probability to Statistics and Back: High-Dimensional Models and Processes. Inst. Math. Stat. (IMS) Collect. 9 317–326. IMS, Beachwood, OH.
  • Wellner, J. A. and Koltchinskii, V. (2003). A note on the asymptotic distribution of Berk–Jones type statistics under the null hypothesis. In High Dimensional Probability, III (Sandjberg, 2002). Progress in Probability 55 321–332. Birkhäuser, Basel.
  • Woodroofe, M. (1976). Frequentist properties of Bayesian sequential tests. Biometrika 63 101–110.
  • Xie, Y. and Siegmund, D. (2013). Sequential multi-sensor change-point detection. Ann. Statist. 41 670–692.
  • Yakir, B. (2013). Extremes in Random Fields: A Theory and Its Applications. Wiley, Chichester.
  • Zhang, N. R., Siegmund, D. O., Ji, H. and Li, J. Z. (2010). Detecting simultaneous changepoints in multiple sequences. Biometrika 97 631–645.