• Bernoulli
  • Volume 22, Number 3 (2016), 1331-1363.

Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests

Veronika Gontscharuk, Sandra Landwehr, and Helmut Finner

Full-text: Open access


Instead of defining goodness of fit (GOF) tests in terms of their test statistics, we present an alternative method by introducing the concept of local levels, which indicate high or low local sensitivity of a test. Local levels can act as a starting point for the construction of new GOF tests. We study the behavior of local levels when applied to some well-known GOF tests such as Kolmogorov–Smirnov (KS) tests, higher criticism (HC) tests and tests based on phi-divergences. The main focus is on a rigorous characterization of the asymptotic behavior of local levels of the original HC tests which leads to several further asymptotic results for local levels of other GOF tests including GOF tests with equal local levels. While local levels of KS tests, which are related to the central range, are asymptotically strictly larger than zero, all local levels of HC tests converge to zero as the sample size increases. Consequently, there exists no asymptotic level $\alpha$ GOF test such that all local levels are asymptotically bounded away from zero. Finally, by means of numerical computations we compare classical KS and HC tests to a GOF test with equal local levels.

Article information

Bernoulli, Volume 22, Number 3 (2016), 1331-1363.

Received: February 2014
Revised: October 2014
First available in Project Euclid: 16 March 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

higher criticism statistic Kolmogorov–Smirnov test local levels minimum $p$-value test Normal and Poisson approximation order statistics


Gontscharuk, Veronika; Landwehr, Sandra; Finner, Helmut. Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests. Bernoulli 22 (2016), no. 3, 1331--1363. doi:10.3150/14-BEJ694.

Export citation


  • [1] Aldor-Noiman, S., Brown, L., Buja, A., Rolke, W. and Stine, R. (2013). The power to see: A new graphical test of normality. Am. Stat. 67 249–260.
  • [2] Bateman, H. and Erdelyi, A. (1953). Higher Transcendental Functions, Vol. 2. New York: McGraw-Hill.
  • [3] Berk, R.H. and Jones, D.H. (1978). Relatively optimal combinations of test statistics. Scand. J. Stat. 5 158–162.
  • [4] Berk, R.H. and Jones, D.H. (1979). Goodness-of-fit test statistics that dominate the Kolmogorov statistics. Z. Wahrscheinlichkeitstheor. Verw. Geb. 47 47–59.
  • [5] Buja, A. and Rolke, W. (2006). Calibration for simultaneity: (Re)Sampling methods for simultaneous inference with applications to function estimation and functional data. Unpublished.
  • [6] Cai, T.T., Jin, J. and Low, M.G. (2007). Estimation and confidence sets for sparse normal mixtures. Ann. Statist. 35 2421–2449.
  • [7] David, H.A. and Nagaraja, H.N. (2003). Order Statistics, 3rd ed. Wiley Series in Probability and Statistics. Hoboken, NJ: Wiley.
  • [8] Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • [9] Donoho, D. and Jin, J. (2008). Higher criticism thresholding: Optimal feature selection when useful features are rare and weak. Proc. Natl. Acad. Sci. USA 105 14790–14795.
  • [10] Donoho, D. and Jin, J. (2009). Feature selection by higher criticism thresholding achieves the optimal phase diagram. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4449–4470.
  • [11] Eicker, F. (1979). The asymptotic distribution of the suprema of the standardized empirical processes. Ann. Statist. 7 116–138.
  • [12] Ferreira, C., López, J.L. and Pérez Sinusía, E. (2005). Incomplete gamma functions for large values of their variables. Adv. in Appl. Math. 34 467–485.
  • [13] Finner, H., Dickhaus, T. and Roters, M. (2009). On the false discovery rate and an asymptotically optimal rejection curve. Ann. Statist. 37 596–618.
  • [14] Gontscharuk, V., Landwehr, S. and Finner, H. (2014). The intermediates take it all: Asymptotics of higher criticism statistics and a powerful alternative based on equal local levels. Biom. J. 57 159–180.
  • [15] Hall, P. and Jin, J. (2008). Properties of higher criticism under strong dependence. Ann. Statist. 36 381–402.
  • [16] Hall, P. and Jin, J. (2010). Innovated higher criticism for detecting sparse signals in correlated noise. Ann. Statist. 38 1686–1732.
  • [17] Ivchenko, G.I. (1974). On comparison of binomial and Poisson distributions. Theory Probab. Appl. 19 584–587.
  • [18] Jaeschke, D. (1979). The asymptotic distribution of the supremum of the standardized empirical distribution function on subintervals. Ann. Statist. 7 108–115.
  • [19] Jager, L. and Wellner, J.A. (2007). Goodness-of-fit tests via phi-divergences. Ann. Statist. 35 2018–2053.
  • [20] Kaplan, D.M. and Goldman, M. (2014). Evenly sensitive KS-type inference on distributions. Preprint. Available at
  • [21] Kaplan, D.M. and Goldman, M. (2014). True equality (of pointwise sensitivity) at last: A Dirichlet alternative to Kolmogorov–Smirnov inference on distributions. Preprint. Avaliable at
  • [22] Khmaladze, E. and Shinjikashvili, E. (2001). Calculation of noncrossing probabilities for Poisson processes and its corollaries. Adv. in Appl. Probab. 33 702–716.
  • [23] Leadbetter, M.R., Lindgren, G. and Rootzén, H. (1983). Extremes and Related Properties of Random Sequences and Processes. Springer Series in Statistics. New York: Springer.
  • [24] Lockhart, R.A. (1991). Overweight tails are inefficient. Ann. Statist. 19 2254–2258.
  • [25] Mary, D. and Ferrari, A. (2014). A non-asymptotic standardization of binomial counts in higher criticism. In 2014 IEEE International Symposium on Information Theory (ISIT) 561–565. Piscataway, NJ: IEEE.
  • [26] Mason, D.M. and Schuenemeyer, J.H. (1983). A modified Kolmogorov–Smirnov test sensitive to tail alternatives. Ann. Statist. 11 933–946.
  • [27] Moskovich Eiger, A., Nadler, B. and Spiegelman, C. (2014). The calibrated Kolmogorov–Smirnov test. Preprint. Available at arXiv:1311.3190.
  • [28] Nikitin, Y. (1995). Asymptotic Efficiency of Nonparametric Tests. Cambridge: Cambridge Univ. Press.
  • [29] Prohorov, Yu.V. (1953). Asymptotic behavior of the binomial distribution [Russian]. Uspekhi Mat. Nauk 8 135–142.
  • [30] Révész, P. (1982/83). A joint study of the Kolmogorov–Smirnov and the Eicker–Jaeschke statistics. Statist. Decisions 1 57–65.
  • [31] Shorack, G. and Wellner, J. (2009). Empirical Processes with Applications to Statistics. Philadelphia: Society for Industrial and Applied Mathematics.
  • [32] Smirnov, N.V. (1933). On probabilities of greater deviations [Russian]. Mathem. Sb. 40 443–454.