The Annals of Statistics

Multiple hypotheses testing and expected number of type I. errors

H. Finner and M. Roters

Full-text: Open access

Abstract

The performance of multiple test procedures with respect to error control is an old issue. Assuming that all hypotheses are true we investigate the behavior of the expected number of type I errors (ENE) as a characteristic of certain multiple tests controlling the familywise error rate (FWER) or the false discovery rate (FDR) at a prespecified level. We derive explicit formulas for the distribution of the number of false rejections as well as for the ENE for single-step, step-down and step-up procedures based on independent $p$-values. Moreover, we determine the corresponding asymptotic distributions of the number of false rejections as well as explicit formulae for the ENE if the number of hypotheses tends to infinity. In case of FWER-control we mostly obtain Poisson distributions and in one case a geometric distribution as limiting distributions; in case of FDR control we obtain limiting distributions which are apparently not named in the literature. Surprisingly, the ENE is bounded by a small number regardless of the number of hypotheses under consideration. Finally, it turns out that in case of dependent test statistics the ENE behaves completely differently compared to the case of independent test statistics.

Article information

Source
Ann. Statist., Volume 30, Number 1 (2002), 220-238.

Dates
First available in Project Euclid: 5 March 2002

Permanent link to this document
https://projecteuclid.org/euclid.aos/1015362191

Digital Object Identifier
doi:10.1214/aos/1015362191

Mathematical Reviews number (MathSciNet)
MR1892662

Zentralblatt MATH identifier
1012.62020

Subjects
Primary: 62J15: Paired and multiple comparisons 62F05
Secondary: 62F03: Hypothesis testing 60F99

Keywords
Asymptotic critical value behavior Ballot theorem Bolshev’s recursion Bonferroni test procedure Dempster’s formula, DKW inequality empirical distribution function familywise error rate false discovery rate independent $p$ -values Lagrange–Bürmann theorem multiple comparisons multiple level multiple test procedure order statistics Schur–Jabotinski theorem step-down test step-up test

Citation

Finner, H.; Roters, M. Multiple hypotheses testing and expected number of type I. errors. Ann. Statist. 30 (2002), no. 1, 220--238. doi:10.1214/aos/1015362191. https://projecteuclid.org/euclid.aos/1015362191


Export citation

References

  • BENJAMINI, Y. and HOCHBERG, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289-300.
  • BENJAMINI, Y. and YEKUTIELI, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165-1188.
  • CSÖRG O, M. and HORVÁTH, L. (1993). Weighted Approximations in Probability and Statistics. Wiley, Chichester.
  • DALAL, S. R. and MALLOWS, C. L. (1992). Buying with exact confidence. Ann. Appl. Probab. 2 752-765.
  • DEMPSTER, A. P. (1959). Generalized D+ n statistics. Ann. Math. Statist. 30 593-597.
  • EKLUND G. and SEEGER, P. (1965). Massignifikansanalys. Statistisk Tidskrift Stockholm, 3rd Ser. 4 355-365.
  • FELLER, W. (1971). An Introduction to Probability Theory and Its Applications 2, 2nd ed. Wiley, New York.
  • FINNER, H., HAYTER, A. J. and ROTERS, M. (1993). On the joint distribution function of order statistics with reference to step-up multiple test procedures. Technical report 93-19, 1-11, Univ. Trier.
  • FINNER, H. and ROTERS, M. (1994). On the limit behaviour of the joint distribution function of order statistics. Ann. Inst. Statist. Math. 46 343-349.
  • FINNER, H. and ROTERS, M. (1998). Asymptotic comparison of step-down and step-up multiple test procedures based on exchangeable test statistics. Ann. Statist. 26 505-524.
  • FINNER, H. and ROTERS, M. (1999). Asymptotic comparison of the critical values of step-down and step-up multiple comparison procedures. J. Statist. Plann. Inference 79 11-30.
  • FINNER, H. and ROTERS, M. (2000). On the critical value behaviour of multiple decision procedures. Scand. J. Statist. 27 563-573.
  • FINNER, H. and ROTERS, M. (2001a). Asymptotic sharpness of product-type inequalities for maxima of random variables with applications in multiple comparisons. J. Statist. Plann. Inference 98 127-144.
  • FINNER, H. and ROTERS, M. (2001b). On the false discovery rate and expected type I errors. Biom. J. 43 985-1005.
  • HENRICI, P. (1974). Applied and Computational Complex Analysis 1. Wiley, New York.
  • HOCHBERG, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800-802.
  • HOCHBERG, Y. and TAMHANE, A. C. (1987). Multiple Comparisons Procedures. Wiley, New York.
  • HOMMEL, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383-386.
  • KARLIN, S. and TAYLOR, H. M. (1981). A Second Course in Stochastic Processes. Academic Press, Boston.
  • MILLER, R. G., JR. (1981). Simultaneous Statistical Inference, 2nd ed. Springer, New York.
  • RÉNYI, A. (1973). On a group of problems in the theory of ordered samples. Selected Translations in Math. Statist. Probab. 13 289-298.
  • ROM, D. M. (1990). A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77 663-665.
  • SARKAR, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 240-258.
  • SEEGER, P. (1966). Variance Analysis of Complete Designs. Some Practical Aspects. Almqvist and Wiksell, Uppsala.
  • SHORACK, G. R. and WELLNER, J. A. (1986). Empirical Processes with Applications to Statistics. Wiley, New York.
  • SIMES, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751-754.
  • SPJØTVOLL, E. (1972). On the optimality of some multiple comparison procedures. Ann. Math. Statist. 43 398-411.