Annals of Statistics

Generalizing Simes’ test and Hochberg’s stepup procedure

Sanat K. Sarkar

Full-text: Open access


In a multiple testing problem where one is willing to tolerate a few false rejections, procedure controlling the familywise error rate (FWER) can potentially be improved in terms of its ability to detect false null hypotheses by generalizing it to control the k-FWER, the probability of falsely rejecting at least k null hypotheses, for some fixed k>1. Simes’ test for testing the intersection null hypothesis is generalized to control the k-FWER weakly, that is, under the intersection null hypothesis, and Hochberg’s stepup procedure for simultaneous testing of the individual null hypotheses is generalized to control the k-FWER strongly, that is, under any configuration of the true and false null hypotheses. The proposed generalizations are developed utilizing joint null distributions of the k-dimensional subsets of the p-values, assumed to be identical. The generalized Simes’ test is proved to control the k-FWER weakly under the multivariate totally positive of order two (MTP2) condition [J. Multivariate Analysis 10 (1980) 467–498] of the joint null distribution of the p-values by generalizing the original Simes’ inequality. It is more powerful to detect k or more false null hypotheses than the original Simes’ test when the p-values are independent. A stepdown procedure strongly controlling the k-FWER, a version of generalized Holm’s procedure that is different from and more powerful than [Ann. Statist. 33 (2005) 1138–1154] with independent p-values, is derived before proposing the generalized Hochberg’s procedure. The strong control of the k-FWER for the generalized Hochberg’s procedure is established in situations where the generalized Simes’ test is known to control its k-FWER weakly.

Article information

Ann. Statist., Volume 36, Number 1 (2008), 337-363.

First available in Project Euclid: 1 February 2008

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J15: Paired and multiple comparisons

Global testing multiple testing single-step procedure stepdown procedure stepup procedure generalized Bonferroni procedure generalized Holm’s procedure generalized Hochberg’s procedure


Sarkar, Sanat K. Generalizing Simes’ test and Hochberg’s stepup procedure. Ann. Statist. 36 (2008), no. 1, 337--363. doi:10.1214/009053607000000550.

Export citation


  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Dudoit, S., Shaffer, J. P. and Boldrick, J. C. (2003). Multiple hypothesis testing in microarray experiments. Statist. Sci. 18 71–103.
  • Dudoit, S., van der Laan, M. and Pollard, K. (2002). Multiple testing: Part I. Single-step procedures for control of general type I error rates. Statist. Appl. Gen. Mol. Biol. 3 Article 13.
  • Guo, W. and Rao, M. B. (2006). On generalized closure principle for generalized familywise error rates. Unpublished report.
  • Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800–802.
  • Hochberg, Y. and Tamhane, A. C. (1987). Multiple Comparison Procedures. Wiley, New York.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70.
  • Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383–386.
  • Hommel, G. and Hoffman, T. (1987). Controlled uncertainty. In Multiple Hypothesis Testing (P. Bauer, G. Hommel and E. Sonnemann, eds.) 154–161. Springer, Heidelberg.
  • Karlin, S. and Rinott, Y. (1980). Classes of orderings of measures and related correlation inequalities I: Multivariate totally positive distributions. J. Multivariate Analysis 10 467–498.
  • Korn, E., Troendle, T., McShane, L. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data. J. Statist. Plann. Inf. 124 279–398.
  • Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34 1850–1873.
  • Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica 73 1237–1282.
  • Sarkar, S. K. (1998). Some probability inequalities for ordered MTP2 random variables: A proof of the Simes conjecture. Ann. Statist. 26 494–504.
  • Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
  • Sarkar, S. K. (2007). Stepup procedures controlling generalized FWER and generalized FDR. Ann. Statist. To appear.
  • Sarkar, S. K. and Chang, C.-K. (1997). Simes method for multiple hypothesis testing with positively dependent test rtatistics. J. Amer. Statist. Assoc. 92 1601–1608.
  • Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • Tong, Y. L. (1989). Inequalities for a class of positively dependent random variables with a common marginal. Ann. Statist. 17 429–435.
  • van der Laan, M., Dudoit, S. and Pollard, K. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat. Appl. Gen. Mol. Biol. 3 Article 15.