Electronic Journal of Statistics

On stepwise control of the generalized familywise error rate

Wenge Guo and M. Bhaskara Rao

Full-text: Open access


A classical approach for dealing with a multiple testing problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of at least one false rejection. In many applications, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of a procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which is called the k-FWER. In this article, a unified approach is presented for deriving the k-FWER controlling procedures. We first generalize the well-known closure principle in the context of the FWER to the case of controlling the k-FWER. Then, we discuss how to derive the k-FWER controlling stepup procedures based on marginal p-values using this principle. We show that, under certain conditions, generalized closed testing procedures can be reduced to stepup procedures, and any stepup procedure is equivalent to a generalized closed testing procedure. Finally, we generalize the well-known Hommel procedure in two directions, and show that any generalized Hommel procedure is equivalent to a generalized closed testing procedure with the same critical values.

Article information

Electron. J. Statist., Volume 4 (2010), 472-485.

First available in Project Euclid: 1 June 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J15: Paired and multiple comparisons
Secondary: 62G10: Hypothesis testing

Closure principle generalized familywise error rate Hommel procedure multiple testing p-values stepup procedure


Guo, Wenge; Rao, M. Bhaskara. On stepwise control of the generalized familywise error rate. Electron. J. Statist. 4 (2010), 472--485. doi:10.1214/08-EJS320. https://projecteuclid.org/euclid.ejs/1275403739

Export citation


  • [1] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing., J. R. Statist. Soc. B, 57, 289–300.
  • [2] Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency., Ann. Statist., 29, 1165–1188.
  • [3] Bernhard, G., Klein, M. and Hommel, G. (2004). Global and multiple test procedures using ordered p-values – A review., Statist. Pap., 45, 1–14.
  • [4] Calian, V., Li, D., and Hsu, J. (2008). Partitioning to uncover conditions for permutation to control multiple testing error rate., Biometrical Journal, 50, 756–766.
  • [5] Dudoit, S. and van der Laan, M. (2008)., Multiple Testing Procedures with Applications to Genomics. Springer, New York.
  • [6] Dudoit, S., van der Laan, M. and Pollard, K. (2004). Multiple testing. Part I. Single-step procedures for control of general type I error rates., Statist. Appl. Gen. Mol. Biol., 3(1), Article 13.
  • [7] Falk, R. W. (1989). Hommel’s Bonferroni-type inequality for unequally spaced levels., Biometrika, 76, 189–191.
  • [8] Finner, H. and Strassburger, K. (2002). The partititoning principle: a powerful tool in multiple decision theory., Ann. Statist., 30 1194–1213.
  • [9], k-FWER control without multiplicity correction, with application to detection of genetic determinants of multiple sclerosis in Italian twins. Biometrics, in press.
  • [10] Grechanovsky, E. and Hochberg, Y. (1999). Closed procedures are better and often admit a shortcut., J. Statist. Plann. Inf., 76, 79–91.
  • [11] Guo, W. and Romano, J. (2007). A generalized Sidak-Holm procedure and control of generalized error rates under independence., Statist. Appl. Gen. Mol. Biol., 6(1), Article 3.
  • [12] Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance., Biometrika, 75, 800–802.
  • [13] Holm, S. (1979). A simple sequentially rejective multiple test procedure., Scand. J. Statist., 6 65–70.
  • [14] Hommel, G. (1986). Multiple test procedures for arbitrary dependence structures., Metrika, 33, 321–336.
  • [15] Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test., Biometrika, 75, 383–386.
  • [16] Hommel, G. (1989). A comparison of two modified Bonferroni procedures., Biometrika, 76, 624–625.
  • [17] Hommel, G. and Hoffmann, T. (1987). Controlled uncertainty. In, Multiple Hypothesis Testing (eds P. Bauer, G. Hommel, and E. Sonnemann), 154-162. Springer, Heidelberg.
  • [18] Huang, Y. and Hsu, J. (2007). Hochberg’s step-up method : cutting corners off Holm’s step-down method., Biometrika, 94, 965–975.
  • [19] Karlin, S. and Rinott, Y. (1980). Classes of orderings of measures and related correlation inequalities. Part I. Multivariate totally positive distributions., J. Mult. Anal., 10, 467–498.
  • [20] Korn, E., Troendle, J., McShane, L. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data., J. Statist. Plann. Inf., 124, 379–398.
  • [21] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate., Ann. Statist., 33, 1138–1154.
  • [22] Liu, W. (1996). Multiple tests of a non-hierarchical family of hypotheses., J. R. Statist. Soc. B, 58, 455–461.
  • [23] Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance., Biometrika, 63, 655–660.
  • [24], Röhmel, J. and Streitberg, B. (1987). Zur Konstruktion globaler Tests. EDV Med. Biol., 18, 7–11.
  • [25] Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate., Ann. Statist., 34, 1850–1873.
  • [26] Romano, J. P. and Wolf, M. (2007). Control of generalized error rates in multiple testing., Ann. Statist., 35, 1378–1408.
  • [27] Rüger, B. (1978). Das maximale Signifikanzniveau des Tests “Lehne, H0 ab, wenn k unter n gegebenen Tests zur Ablehnung führen”. Metrika, 25, 171–178.
  • [28] Sarkar, S. K. (1998). Some probability inequalities for ordered, MTP2 random variables: a proof of the Simes conjecture. Ann. Statist., 26, 494–504.
  • [29] Sarkar, S. K. (2007). Stepup procedures controlling generalized FWER and generalized FDR., Ann. Statist. 35 2405–2420.
  • [30] Sarkar, S. K. (2008). Generalizing Simes’ test and Hochberg’s stepup procedure., Ann. Statist. 36, 337–363.
  • [31] Sarkar, S. K. and Guo, W. (2009a). On a generalized false discovery rate., Ann. Statist., 37, 337–363.
  • [32] Sarkar, S. K. and Guo, W. (2009b). Procedures controlling generalized false discovery rate., Statistica Sinica, in press.
  • [33] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance., Biometrika, 73, 751–754.
  • [34] Stefansson, G., Kim, W. and Hsu, J. C. (1988). On confidence sets in multiple comparisons. In Gupta, S. S. and Berger, J. O. (eds.), Statistical Decision Theory and Related Topics IV, Vol. 2, 89–104. Springer-Verlag, New York.
  • [35] van der Laan, M., Birkner, M. and Hubbard, A. (2005). Resampling based multiple testing procedure controlling tail probability of the proportion of false positives., Statist. Appl. Gen. Mol. Biol., 4(1), Article 29.
  • [36] van der Laan, M., Dudoit, S. and Pollard, K. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives., Statist. Appl. Gen. Mol. Biol., 3(1), Article 15.
  • [37] Xu, H. and Hsu, J. (2007). Using the Partitioning Principle to control the generalized Family Error Rate., Biometrical Journal, 49, 52–67.
  • [38] Yekutieli, D. (2008). False discovery rate control for non-positively regression dependent test statistics., Journal of Statistical Planning and Inference, 138, 405–415.