The Annals of Statistics

The sequential rejection principle of familywise error control

Jelle J. Goeman and Aldo Solari

Full-text: Open access


Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of Holm, Shaffer and Hochberg, parallel and serial gatekeeping procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves. We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure.

Article information

Ann. Statist., Volume 38, Number 6 (2010), 3782-3810.

First available in Project Euclid: 30 November 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62H15: Hypothesis testing
Secondary: 62J15: Paired and multiple comparisons

Multiple testing familywise error rate graph


Goeman, Jelle J.; Solari, Aldo. The sequential rejection principle of familywise error control. Ann. Statist. 38 (2010), no. 6, 3782--3810. doi:10.1214/10-AOS829.

Export citation


  • Bretz, F., Maurer, W., Brannath, W. and Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Stat. Med. 28 586–604.
  • Burman, C.-F., Sonesson, C. and Guilbaud, O. (2009). A recycling framework for the construction of Bonferroni-based multiple tests. Stat. Med. 28 739–761.
  • Calian, V., Li, D. M. and Hsu, J. C. (2008). Partitioning to uncover conditions for permutation tests to control multiple testing error rates. Biometrical J. 50 756–766.
  • Dmitrienko, A., Offen, W. W. and Westfall, P. H. (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Stat. Med. 22 2387–2400.
  • Dmitrienko, A. and Tamhane, A. C. (2007). Gatekekeeping procedures with clinical trial applications. Pharmaceutical Statistics 6 171–180.
  • Dmitrienko, A., Tamhane, A. C., Wang, X. and Chen, X. (2006). Stepwise gatekeeping procedures in clinical trial applications. Biometrical J. 48 984–991.
  • Dmitrienko, A., Tamhane, A. C. and Wiens, B. L. (2008). General multistage gatekeeping procedures. Biometrical J. 50 667–677.
  • Dmitrienko, A., Wiens, B. L., Tamhane, A. C. and Wang, X. (2007). Tree-structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives. Stat. Med. 26 2465–2478.
  • Dudoit, S. and Van der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genomics. Springer, New York.
  • Dudoit, S., Van der Laan, M. J. and Pollard, K. S. (2004). Multiple testing part I: Single-step procedures for control of general type I error rates. Stat. Appl. Genet. Mol. Biol. 3 Article 13.
  • Edwards, D. and Madsen, J. (2007). Constructing multiple test procedures for partially ordered hypotheses sets. Stat. Med. 26 5116–5124.
  • Finner, H. and Strassburger, K. (2002). The partitioning principle: A powerful tool in multiple decision theory. Ann. Statist. 30 1194–1213.
  • Ge, Y., Dudoit, S. and Speed, T. P. (2003). Resampling-based multiple testing for microarray data analysis. Test 12 1–77.
  • Goeman, J. J. and Mansmann, U. (2008). Multiple testing on the directed acyclic graph of gene ontology. Bioinformatics 24 537–544.
  • Guilbaud, O. (2007). Bonferroni parallel gatekeeping—Transparant generalizations, adjusted p-values, and short direct proofs. Biometrical J. 49 917–927.
  • Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800–802.
  • Hochberg, Y. and Rom, D. (1995). Extensions of multiple testing procedures based on Simes’ test. J. Statist. Plann. Inference 48 141–152.
  • Holland, B. S. and DiPinzio Copenhaver, M. (1987). An improved sequentially rejective Bonferroni test procedure. Biometrics 43 417–423.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70.
  • Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383–386.
  • Hommel, G. and Bernhard, G. (1999). Bonferroni procedures for logically related hypotheses. J. Statist. Plann. Inference 82 119–128.
  • Hommel, G., Bretz, F. and Maurer, W. (2007). Powerful short-cuts for multiple testing procedures with special reference to gatekeeping strategies. Stat. Med. 26 4063–4073.
  • Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses. Springer, New York.
  • Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660.
  • Meinshausen, N. (2008). Hierarchical testing of variable importance. Biometrika 95 265–278.
  • Romano, J. and Wolf, M. (2005). Exact and approximate stepdown methods for multiple hypotheses testing. J. Amer. Statist. Assoc. 100 94–108.
  • Romano, J. and Wolf, M. (2010). Balanced control of generalized error rates. Ann. Statist. 38 598–633.
  • Rosenbaum, P. R. (2008). Testing hypotheses in order. Biometrika 25 248–252.
  • Sarkar, S. K. (1998). Some probability inequalities for ordered MTP2 random variables: A proof of the Simes conjecture. Ann. Statist. 26 494–504.
  • Shaffer, J. P. (1986). Modified sequentially rejective multiple test procedures. J. Amer. Statist. Assoc. 81 826–831.
  • Šidák, Z. (1967). Rectangular confidence regions for the means of multivariate normal distributions. J. Amer. Statist. Assoc. 62 626–633.
  • Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • Stefansson, G., Kim, W. and Hsu, J. C. (1988). On confidence sets in multiple comparisons. In Statistical Decision Theory and Related Topics IV (S. S. Gupta and J. O. Berger, eds.) 2 89–104. Springer, New York.
  • Westfall, P. H. and Krishen, A. (2001). Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. J. Statist. Plann. Inference 99 25–40.
  • Westfall, P. H. and Troendle, J. F. (2008). Multiple testing with minimal assumptions. Biometrical J. 50 745–755.
  • Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. Wiley, New York.
  • Wiens, B. L. and Dmitrienko, A. (2005). The fallback procedure for evaluating a single family of hypotheses. J. Biopharm. Statist. 15 929–942.