International Statistical Review

Simple Stepwise Tests of Hypotheses and Multiple Comparisons

Eugene Seneta and John T. Chen

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Holm's (1979) step-down and Hochberg's (1988) step-up procedures for tests of multiple hypotheses are simple to apply and are widely used. Holm's procedure controls the familywise error rate (FWE), while Hochberg's is more powerful. This paper investigates a step-down procedure (labelled CS) of Seneta & Chen (1997) which is a sharpening of Holm's, takes into account the degree of association between test statistics, and also controls the FWE. Computation for the CS procedure may be minimized by using the procedure as an adjustment to Holm's. The computational steps are detailed, and the adjustment is then illustrated by an application to a text-book example of multiple comparisons, in which step-wise procedures are shown to perform better than the usual Tukey T-comparison. Simulation investigations in a standard comparison with a control setting show that the CS step-down procedure is more powerful than Hochberg's step-up procedure and the procedure of Simes (1986), especially in regard to error rate, and not much less powerful than an optimal, but very specific, step-up procedure of Dunnett & Tamhane (1992).

Article information

Internat. Statist. Rev., Volume 73, Number 1 (2005), 21-34.

First available in Project Euclid: 31 March 2005

Permanent link to this document

Zentralblatt MATH identifier

Multiple hypotheses Step-down tests Familywise error rate Holm's procedure Hochberg's procedure Multiple comparisons P-value Correlation Simes' test Dunnett's test Multivariate t-distribution


Seneta, Eugene; Chen, John T. Simple Stepwise Tests of Hypotheses and Multiple Comparisons. Internat. Statist. Rev. 73 (2005), no. 1, 21--34.

Export citation


  • [1] Chen, J.T. & Seneta, E. (1999). A stepwise rejective test procedure with strong control of familywise error rate. Bulletin of the International Statistical Institute, ISI 99, 52nd Session. Contributed papers. Tome LVII, Three Books. Book 3. Helsinki, 1999, pp. 241-242.
  • [2] Devore, J.L. (2000). Probability and Statistics for Engineering and the Sciences. Pacific Grove, CA: Duxbury.
  • [3] Dunnett, C.W. & Tamhane, A.C. (1992). A step-up multiple test procedure. J. Amer. Statist. Assoc., 87, 162-170.
  • [4] Hattan, D.G. & Eacho, P.I. (1978). Relationship of ethanol blood level to REM and non-REM sleep time and distribution in the rat. Life Sciences, 22, 839-846.
  • [5] Hayter, A.J. (1984). A proof of the conjecture that the Tukey-Kramer multiple comparisons procedure is conservative. Annals of Statistics, 12, 61-75.
  • [6] Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800-802.
  • [7] Hochberg, Y. & Rom, D. (1995). Extensions of multiple testing procedures based on Simes' test. J. Statist. Plan. Inf., 48, 141-152.
  • [8] Hochberg, Y. & Tamhane, A.C. (1987). Multiple Comparison Procedures. New York: Wiley.
  • [9] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65-70.
  • [10] Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75, 383-386.
  • [11] Hommel, G. (1989). A comparison of two modified Bonferroni procedures. Biometrika, 76, 624-625.
  • [12] Hunter, D. (1976). An upper bound for the probability of a union. Journal of Applied Probability, 13, 597-603.
  • [13] Kounias, E.G. (1968). Bounds for the probability of a union of events, with applications. Ann. Math. Statist., 39, 2154-2158.
  • [14] Margaritescu, E. (1986). A note on Bonferroni's inequalities. Biometrical Journal, 28, 937-943.
  • [15] Miller, R.G. (1981). Simultaneous Statistical Inference. 2nd Edition. New York: Springer.
  • [16] Miller, R.G. (1985). Multiple comparisons. In Encyclopedia of Statistical Sciences, Eds. S. Kotz and N.L. Johnson, 5, pp. 679-689. New York: Wiley.
  • [17] Miller, R.G. (1997). Beyond ANOVA, Basics of Applied Statistics. London: Chapman and Hall.
  • [18] Rom, D.M. (1990). A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika, 77, 663-666.
  • [19] Samuel-Cahn, E. (1996). Is Simes improved Bonferroni procedure conservative? Biometrika, 83, 929-933.
  • [20] Sarkar, S.K. (1998). Some probability inequalities for ordered MTP_2 random variables: a proof of the Simes conjecture. Ann. Statist., 26, 494-504.
  • [21] Sarkar, S.K. & Chang, C.K. (1997). The Simes method for multiple hypothesis testing with positively dependent test statistics. J. Amer. Statist. Assoc., 92, 1601-1608.
  • [22] Seneta, E. (1988). Degree, iteration and permutation in improving Bonferroni-type bounds. Australian J. Statist., 30A, 27-38.
  • [23] Seneta, E. (1993). Probability Inequalities and Dunnett's Test. In Multiple Comparisons, Selection, and Applications in Biometry, Ed. F.M. Hoppe, Chapter 3, pp 29-45. New York: Marcel Dekker.
  • [24] Seneta, E. & Chen, T. (1997). A sequentially rejective test procedure. Theory of Stochastic Processes, 3(19), No. 3-4, 393-402. Kyiv, Ukraine: TBIMC Scientific Publishers.
  • [25] Shaffer, J.P. (1986). Modified sequentially rejective multiple test procedure. J. Amer. Statist. Assoc., 81, 826-831.
  • [26] Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73, 751-754.
  • [27] Stoline, M.R. (1983). The Hunter method of simultaneous inference and its recommended use for applications having large known correlation structures. J. Amer. Statist. Assoc., 78, 366-370.
  • [28] Wright, S.P. (1992). Adjusted P-Values for simultaneous inference. Biometrics, 48, 1005-1013.