Institute of Mathematical Statistics Collections

Multiple testing of pairwise comparisons

Arthur Cohen, Harold Sackrowitz, and Chuanwen Chen

Full-text: Open access


Multiple testing for pairwise comparisons in a one way fixed and balanced analysis of variance model is studied. Normality, independence and homogeneity of variance is assumed. Two sided alternatives are considered. The usual stepwise procedures are shown to lack an intuitive and important interval property for acceptance sections of individual tests. This renders them inadmissible in terms of both types of errors. Alternative procedures that do have the interval property are suggested. The new procedures are compared to the standard procedures in terms of power and in terms of practicality. One of the new procedures follows logical restrictions which may be a desirable property in some instances.

Chapter information

James O. Berger, T. Tony Cai and Iain M. Johnstone, eds., Borrowing Strength: Theory Powering Applications – A Festschrift for Lawrence D. Brown (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2010), 144-157

First available in Project Euclid: 26 October 2010

Permanent link to this document

Digital Object Identifier

Primary: 62H15: Hypothesis testing 62C15: Admissibility

admissibility logical restrictions maximum residual down (MRD) step-down procedure Tukey’s method

Copyright © 2010, Institute of Mathematical Statistics


Cohen, Arthur; Sackrowitz, Harold; Chen, Chuanwen. Multiple testing of pairwise comparisons. Borrowing Strength: Theory Powering Applications – A Festschrift for Lawrence D. Brown, 144--157, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2010. doi:10.1214/10-IMSCOLL610.

Export citation


  • [1] Benjamini, Y. and Gavrilov, Y. (2009). A simple forward selection procedure based on false discovery rate control. Ann. Appl. Statist. 3 179–198.
  • [2] Cohen, A. and Sackrowitz, H. B. (2008). Multiple testing to two-sided alternatives with dependent data. Statist. Sinica 18 1593–1602.
  • [3] Cohen, A., Sackrowitz, H. B. and Xu, M. (2009). A new multiple testing method in the dependent case. Ann. Statist. 37 1518–1544.
  • [4] Dudoit, S. and Van Der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genomics. Springer.
  • [5] Devore, J. L. (2008). Probability and Statistics for Engineering and the Sciences: Enhanced, 7th ed. Duxbury Pr.
  • [6] Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. Roy. Statist. Soc. Ser. B 64 499–518.
  • [7] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70.
  • [8] Ishwaran, H. and Rao, J. S. (2003). Detecting differentially expressed genes in microarrays using Bayesian model selection. J. Amer. Statist. Assoc. 98 438–455.
  • [9] Lehmacher, W., Wassmer, G. and Reitmeir, P. (1991). Procedures for two-sample comparisons with multiple endpoints controlling the experimentwise error rate. Biometrics 47 511–521.
  • [10] Lehmann, E. L. (1957). A theory of some multiple decision problems, I. Ann. Math. Statist. 28 1–25.
  • [11] Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses, 2nd ed. Springer.
  • [12] Matthes, T. K. and Truax, D. R. (1967). Tests of composite hypotheses for the multivariate exponential family. Ann. Math. Statist. 38 681–697.
  • [13] Royen, T. (1989). Generalized maximum range tests for pairwise comparisons of several populations. Biometrical Journal 31 905–929.
  • [14] Shaffer, J. P. (1986). Modified sequentially rejective multiple test procedures. J. Amer. Statist. Assoc. 81 826–831.
  • [15] Westfall, P. H. and Tobias, R. D. (2007). Multiple testing of general contrasts: Truncated closure and the extended Shaffer-Royen method. J. Amer. Statist. Assoc. 102 487–494.