Electronic Journal of Statistics

Rank-based multiple test procedures and simultaneous confidence intervals

Frank Konietschke, Ludwig A. Hothorn, and Edgar Brunner

Full-text: Open access


We study simultaneous rank procedures for unbalanced designs with independent observations. The hypotheses are formulated in terms of purely nonparametric treatment effects. In this context, we derive rank-based multiple contrast test procedures and simultaneous confidence intervals which take the correlation between the test statistics into account. Hereby, the individual test decisions and the simultaneous confidence intervals are compatible. This means, whenever an individual hypothesis has been rejected by the multiple contrast test, the corresponding simultaneous confidence interval does not include the null, i.e. the hypothetical value of no treatment effect. The procedures allow for testing arbitrary purely nonparametric multiple linear hypotheses (e.g. many-to-one, all-pairs, changepoint, or even average comparisons). We do not assume homogeneous variances of the data; in particular, the distributions can have different shapes even under the null hypothesis. Thus, a solution to the multiple nonparametric Behrens-Fisher problem is presented in this unified framework.

Article information

Electron. J. Statist., Volume 6 (2012), 738-759.

First available in Project Euclid: 3 May 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Rank statistics multiple comparisons nonparametric Behrens-Fisher problem


Konietschke, Frank; Hothorn, Ludwig A.; Brunner, Edgar. Rank-based multiple test procedures and simultaneous confidence intervals. Electron. J. Statist. 6 (2012), 738--759. doi:10.1214/12-EJS691. https://projecteuclid.org/euclid.ejs/1336049813

Export citation


  • [1] Akritas, M. G., Arnold, S. F. and Brunner, E. (1997). Nonparametric hypotheses and rank statistics for unbalanced factorial designs., Journal of the American Statistical Association 92 258–265.
  • [2] Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one-way classification., Annals of Mathematical Statistics 25 290–302.
  • [3] Bretz, F., Genz, A. and Hothorn, L. A. (2001). On the numerical availability of multiple comparison procedures., Biometrical Journal 43 645–656.
  • [4] Bretz, F., Hothorn T. and Westfall, P. (2010)., Multiple Comparisons Using R, Chapman and Hall, London.
  • [5] Brown, B. M. and Hettmansperger, T. P. (2002). Kruskal-Wallis, multiple comparisons and Efron dice., Australian and New Zealand Journal of Statistics 44 427–438.
  • [6] Brunner, E., Dette, H. and Munk, A. (1997). Box-Type approximations in nonparametric factorial designs., Journal of the American Statistical Association 92 1494–1502.
  • [7] Brunner, E., Domhof, S. and Langer, F. (2002)., Nonparametric Analysis of Longitudinal Data in Factorial Experiments. New York, Wiley.
  • [8] Brunner, E. and Munzel, U. (2000). The nonparametric Behrens-Fisher problem - Asymptotic theory and small sample approximations., Biometrical Journal 42 17–25.
  • [9] Brunner E., Munzel U. and Puri M. L. (2002). The multivariate nonparametric Behrens-Fisher problem., Journal of Statistical Planning and Inference 108 37–53.
  • [10] Brunner, E. and Puri, M. L. (2001). Nonparametric methods in factorial designs., Statistical Papers 42 1–52.
  • [11] Dunn, O. J. (1960). Multiple comparisons using rank sums., Technometrics 2 241–252.
  • [12] Efron, B. and Tibshirani, R. J. (1993)., An Introduction to the Bootstrap. New York, Chapman and Hall.
  • [13] Elliott, A. C. and Hynan, L. S. (2011). A SAS macro implementation of a multiple comparison post hoc test for a Kruskal-Wallis analysis., Computer Methods and Programs in Biomedicine 102 75 –80.
  • [14] Gabriel, K. R. (1969). Simultaneous test procedures - some theory of multiple comparisons., Annals of Mathematical Statistics 40 224–250.
  • [15] Gao, X. and Alvo, M. (2005). A unified nonparametric approach for unbalanced factorial designs., Journal of the American Statistical Association 100 926–941.
  • [16] Gao, X. and Alvo, M. (2008). Nonparametric multiple comparison procedures for unbalanced two-way factorial designs., Journal of Statistical Planning and Inference 138 3674–3686.
  • [17] Gao, X., Alvo, M., Chen, J. and Li, G. (2008). Nonparametric multiple comparison procedures for unbalanced one-way factorial designs., Journal of Statistical Planning and Inference 138 2574–2591.
  • [18] Gardner, M. (1974). On the paradoxical situation that arise from nontransitive relations., Scientific American 231 120–125.
  • [19] Genz, A. and Bretz, F. (2009)., Computation of multivariate normal and t probabilities. Lecture Notes in Statistics, New York, Springer.
  • [20] Hasler, M. and Hothorn, L. A. (2008). Multiple contrast tests in the presence of heteroscedasticity., Biometrical Journal 50 793–800.
  • [21] Herberich, E., Sikorski, J. and Hothorn, T. (2010). A robust procedure for comparing multiple means under heteroscedasticity in unbalanced designs., PLoS ONE, DOI:10.1371/journal.pone.0009788.
  • [22] Hochberg, Y. and Tamhane, A. C. (1987)., Multiple Comparison Procedures. Wiley, New York.
  • [23] Hothorn, T., Bretz, F. and Westfall, P. (2008). Simultaneous inference in general parametric models., Biometrical Journal 50, 346–363.
  • [24] ICH (1998)., Statistical Principles for Clinical Trials. Guideline, Available at http://private.ich.org.
  • [25] Konietschke, F. and Hothorn, L. A. (2012). Evaluation of toxicological studies using a nonparametric Shirley-Type trend test for comparing several dose levels with a control group., Statistics in Biopharmaceutical Research, in press (DOI: 10.1080/19466315.2011.633861).
  • [26] Konietschke, F., Libiger, O. and Hothorn, L. A. (2012).Nonparametric Evaluation of Quantitative Traits in Population-Based Association Studies when the Genetic Model is Unknown., PLoS ONE 7(2): e31242. doi:10.1371/journal.pone.0031242.
  • [27] Lévy, P. (1925)., Calcul des probabilitées, Gauthiers-Villars, Paris.
  • [28] Mukerjee, H., Robertson, T. and Wright, F. T. (1987). Comparison of several treatments with a control using multiple contrasts., Journal of the American Statistical Association 82 902–910.
  • [29] Munzel, U. (1999). Linear rank score statistics when ties are present., Statistics and Probability Letters 41 389–395.
  • [30] Munzel, U. and Hothorn, L. A. (2001). A unified approach to simultaneous rank test procedures in the unbalanced one-way layout., Biometrical Journal 43 553–569.
  • [31] Orban, J. and Wolfe, D. A. (1982). A class of distribution-free two-sample tests based on placements., Journal of the American Statistical Association 77 666–672.
  • [32] Puri, M. L. (1964). Asymptotic efficiency of a class of c-sample tests., Annals of Mathematical Statistics 35 102–121.
  • [33] Rust, S. W. and Fligner, M. A. (1984). A modification of the Kruskal-Wallis statistic for the generalized Behrens-Fisher problem., Communications in Statistics - Theory and Methods 13 2013–2027.
  • [34] Ruymgaart, F. H. (1980). A unified approach to the asymptotic distribution theory of certain midrank statistics., Statistique non Parametrique Asymptotique. J.P. Raoult (Ed.), Lecture Notes on Mathematics, No. 821, Springer, Berlin.
  • [35] Ryu, E. (2009). Simultaneous confidence intervals using ordinal effect measures for ordered categorical outcomes., Statistics in Medicine 28 3179–3188.
  • [36] Ryu, E. and Agresti, A. (2008). Modeling and inference for an ordinal effect size measure., Statistics in Medicine 27 1703–1717.
  • [37] Steel, R. D. G. (1959). A multiple comparison rank test: treatments versus control., Biometrics 15 560–572.
  • [38] Steel, R. D. G. (1960). A rank sum test for comparing all pairs of treatments., Technometrics 2 197–207.
  • [39] Thangavelu, K. and Brunner, E. (2006). Wilcoxon Mann-Whitney test and Efrons paradox dice., Journal of Statistical Planning and Inference 137 720–737.
  • [40] Wolfsegger, M. and Jaki, T. (2006). Simultaneous confidence intervals by iteratively adjusted alpha for relative effects in the one-way layout., Statistics and Computing 16 15–23.