Implementing Shaffer’s multiple comparison procedure for a large number of groups

John R. Donoghue

Source: Y. Benjamini, F. Bretz and S. Sarkar, eds., Recent Developments in Multiple Comparison Procedures (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2004), 1-23.

Abstract

Shaffer's Modified sequentially rejective multiple test procedures presented two more powerful modifications (commonly referred to as S1 and S2) of Holm’s (1979) A simple sequentially rejective multiple test procedure, sequentially rejective Bonferroni procedure. Unfortunately, use of the more-powerful S2 procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer’s S2 procedure to be used in a much larger class of problems.

Theoretical results concerning an aspect of Shaffer’s S2 procedure are derived. Use of these results in implementing the procedure is described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer’s procedure is outlined.

The present work allows Shaffer’s procedure for multiple comparisons to be applied to a large number of groups. Unlike many other methods, no assumptions about the joint distributions of the test statistics need be made. Shaffer’s S2 method and other methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.