Shaffer's Modified sequentially rejective multiple test procedures presented two more powerful modifications (commonly referred to as S1 and S2) of Holm’s (1979) A simple sequentially rejective multiple test procedure, sequentially rejective Bonferroni procedure. Unfortunately, use of the more-powerful S2 procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer’s S2 procedure to be used in a much larger class of problems.

Theoretical results concerning an aspect of Shaffer’s S2 procedure are derived. Use of these results in implementing the procedure is described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer’s procedure is outlined.

The present work allows Shaffer’s procedure for multiple comparisons to be applied to a large number of groups. Unlike many other methods, no assumptions about the joint distributions of the test statistics need be made. Shaffer’s S2 method and other methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.

Various multiple comparison procedures involve the evaluation of multivariate normal and $t$ integrals with non-decomposable correlation matrices. While exact methods exist for their computations, it is sometimes necessary to consider simpler and faster approximations. We consider approximations based on approximations to the correlation matrix (methods which provide no error control) as well as inequality based methods (where, by definition, the sign of the error is known). Comparisons of different methods, to assess accuracy, are given for particular multiple comparison problems which require high-dimensional integrations.

During the course of a study it would be desirable to take advantage of new internal or external information in order to modify design features laid down in the study protocol. However, for conventional controlled experiments this does not seem feasible without casting doubt on the statistical validity of the whole experiment. In contrast, modifications pertaining to treatment arms, endpoints, hypotheses, statistical methods etc. are possible within adaptive designs, thus enabling the conduct of complex controlled experiments which may still be tailored after onset to meet ethical and scientific as well as economic requirements.

This article briefly reviews recent statistical methods for adaptive study design, particularly those built from $p$-value combination rules or, equivalently, conditional error functions. Its main focus, however, is on the application of adaptive testing methods to clinical experiments with multiple objectives, e.g., multiple treatment arms or endpoints. The authors demonstrate in this overview that as a consequence of using an adaptive interim analysis, null hypotheses may be dropped or added, and test statistics may be exchanged, whilst the studywise type I error rate remains under (strong) control. Moreover, adaptive designs may be applied to experiments aiming to establish doseresponse relationships, or to demonstrate non-inferiority, superiority or equivalence of multiple treatment arms. An example from the literature, which has not previously been discussed from an adaptive viewpoint, is provided as a worked illustration.

Our paper compares the any-pair power, all-pairs power and per-pair power of several test procedures that control either the familywise error rate (FWE) or the false discovery rate (FDR). Our investigations are restricted to one-sided many-one comparisons of normal distributions. Some of the investigated procedures make use of this known special structure and some do not. The numbers $k$ of hypotheses we considered were 5, 10 and 100. The statements with small and large $k$ were quite similar. We found that all methods except one do not essentially differ in their any-pair powers. The most remarkable differences between the test procedures can be observed concerning their all-pairs powers.

We also compared the sample sizes that are necessary for the different procedures to ensure a specified power and investigated their dependency on the number $k$ of hypotheses. With specified per-pair or any-pair power, the FWE controlling methods need similar sample sizes as the FDR controlling methods. With specified all-pairs power, the sample sizes for four of the six FDR controlling procedures studied increase with $k$ at noticeably lower rates than for the FWE controlling procedures.

The accuracy of a medical diagnostic tool depends on its specificity, the probability that it classifies a normal person as normal, and its sensitivity, the probability that it classifies a diseased person as diseased. The receiver operating characteristic (ROC) curve of such a tool is its sensitivity plotted against (1$-$specificity) as the threshold defining "normal" versus "diseased" ranges over all possible values. A common, global measure of the accuracy of a diagnostic tool is the area under the curve (AUC), the curve being the ROC curve. Thus, one way to compare the accuracies of medical diagnostic tools is to compare their AUCs. By comparing each diagnostic tool with the truly most accurate diagnostic tool, one can eliminate diagnostic tools that are not the most accurate, and discover diagnostic tools which are either the most accurate or practically the most accurate. This article shows how the method of multiple comparison with the best (MCB) for normal error general linear models can be adapted to compare diagnostic tools in terms of AUCs of their ROC curves. MCB of AUCs of ROC curves is illustrated by comparing diagnostic variables for predicting the need for emergency Cesarean section, and for predicting the onset of juvenile myopia.

In this article we obtain some new results and extensions of the OLS and GLS tests proposed by O’Brien (1984), Procedures for comparing samples with multiple endpoints, for the one-sided multivariate testing problem. In particular, we empirically obtain an accurate small sample approximation to the critical point of the OLS test. Next we give a power comparison between the OLS test and a competing test proposed by Läuter (1996). Exact t and F tests for analyzing studies with multiple endpoints. Lastly, we extend the OLS and GLS tests to the heteroscedastic setup where the control and treatment populations have different covariance matrices.

Two known results in multiple testing, one relating to the directional error control of augmented step-down procedure proved by Shaffer (1980) and the other on the monotonicity of the critical values of step-up procedure proved by Dalal and Mallows (1992), are revisited and given alternative proofs in this article.

Controlling the false discovery rate has been increasingly utilized in high dimensional screening studies where multiplicity is a problem. Most methods do not explicitly take the correlation between the data or the test statistics into account, with consequent loss of power. In this paper, we use least favorable configurations to obtain critical values for both step-down and step-up procedures, valid for both dependent and independent hypotheses. The concept of a "minimum critical value" (MCV) is introduced. For the step-down case with MCV${} = 0$, our step-down procedure is the same as that of Troendle (2000). It is conjectured that, for a given MCV, there is no uniformly more powerful step-down FDR procedure. Empirical results suggest that, for maximizing power, the "optimum" MCV is a decreasing function of the number of false hypotheses. Various tables are given, with a special "condensed" table valid for numbers of hypotheses from 30 to 10,000 and $\rho = .5$ specifically designed for the case where few false hypotheses are anticipated or where a satisfactory outcome is the discovery of a few false hypotheses. Intermediate values for the latter table may be obtained by interpolation. An application to high dimensional genomic data is given.

This article provides a survey of recent work on comparing more than one experimental treatment to more than one control based on differences in the location parameters. The primary focus is on simultaneous confidence intervals for the amount of improvement that the experimental treatments offer relative to each control. Some discussion of single-step and step-down tests is also included. Both the parametric and nonparametric cases are considered. Data from a medical example are used to illustrate the techniques.

The intersection union test is a standard test in situations where the rejection of all elements of a set of $k$ hypotheses is required. In particular, the intersection union test is known to be uniformly most powerful within a certain class of monotone level$-\alpha$ tests. In this article we consider the special case of $k=2$. We consider the problem of deriving simultaneous confidence intervals which are compatible with the associated test decisions. We apply the general partitioning principle of Finner and Strassburger (2002) to derive a general method to construct confidence intervals which are compatible to a given test. Several examples of partitioning the two-dimensional parameter space are given and their characteristics are discussed in detail. The methods in this paper are illustrated by two gold standard clinical trials, where a new treatment under investigation is compared to both a placebo group and a standard therapy.

With high dimensionality, standard Bonferroni-style procedures can suffer from loss of power, since the significance level $\alpha$ must be divided by $k$ to declare significance. Kropf and Läuter (KL) show that certain data-dependent quadratic forms can be used to "pre-specify hypotheses, which can then be tested in a fixed, data-dependent order, without multiplicity adjustment. In this article we extend the KL procedure to a class of weighted procedures, using the same quadratic forms. The class includes the KL method, the Bonferroni-Holm method, and other, new procedures. We establish strong FWE control for all procedures, and compare power and level of various weighting methods using analytical and simulation results. The method is applied using a high-dimensional mixture model that is suggested by the analysis of real gene expression data.