On the Optimality of Some Multiple Comparison Procedures

Abstract

Optimality criteria formulated in terms of the power functions of the individual tests are given for problems where several hypotheses are tested simultaneously. Subject to the constraint that the expected number of false rejections is less than a given constant $\gamma$ when all null hypotheses are true, tests are found which maximize the minimum average power and the minimum power of the individual tests over certain alternatives. In the common situations in the analysis of variance this leads to application of multiple $t$-tests. In that case the resulting procedure is to use Fisher's "least significant difference," but without a preliminary $F$-test and with a smaller level of significance. Recommendations for choosing the value of $\gamma$ are given by relating $\gamma$ to the probability of no false rejections if all hypotheses are true. Based upon the optimality of the tests, a similar optimality property of joint confidence sets is also derived.