Significance Level and Power

Abstract

Significance testing, as described in most textbooks, consists in fixing a standard significance level $\alpha$ such as .01 or .05 and rejecting the hypothesis $\theta = \theta_0$ if a suitable statistic $Y$ exceeds $C$ where $P_{\theta_0}\{Y > C\} = \alpha$. Such a procedure controls the probability of false rejection (error of the first kind) at the desired level $\alpha$ but leaves the power of the test and hence the probability of an error of the second kind to the mercy of the experiment. It seems more natural when deciding on a significance level (and this suggestion is certainly not new) to take into account also what power can be achieved with the given experiment. In Section 3 a specific suggestion will be made as to how to balance $\alpha$ against the power $\beta$ obtainable against the alternatives of interest. The adoption of this or some similar rule for choosing a significance level has important consequences for the theory of testing composite hypotheses, where nuisance parameters are present. Since the quantity $\alpha$ is then potentially a function of the nuisance parameter $\vartheta$, the classical rule of a fixed significance level leads to the condition that the tests be exact or similar, that is, that $\alpha(\vartheta)$ equal the preassigned value $\alpha$ for all $\vartheta$. On the other hand, the power $\beta$ that can be attained against any alternative $\theta = \theta_1$ frequently depends on $\vartheta$. The requirement that $\alpha(\vartheta)$ and $\beta(\vartheta)$ be in a certain balance thus leads to tests which are not similar and hence do not agree with the standard solutions. To obtain a suitable setting for this discussion, we consider first a minimal complete class of tests for testing the hypothesis $H:\theta \leqq \theta_0$ in a multiparameter exponential family (Section 2). The proposed $\alpha, \beta$-relation is discussed in Section 3, and in Section 4 is applied to the exponential family. Section 5 gives some illustrations of the theory.