Design for the Control of Selection Bias

Abstract

Suppose an experimenter $E$ wishes to compare the effectiveness of two treatments, $A$ and $B$, on a somewhat vaguely defined population. As individuals arrive, $E$ decides whether they are in the population, and if he decides that they are, he administers $A$ or $B$ and notes the result, until $nA$'s and $nB$'s have been administered. Plainly, if $E$ is aware, before deciding whether an individual is in the population, which treatment is to be administered next, he may, not necessarily deliberately, introduce a bias into the experiment. This bias we call selection bias. We propose to investigate the extent to which a statistician $S$, by determining the order in which treatments are administered, and not revealing to $E$ which treatment comes next until after the individual who is to receive it has been selected, can control this selection bias. Thus a design $d$ is a distribution over the set $T$ of the $\binom{2n}{n}$ sequences of length $2n$ containing $nA$'s and $nB's$. We shall measure the bias of a design by the maximum expected number of correct guesses which an experimenter can achieve, knowing $d$, attempting to guess the successive elements of a sequence $t \varepsilon T$ selected by $d$, and being told after each guess whether or not it is correct. The distribution of the number $G$ of correct guesses depends both on $d$ and on the prediction method $p$ used by the experimenter. We shall consider particularly two designs, the truncated binomial, in which the successive treatments are selected independently with probability $\frac{1}{2}$ each until $n$ treatments of one kind have occurred, and the sampling design, in which all $\binom{2n}{n}$ sequences are equally likely. We shall consider particularly two prediction methods, the convergent prediction, which predicts that treatment which has hitherto occurred less often, and the divergent prediction, which predicts that treatment which has hitherto occurred more often, except that after $n$ treatments of one kind have been administered, the divergent prediction agrees with the convergent predictions that the other treatment will follow; when both treatments have occurred equally often, either method predicts $A$ or $B$ by tossing a fair coin, independently for each case of equality. We find that among all designs, the truncated binomial minimizes the maximum expected number of correct guesses. For this design, the expected number of correct guesses is independent of the prediction method, and is $$n + n \binom{2n}{n} \big/ 2^{2n} \sim n + (n/\pi)^{1/2}$$ With the truncated binomial design, the variance in the number of correct guesses is largest for the divergence strategy and is $$3n/2 - D - D^2/4 \sim (3\pi - 2)n/2\pi - 2(n/\pi)^{1/2},$$ where $D = n \binom{2n}{n} \big/ 2^{2n - 1}$, and is smallest for the convergence strategy, and is $n/2 - D^2/4 \sim (\pi - 1)n/2\pi$. For the sampling design, convergent prediction maximizes the expected number of correct guesses; this maximum is $$n + 2^{2n - 1} \big/ \binom{2n}{n} - \frac{1}{2} \sim n + (\pi n/4)^{1/2}.$$ Finally we note that, if treatments are selected independently at random, bias of the kind we discuss disappears, but the treatment numbers can no longer be preassigned. Three such designs are considered: the fixed total design, in which the total number of treatments is a fixed number $s$, the fixed factor design, in which we continue until $1/X + 1/Y \leqq 2/n$, where $X$ is the number of $A$ treatments and $Y$ is the number of $B$ treatments administered, and the fixed minimum design, in which we continue until $\min (X, Y) = n$. For the fixed total design, we find that, for $s = 2n + 4, \mathrm{Pr}(1/X + 1/Y \leqq 2/n) \sim 0.955$ for large $n$; at the expense of 4 extra observations, we have a bias-free design whose variance factor will with probability 0.955 be smaller than that in which treatment numbers are preassigned. For the fixed factor design, the additional number of observations required to achieve the given precision has for large $n$ the distribution of the square of a normal deviate. For the fixed minimum design, in which we guarantee precision for the estimated effect of each treatment, the expected number of additional observations is roughly 1.13 $(n)^{1/2}$.