On Permutation Tests for Hidden Biases in Observational Studies: An Application of Holley's Inequality to the Savage Lattice

Abstract

Randomized experiments and observational studies both attempt to estimate the effects produced by a treatment, but in observational studies, subjects are not randomly assigned to treatment or control. A theory of observational studies would closely resemble the theory for randomized experiments in all but one critical respect: In observational studies, the distribution of treatment assignments is not known. The problems that are special to observational studies revolve around our uncertainty about how treatments were assigned. In this connection, tools are needed for describing distributions of treatment assignments that do not assign equal probabilities to all assignments. Two such tools are a lattice of treatment assignments first studied by Savage and an inequality due to Holley for probability distributions on a lattice. Using these tools, it is shown that certain permutation tests are unbiased as tests of the null hypothesis that the distribution of treatment assignments resembles a randomization distribution against the alternative hypothesis that subjects with higher responses are more likely to receive the treatment. In particular, these tests are unbiased against alternatives formulated in terms of a model previously used in connection with sensitivity analyses.