The Annals of Statistics

On Permutation Tests for Hidden Biases in Observational Studies: An Application of Holley's Inequality to the Savage Lattice

Paul R. Rosenbaum

Full-text: Open access


Randomized experiments and observational studies both attempt to estimate the effects produced by a treatment, but in observational studies, subjects are not randomly assigned to treatment or control. A theory of observational studies would closely resemble the theory for randomized experiments in all but one critical respect: In observational studies, the distribution of treatment assignments is not known. The problems that are special to observational studies revolve around our uncertainty about how treatments were assigned. In this connection, tools are needed for describing distributions of treatment assignments that do not assign equal probabilities to all assignments. Two such tools are a lattice of treatment assignments first studied by Savage and an inequality due to Holley for probability distributions on a lattice. Using these tools, it is shown that certain permutation tests are unbiased as tests of the null hypothesis that the distribution of treatment assignments resembles a randomization distribution against the alternative hypothesis that subjects with higher responses are more likely to receive the treatment. In particular, these tests are unbiased against alternatives formulated in terms of a model previously used in connection with sensitivity analyses.

Article information

Ann. Statist., Volume 17, Number 2 (1989), 643-653.

First available in Project Euclid: 12 April 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier


Primary: 62G10: Hypothesis testing
Secondary: 60C05: Combinatorial probability 06D99: None of the above, but in this section

Observational studies permutation test lattice theory Holley's inequality decreasing in transposition decreasing reflection function rank sum test signed rank test Mantel-Haenszel test McNemar-Cox test unbiased test


Rosenbaum, Paul R. On Permutation Tests for Hidden Biases in Observational Studies: An Application of Holley's Inequality to the Savage Lattice. Ann. Statist. 17 (1989), no. 2, 643--653. doi:10.1214/aos/1176347131.

Export citation