A General Framework for the Analysis of Adaptive Experiments

Abstract

Adaptive experiments have design features that adapt to the accumulating data and are therefore informative about the parameter of interest. As a consequence, the overall information in an adaptive experiment is a combination of information from two sources, the realized design and the observed outcomes. This paper presents a general framework for the analysis of adaptive experiments, based on the decomposition of overall information into design information and outcome information. Likelihood inference is discussed, beginning with assumptions that guarantee insensitivity of the likelihood to the adaptive design. We then focus on the relative merits of unconditional and conditional inference. Although conditional inference is inefficient due to the nonancillary design, unconditional inference may be biased conditional on the realized design. Identifying such conditional bias in a given experiment is a motivation of the proposed framework. We show that conditional bias stems from correlation between the total information and the design information, and that this bias is most pronounced in samples where the design information is inconsistent with the outcome information. Thus, by viewing the unconditional likelihood as the aggregation of information from a design likelihood and a conditional likelihood, we can use meta-analysis principles to assess heterogeneity between the two information sources. When such heterogeneity is detected, conditional inference may be more appropriate. Interpretation from a Bayesian perspective is also discussed.