Benefits and costs of matching prior to a difference in differences analysis when parallel trends does not hold

Abstract

The consequence of a change in school leadership (e.g., principal turnover) on student achievement has important implications for education policy. The impact of such an event can be estimated via the popular difference in difference (DiD) estimator, where those schools with a turnover event are compared to a selected set of schools that did not have such an event. The strength of this comparison depends on the plausibility of the “parallel trends” assumption that the “treated group” of those schools which had leadership turnover, absent such turnover, would have changed “similarly” to those which did not. To bolster such a claim, one might generate a comparison group, via matching, that is similar to the treated group with respect to pretreatment outcomes and/or pretreatment covariates. Unfortunately, as has been previously pointed out, this intuitively appealing approach also has a cost in terms of bias. To assess the trade-offs of matching in our application, we first characterize the bias of matching prior to a DiD analysis under a linear structural model that allows for time-invariant observed and unobserved confounders with time-varying effects on the outcome. Given our framework, we verify that matching on baseline covariates generally reduces bias. We further show how additionally matching on pretreatment outcomes has both cost and benefit. First, matching on pretreatment outcomes partially balances unobserved confounders, which mitigates some bias. This reduction is proportional to the outcome’s reliability, a measure of how coupled the outcomes are with the latent covariates. Offsetting these gains, matching also injects bias into the final estimate by undermining the second difference in the DiD via a regression-to-the-mean effect. Consequently, we provide heuristic guidelines for determining to what degree the bias reduction of matching is likely to outweigh the bias cost. We illustrate our guidelines by reanalyzing a principal turnover study that used matching prior to a DiD analysis and find that matching on both the pretreatment outcomes and observed covariates makes the estimated treatment effect more credible.