## The Annals of Applied Statistics

- Ann. Appl. Stat.
- Volume 12, Number 4 (2018), 2699-2733.

### Split-door criterion: Identification of causal effects through auxiliary outcomes

Amit Sharma, Jake M. Hofman, and Duncan J. Watts

#### Abstract

We present a method for estimating causal effects in time series data when fine-grained information about the outcome of interest is available. Specifically, we examine what we call the *split-door* setting, where the outcome variable can be split into two parts: one that is potentially affected by the cause being studied and another that is independent of it, with both parts sharing the same (unobserved) confounders. We show that under these conditions, the problem of identification reduces to that of testing for independence among observed variables, and propose a method that uses this approach to automatically find subsets of the data that are causally identified. We demonstrate the method by estimating the causal impact of Amazon’s recommender system on traffic to product pages, finding thousands of examples within the dataset that satisfy the split-door criterion. Unlike past studies based on natural experiments that were limited to a single product category, our method applies to a large and representative sample of products viewed on the site. In line with previous work, we find that the widely-used click-through rate (CTR) metric overestimates the causal impact of recommender systems; depending on the product category, we estimate that 50–80% of the traffic attributed to recommender systems would have happened even without any recommendations. We conclude with guidelines for using the split-door criterion as well as a discussion of other contexts where the method can be applied.

#### Article information

**Source**

Ann. Appl. Stat., Volume 12, Number 4 (2018), 2699-2733.

**Dates**

Received: January 2017

Revised: April 2018

First available in Project Euclid: 13 November 2018

**Permanent link to this document**

https://projecteuclid.org/euclid.aoas/1542078061

**Digital Object Identifier**

doi:10.1214/18-AOAS1179

**Mathematical Reviews number (MathSciNet)**

MR3875717

**Keywords**

Causal inference data mining causal graphical model natural experiment recommendation systems

#### Citation

Sharma, Amit; Hofman, Jake M.; Watts, Duncan J. Split-door criterion: Identification of causal effects through auxiliary outcomes. Ann. Appl. Stat. 12 (2018), no. 4, 2699--2733. doi:10.1214/18-AOAS1179. https://projecteuclid.org/euclid.aoas/1542078061

#### Supplemental materials

- Code for split-door criterion. We provide an R package that implements the split-door criterion, along with code samples for applying the criterion to new applications.Digital Object Identifier: doi:10.1214/18-AOAS1179SUPP