August 2023 Experimental Design in Marketplaces
Patrick Bajari, Brian Burdick, Guido W. Imbens, Lorenzo Masoero, James McQueen, Thomas S. Richardson, Ido M. Rosen
Author Affiliations +
Statist. Sci. 38(3): 458-476 (August 2023). DOI: 10.1214/23-STS883


Classical Randomized Controlled Trials (RCTs), or A/B tests, are designed to draw causal inferences about a population of units, for example, individuals, plots of land or visits to a website. A key assumption underlying a standard RCT is the absence of interactions between units, or the stable unit treatment value assumption (Ann. Statist. 6 (1978) 34–58). Modern experimentation, however, is often conducted in settings characterized by complex interactions between units. Such interactions can invalidate the standard estimators and make classical experimental designs ineffective. Although the presence of interference forces us to make untestable assumptions on the nature of the interactions even under randomization, sophisticated experimental designs can ameliorate the dependence on such assumptions. In this manuscript, we review the recent and rapidly growing literature on novel experimental designs for these settings. One key feature common to many of these designs is the presence of multiple layers of randomization within the same experiment. We discuss a novel experimental design, called Multiple Randomization Designs or MRDs, that provides a general framework for such experiments. Through these complex designs, we can study questions about causal effects in the presence of interference that cannot be answered by classical RCTs.


We are grateful for discussions with Susan Athey, John Geweke, Matt Taddy, and Johan Ugander and for comments by participants in the following conferences: CODE@MIT, NABE (2019), ASSA (2020) and ICSDS (2022). This research was carried out as part of Imbens’ and Richardson’s consulting relationship with Amazon. Brian Burdick contributed while working for Amazon.


Download Citation

Patrick Bajari. Brian Burdick. Guido W. Imbens. Lorenzo Masoero. James McQueen. Thomas S. Richardson. Ido M. Rosen. "Experimental Design in Marketplaces." Statist. Sci. 38 (3) 458 - 476, August 2023.


Published: August 2023
First available in Project Euclid: 20 August 2023

MathSciNet: MR4630378
Digital Object Identifier: 10.1214/23-STS883

Keywords: Causal inference , Experimental design , multiple randomization designs , online experimentation , two-sided marketplaces

Rights: Copyright © 2023 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.38 • No. 3 • August 2023
Back to Top