September 2023 Leveraging population outcomes to improve the generalization of experimental results: Application to the JTPA study
Melody Huang, Naoki Egami, Erin Hartman, Luke Miratrix
Author Affiliations +
Ann. Appl. Stat. 17(3): 2139-2164 (September 2023). DOI: 10.1214/22-AOAS1712


Generalizing causal estimates in randomized experiments to a broader target population is essential for guiding decisions by policymakers and practitioners in the social and biomedical sciences. While recent papers have developed various weighting estimators for the population average treatment effect (PATE), many of these methods result in large variance because the experimental sample often differs substantially from the target population and estimated sampling weights are extreme. We investigate this practical problem motivated by an evaluation study of the Job Training Partnership Act (JTPA), where we examine how well we can generalize the causal effect of job training programs beyond a specific population of economically disadvantaged adults and youths. In particular, we propose post-residualized weighting in which we use the outcome measured in the observational population data to build a flexible predictive model (e.g., with machine learning) and residualize the outcome in the experimental data before using conventional weighting methods. We show that the proposed PATE estimator is consistent under the same assumptions required for existing weighting methods, importantly without assuming the correct specification of the predictive model. We demonstrate the efficiency gains from this approach through our JTPA application: we find a reduction of between 5% and 25% in variance.

Funding Statement

Melody Huang is supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2146752. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation.


The authors would like to thank Nicole Pashley, Dustin Tingley, Tara Slough, the Miratrix CARES Lab, and the UCLA Causal Inference reading group.


Download Citation

Melody Huang. Naoki Egami. Erin Hartman. Luke Miratrix. "Leveraging population outcomes to improve the generalization of experimental results: Application to the JTPA study." Ann. Appl. Stat. 17 (3) 2139 - 2164, September 2023.


Received: 1 October 2021; Revised: 1 October 2022; Published: September 2023
First available in Project Euclid: 7 September 2023

Digital Object Identifier: 10.1214/22-AOAS1712

Keywords: Causal inference , External validity , generalizability , randomized experiments

Rights: Copyright © 2023 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.17 • No. 3 • September 2023
Back to Top