The Annals of Applied Statistics

Identifying and estimating principal causal effects in a multi-site trial of Early College High Schools

Lo-Hua Yuan, Avi Feller, and Luke W. Miratrix

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

Randomized trials are often conducted with separate randomizations across multiple sites such as schools, voting districts, or hospitals. These sites can differ in important ways, including the site’s implementation quality, local conditions, and the composition of individuals. An important question in practice is whether—and under what assumptions—researchers can leverage this cross-site variation to learn more about the intervention. We address these questions in the principal stratification framework, which describes causal effects for subgroups defined by post-treatment quantities. We show that researchers can estimate certain principal causal effects via the multi-site design if they are willing to impose the strong assumption that the site-specific effects are independent of the site-specific distribution of stratum membership. We motivate this approach with a multi-site trial of the Early College High School Initiative, a unique secondary education program with the goal of increasing high school graduation rates and college enrollment. Our analyses corroborate previous studies suggesting that the initiative had positive effects for students who would have otherwise attended a low-quality high school, although power is limited.

Article information

Source
Ann. Appl. Stat., Volume 13, Number 3 (2019), 1348-1369.

Dates
Received: March 2018
Revised: December 2018
First available in Project Euclid: 17 October 2019

Permanent link to this document
https://projecteuclid.org/euclid.aoas/1571277756

Digital Object Identifier
doi:10.1214/18-AOAS1235

Mathematical Reviews number (MathSciNet)
MR4019142

Keywords
Principal causal effects principal stratification covariate restrictions multi-site randomized trials noncompliance Early College High School

Citation

Yuan, Lo-Hua; Feller, Avi; Miratrix, Luke W. Identifying and estimating principal causal effects in a multi-site trial of Early College High Schools. Ann. Appl. Stat. 13 (2019), no. 3, 1348--1369. doi:10.1214/18-AOAS1235. https://projecteuclid.org/euclid.aoas/1571277756


Export citation

References

  • Allensworth, E. (2005). Graduation and Dropout Trends in Chicago: A Look at Cohorts of Students from 1991 through 2004. Report Highlights. Consortium on Chicago School Research. Available at https://files.eric.ed.gov/fulltext/ED486035.pdf.
  • Angrist, J. D., Imbens, G. W. and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. J. Amer. Statist. Assoc. 91 444–455.
  • Bloom, H. S., Raudenbush, S. W., Weiss, M. J. and Porter, K. (2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness 10 817–842.
  • Bowden, J., Davey Smith, G. and Burgess, S. (2015). Mendelian randomization with invalid instruments: Effect estimation and bias detection through Egger regression. Int. J. Epidemiol. 44 512–525.
  • Carnegie, N. B., Harada, M. and Hill, J. L. (2016). Assessing sensitivity to unmeasured confounding using a simulated potential confounder. Journal of Research on Educational Effectiveness 9 395–420.
  • Ding, P., Geng, Z., Yan, W. and Zhou, X.-H. (2011). Identifiability and estimation of causal effects by principal stratification with outcomes truncated by death. J. Amer. Statist. Assoc. 106 1578–1591.
  • Edmunds, J. A., Bernstein, L., Glennie, E., Willse, J., Arshavsky, N., Unlu, F., Bartz, D., Silberman, T., Scales, W. D. et al. (2010). Preparing students for college: The implementation and impact of the Early College High School model. Peabody Journal of Education 85 348–364.
  • Edmunds, J. A., Bernstein, L., Unlu, F., Glennie, E., Willse, J., Smith, A. and Arshavsky, N. (2012). Expanding the start of the College Pipeline: Ninth-grade findings from an experimental study of the impact of the Early College High School model. Journal of Research on Educational Effectiveness 5 136–159.
  • Edmunds, J. A., Unlu, F., Glennie, E., Bernstein, L., Fesler, L., Furey, J. and Arshavsky, N. (2017). Smoothing the transition to postsecondary education: The impact of the Early College model. Journal of Research on Educational Effectiveness 10 297–325.
  • Egger, M., Smith, G. D., Schneider, M. and Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ 315 629–634.
  • Feller, A., Grindal, T., Miratrix, L. and Page, L. C. (2016a). Compared to what? Variation in the impacts of early childhood education by alternative care type. Ann. Appl. Stat. 10 1245–1285.
  • Feller, A., Greif, E., Miratrix, L. and Pillai, N. (2016b). Principal stratification in the Twilight Zone: Weakly separated components in finite mixture models. Available at arXiv:1602.06595.
  • Frangakis, C. E. and Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics 58 21–29.
  • Gelman, A., Park, D. K., Ansolabehere, S., Price, P. N. and Minnite, L. C. (2001). Models, assumptions and model checking in ecological regressions. J. Roy. Statist. Soc. Ser. A 164 101–118.
  • Hull, P. (2018). IsoLATEing: Identifying Counterfactual-Specific Treatment Effects with Cross-Stratum Comparisons. Working paper.
  • Jiang, Z., Ding, P. and Geng, Z. (2016). Principal causal effect identification and surrogate end point evaluation by multiple trials. J. R. Stat. Soc. Ser. B. Stat. Methodol. 78 829–848.
  • Jo, B. (2002). Estimation of intervention effects with noncompliance: Alternative model specifications. J. Educ. Behav. Stat. 27 385–409.
  • Kang, H., Zhang, A., Cai, T. T. and Small, D. S. (2016). Instrumental variables estimation with some invalid instruments and its application to Mendelian randomization. J. Amer. Statist. Assoc. 111 132–144.
  • Kline, P. and Walters, C. R. (2016). Evaluating public programs with close substitutes: The case of head start. Q. J. Econ. 131 1795–1848.
  • Kolesár, M., Chetty, R., Friedman, J., Glaeser, E. and Imbens, G. W. (2015). Identification and inference with many invalid instruments. J. Bus. Econom. Statist. 33 474–484.
  • MacKinnon, J. G. and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. J. Econometrics 29 305–325.
  • Mealli, F., Pacini, B. and Stanghellini, E. (2016). Identification of principal causal effects using additional outcomes in concentration graphs. J. Educ. Behav. Stat. 41 463–480.
  • Miratrix, L., Furey, J., Feller, A., Grindal, T. and Page, L. C. (2018). Bounding, an accessible method for estimating principal causal effects, examined and explained. Journal of Research on Educational Effectiveness 11 133–162.
  • Page, L. C., Feller, A., Grindal, T., Miratrix, L. and Somers, M.-A. (2015). Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation 36 514–531.
  • Peck, L. R. (2003). Subgroup analysis in social experiments: Measuring program impacts based on post-treatment choice. American Journal of Evaluation 24 157–187.
  • Raudenbush, S. W. and Bloom, H. S. (2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation 36 475–499.
  • Raudenbush, S. W., Reardon, S. F. and Nomi, T. (2012). Statistical analysis for multisite trials using instrumental variables with random coefficients. Journal of Research on Educational Effectiveness 5 303–332.
  • Raudenbush, S. and Schwartz, D. (2017). Identification and estimation in multisite randomized trials with heterogeneous treatment effects. Submitted.
  • Reardon, S. F. and Raudenbush, S. W. (2013). Under what assumptions do site-by-treatment instruments identify average causal effects? Sociol. Methods Res. 42 143–163.
  • Reardon, S. F., Unlu, F., Zhu, P. and Bloom, H. S. (2014). Bias and bias correction in multisite instrumental variables analysis of heterogeneous mediator effects. J. Educ. Behav. Stat. 39 53–86.
  • Rubin, D. B. (1980). Comment on “Randomization analysis of experimental data: The Fisher randomization test”. J. Amer. Statist. Assoc. 75 591–593.
  • Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley, New York.
  • Wang, L., Zhou, X.-H. and Richardson, T. S. (2017). Identification and estimation of causal effects with outcomes truncated by death. Biometrika 104 597–612.
  • Yuan, L.-H. (2018). Regressions for estimating main and principal causal effects in multi-site randomized trials and small sample designs. Doctoral dissertation. Available at https://dash.harvard.edu/handle/1/40050155.
  • Yuan, L.-H., Feller, A. and Miratrix, L. (2019). Supplement to “Identifying and Estimating Principal Causal Effects in a Multi-site Trial of Early College High Schools.” DOI:10.1214/18-AOAS1235SUPP.

Supplemental materials

  • Supplement to: “Identifying and estimating principal causal effects in a multi-site trial of Early College High Schools”. The Supplementary Material includes additional analyses, proofs and other technical materials.