Statistical Science

Nonparametric Bounds and Sensitivity Analysis of Treatment Effects

Amy Richardson, Michael G. Hudgens, Peter B. Gilbert, and Jason P. Fine

Full-text: Open access


This paper considers conducting inference about the effect of a treatment (or exposure) on an outcome of interest. In the ideal setting where treatment is assigned randomly, under certain assumptions the treatment effect is identifiable from the observable data and inference is straightforward. However, in other settings such as observational studies or randomized trials with noncompliance, the treatment effect is no longer identifiable without relying on untestable assumptions. Nonetheless, the observable data often do provide some information about the effect of treatment, that is, the parameter of interest is partially identifiable. Two approaches are often employed in this setting: (i) bounds are derived for the treatment effect under minimal assumptions, or (ii) additional untestable assumptions are invoked that render the treatment effect identifiable and then sensitivity analysis is conducted to assess how inference about the treatment effect changes as the untestable assumptions are varied. Approaches (i) and (ii) are considered in various settings, including assessing principal strata effects, direct and indirect effects and effects of time-varying exposures. Methods for drawing formal inference about partially identified parameters are also discussed.

Article information

Statist. Sci., Volume 29, Number 4 (2014), 596-618.

First available in Project Euclid: 15 January 2015

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Causal inference nonparametric bounds partially identifiable models sensitivity analysis


Richardson, Amy; Hudgens, Michael G.; Gilbert, Peter B.; Fine, Jason P. Nonparametric Bounds and Sensitivity Analysis of Treatment Effects. Statist. Sci. 29 (2014), no. 4, 596--618. doi:10.1214/14-STS499.

Export citation


  • Angrist, J. D., Imbens, G. W. and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. J. Amer. Statist. Assoc. 91 444–455.
  • Armstrong, C. S., Guay, W. R. and Weber, J. P. (2010). The role of information and financial reporting in corporate governance and debt contracting. J. Accounting and Economics 50 179–234.
  • Balke, A. and Pearl, J. (1993). Nonparametric bounds on causal effects from partial compliance data. Technical report, Univ. California, Los Angeles.
  • Balke, A. and Pearl, J. (1997). Bounds on treatment effects from studies with imperfect compliance. J. Amer. Statist. Assoc. 92 1171–1177.
  • Brumback, B. A., Hernán, M. A., Haneuse, S. J. P. A. and Robins, J. M. (2004). Sensitivity analyses for unmeasured confounding assuming a marginal structural model for repeated measures. Stat. Med. 23 749–767.
  • Bugni, F. A. (2010). Bootstrap inference in partially identified models defined by moment inequalities: Coverage of the identified set. Econometrica 78 735–753.
  • Cai, Z., Kuroki, M., Pearl, J. and Tian, J. (2008). Bounds on direct effects in the presence of confounded intermediate variables. Biometrics 64 695–701.
  • Chernozhukov, V., Hong, H. and Tamer, E. (2007). Estimation and confidence regions for parameter sets in econometric models. Econometrica 75 1243–1284.
  • Chiburis, R. C. (2010). Semiparametric bounds on treatment effects. J. Econometrics 159 267–275.
  • Chickering, D. M. and Pearl, J. (1996). A clinician’s tool for analyzing non-compliance. In AAAI-96 Proceedings 1269–1276. AAAI Press, Menlo Park, CA.
  • Cole, S. R., Hernán, M. A., Margolick, J. B., Cohen, M. H. and Robins, J. M. (2005). Marginal structural models for estimating the effect of highly active antiretroviral therapy initiation on CD4 cell count. Amer. J. Epidemiol. 162 471–478.
  • Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B. and Wynder, E. L. (1959). Smoking and lung cancer: Recent evidence and a discussion of some questions. J. Natl. Cancer Inst. 22 173–203.
  • Dawid, A. P. (2003). Causal inference using influence diagrams: The problem of partial compliance. In Highly Structured Stochastic Systems (P. J. Green, N. L. Hjort and S. Richardson, eds.) 45–81. Oxford Univ. Press, Oxford.
  • Frangakis, C. E. and Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics 58 21–29.
  • Gilbert, P. B., Bosch, R. J. and Hudgens, M. G. (2003). Sensitivity analysis for the assessment of causal vaccine effects on viral load in HIV vaccine trials. Biometrics 59 531–541.
  • Grilli, L. and Mealli, F. (2008). Nonparametric bounds on the causal effect of university studies on job opportunities using principal stratification. J. Educ. Behav. Stat. 33 111–130.
  • Gustafson, P. (2010). Bayesian inference for partially identified models. Int. J. Biostat. 6 1–18.
  • Gustafson, P., McCandless, L. C., Levy, A. R. and Richardson, S. (2010). Simplified Bayesian sensitivity analysis for mismeasured and unobserved confounders. Biometrics 66 1129–1137.
  • Hafeman, D. M. (2011). Confounding of indirect effects: A sensitivity analysis exploring the range of bias due to a cause common to both the mediator and the outcome. Amer. J. Epidemiol. 174 710–717.
  • Heckman, J. J. (2001). Micro data, heterogeneity, and the evaluation of public policy: Nobel lecture. J. Political Economy 109 673–748.
  • Heller, R., Rosenbaum, P. R. and Small, D. S. (2009). Split samples and design sensitivity in observational studies. J. Amer. Statist. Assoc. 104 1090–1101.
  • Hernán, M. A. and Robins, J. M. (1999). Assessing the sensitivity of regression results to unmeasured confounders in observational studies [letter]. Biometrics 55 1316–1317.
  • Hernán, M. A. and Robins, J. M. (2006). Instruments for causal inference: An epidemiologist’s dream? Epidemiology 17 360–372.
  • Horowitz, J. L. and Manski, C. F. (2000). Nonparametric analysis of randomized experiments with missing covariate and outcome data. J. Amer. Statist. Assoc. 95 77–84.
  • Horowitz, J. L. and Manski, C. F. (2006). Identification and estimation of statistical functionals using incomplete data. J. Econometrics 132 445–459.
  • Hu, J. C., Williams, S. B., O’Malley, A. J., Smith, M. R., Nguyen, P. L. and Keating, N. L. (2012). Androgen-deprivation therapy for nonmetastatic prostate cancer is associated with an increased risk of peripheral arterial disease and venous thromboembolism. Eur. Urol. 61 1119–1128.
  • Hudgens, M. G. and Halloran, M. E. (2006). Causal vaccine effects on binary postinfection outcomes. J. Amer. Statist. Assoc. 101 51–64.
  • Hudgens, M. G., Hoering, A. and Self, S. G. (2003). On the analysis of viral load endpoints in HIV vaccine trials. Stat. Med. 22 2281–2298.
  • Imai, K., Keele, L. and Yamamoto, T. (2010). Identification, inference and sensitivity analysis for causal mediation effects. Statist. Sci. 25 51–71.
  • Imbens, G. W. and Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica 62 467–475.
  • Imbens, G. W. and Manski, C. F. (2004). Confidence intervals for partially identified parameters. Econometrica 72 1845–1857.
  • Joffe, M. (2011). Principal stratification and attribution prohibition: Good ideas taken too far. Int. J. Biostat. 7 1–22.
  • Kaufman, S., Kaufman, J. S. and MacLehose, R. F. (2009). Analytic bounds on causal risk differences in directed acyclic graphs involving three observed binary variables. J. Statist. Plann. Inference 139 3473–3487.
  • Lee, M.-j. (2005). Micro-econometrics for Policy, Program, and Treatment Effects. Oxford Univ. Press, Oxford.
  • Lee, D. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Rev. Econom. Stud. 76 1071–1102.
  • Lin, D. Y., Psaty, B. M. and Kronmal, R. A. (1998). Assessing the sensitivity of regression results to unmeasured confounders in observational studies. Biometrics 54 948–963.
  • Long, D. M. and Hudgens, M. G. (2013). Sharpening bounds on principal effects with covariates. Biometrics 69 812–819.
  • Manski, C. F. (1990). Nonparametric bounds on treatment effects. Am. Econ. Rev. 80 319–323.
  • Manski, C. F. (1997). Monotone treatment response. Econometrica 65 1311–1334.
  • Manski, C. F. (2013). Identification of treatment response with social interactions. Econom. J. 16 S1–S23.
  • Manski, C. F. and Pepper, J. V. (2000). Monotone instrumental variables: With an application to the returns to schooling. Econometrica 68 997–1010.
  • McCandless, L. C., Gustafson, P. and Levy, A. (2007). Bayesian sensitivity analysis for unmeasured confounding in observational studies. Stat. Med. 26 2331–2347.
  • Moon, H. R. and Schorfheide, F. (2012). Bayesian and frequentist inference in partially identified models. Econometrica 80 755–782.
  • Morgan, S. L. and Winship, C. (2007). Counterfactuals and Causal Inference. Cambridge Univ. Press, New York.
  • Pearl, J. (2001). Direct and indirect effects. In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence. UAI’01 411–420. Morgan Kaufmann, San Francisco, CA.
  • Pearl, J. (2009). Causality: Models, Reasoning, and Inference, 2nd ed. Cambridge Univ. Press, Cambridge.
  • Pearl, J. (2010). On the consistency rule in causal inference: Axiom, definition, assumption, or theorem? Epidemiology 21 872–875.
  • Pearl, J. (2011). Principal stratification: A goal or a tool? Int. J. Biostat. 7 1–14.
  • Préziosi, M.-P. and Halloran, M. E. (2003). Effects of pertussis vaccination on disease: Vaccine efficacy in reducing clinical severity. Clin. Infect. Dis. 37 772–779.
  • Rerks-Ngarm, S., Pitisuttithum, P., Nitayaphan, S., Kaewkungwal, J., Chiu, J., Paris, R., Premsri, N., Namwat, C., de Souza, M., Adams, E., Benenson, M., Gurunathan, S., Tartaglia, J., McNeil, J. G., Francis, D. P., Stablein, D., Birx, D. L., Chunsuttiwat, S., Khamboonruang, C., Thongcharoen, P., Robb, M. L., Michael, N. L., Kunasol, P. and Kim, J. H. (2009). Vaccination with ALVAC and AIDSVAX to prevent HIV-1 infection in Thailand. N. Engl. J. Med. 361 2209–2220.
  • Richardson, T. S., Evans, R. J. and Robins, J. M. (2011). Transparent parametrizations of models for potential outcomes. In Bayesian Statistics 9 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 569–610. Oxford Univ. Press, Oxford.
  • Robins, J. M. (1989). The analysis of randomized and non-randomized AIDS treatment trials using a new approach to causal inference in longitudinal studies. Health Service Research Methodology: A Focus on AIDS 113–159.
  • Robins, J. M. (1994). Correcting for non-compliance in randomized trials using structural nested mean models. Comm. Statist. Theory Methods 23 2379–2412.
  • Robins, J. M. (1997). Non-response models for the analysis of non-monotone non-ignorable missing data. Stat. Med. 16 21–37.
  • Robins, J. M. (1999). Association, causation, and marginal structural models. Statistics and causation. Synthese 121 151–179.
  • Robins, J. M. (2002). Comment on “Covariance adjustment in randomized experiments and observational studies”. Statist. Sci. 17 309–321.
  • Robins, J. M. (2003). Semantics of causal DAG models and the identification of direct and indirect effects. Oxford Statist. Sci. Ser. 70–82.
  • Robins, J. M. and Greenland, S. (1992). Identifiability and exchangeability for direct and indirect effects. Epidemiology 3 143–155.
  • Robins, J. M. and Greenland, S. (1996). Comment on “Identification of causal effects using instrumental variables” by Angrist, Imbens and Rubin. J. Amer. Statist. Assoc. 91 456–458.
  • Robins, J. M. and Richardson, T. S. (2010). Alternative graphical causal models and the identification of direct effects. In Causality and Psychopathology: Finding the Determinants of Disorders and Their Cures (P. Shrout, ed.). Oxford Univ. Press, Oxford.
  • Robins, J. M., Rotnitzky, A. and Scharfstein, D. O. (2000). Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In Statistical Models in Epidemiology, the Environment, and Clinical Trials (Minneapolis, MN, 1997). IMA Vol. Math. Appl. 116 1–94. Springer, New York.
  • Romano, J. P. and Shaikh, A. M. (2008). Inference for identifiable parameters in partially identified econometric models. J. Statist. Plann. Inference 138 2786–2807.
  • Rosenbaum, P. R. (1999). Choice as an alternative to control in observational studies. Statist. Sci. 14 259–278.
  • Rosenbaum, P. R. (2002). Observational Studies, 2nd ed. Springer, New York.
  • Rosenbaum, P. R. (2010a). Design sensitivity and efficiency in observational studies. J. Amer. Statist. Assoc. 105 692–702.
  • Rosenbaum, P. R. (2010b). Evidence factors in observational studies. Biometrika 97 333–345.
  • Rosenbaum, P. R. (2011). Some approximate evidence factors in observational studies. J. Amer. Statist. Assoc. 106 285–295.
  • Rosenbaum, P. R. and Rubin, D. B. (1983). Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J. R. Stat. Soc. Ser. B Stat. Methodol. 45 212–218.
  • Rotnitzky, A. and Jemiai, Y. (2003). Sharp bounds and sensitivity analysis for treatment effects in the presence of censoring by death. In Harvard Schering-Plough Workshop on Development and Approval of Oncology Drug Products: Impact of Statistics.
  • Rubin, D. B. (1980). Discussion of “Randomization analysis of experimental data in the Fisher randomization test,” by D. Basu. J. Amer. Statist. Assoc. 75 591–593.
  • Rubin, D. B. (2000). Comment on “Causal inference without counterfactuals”. J. Amer. Statist. Assoc. 95 435–437.
  • Scharfstein, D. O., Rotnitzky, A. and Robins, J. M. (1999). Adjusting for nonignorable drop-out using semiparametric nonresponse models. J. Amer. Statist. Assoc. 94 1096–1146.
  • Schlesselman, J. J. (1978). Assessing effects of confounding variables. Amer. J. Epidemiol. 108 3–8.
  • Shepherd, B. E., Gilbert, P. B. and Mehrotra, D. V. (2007). Eliciting a counterfactual sensitivity parameter. Amer. Statist. 61 56–63.
  • Sianesi, B. (2004). An evaluation of the Swedish system of active labor market programs in the 1990s. The Review of Economics and Statistics 86 133–155.
  • Sjölander, A. (2009). Bounds on natural direct effects in the presence of confounded intermediate variables. Stat. Med. 28 558–571.
  • Stoye, J. (2009). More on confidence intervals for partially identified parameters. Econometrica 77 1299–1315.
  • Tchetgen Tchetgen, E. J., Glymour, M. M., Weuve, J. and Robins, J. (2012a). A cautionary note on specification of the correlation structure in inverse-probability-weighted estimation for repeated measures. Technical Report 140, Harvard Univ. Biostatistics Working Paper Series.
  • Tchetgen Tchetgen, E. J. T., Glymour, M. M., Weuve, J. and Robins, J. (2012b). Specifying the correlation structure in inverse-probability-weighting estimation for repeated measures. Epidemiology 23 644–646.
  • Todem, D., Fine, J. and Peng, L. (2010). A global sensitivity test for evaluating statistical hypotheses with nonidentifiable models. Biometrics 66 558–566.
  • VanderWeele, T. J. (2008). Sensitivity analysis: Distributional assumptions and confounding assumptions. Biometrics 64 645–649.
  • VanderWeele, T. J. (2010). Bias formulas for sensitivity analysis for direct and indirect effects. Epidemiology 21 540–551.
  • VanderWeele, T. J. (2011a). Controlled direct and mediated effects: Definition, identification and bounds. Scand. J. Stat. 38 551–563.
  • VanderWeele, T. J. (2011b). Sensitivity analysis for contagion effects in social networks. Sociol. Methods Res. 40 240–255.
  • VanderWeele, T. J. and Arah, O. A. (2011). Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders. Epidemiology 22 42–52.
  • VanderWeele, T. J. and Hernández-Diaz, S. (2011). Is there a direct effect of pre-eclampsia on cerebral palsy not through preterm birth? Paediatric and Perinatal Epidemiology 25 111–115.
  • VanderWeele, T. J., Mukherjee, B. and Chen, J. (2012). Sensitivity analysis for interactions under unmeasured confounding. Stat. Med. 31 2552–2564.
  • Vansteelandt, S. and Goetghebeur, E. (2001). Analyzing the sensitivity of generalized linear models to incomplete outcomes via the IDE algorithm. J. Comput. Graph. Statist. 10 656–672.
  • Vansteelandt, S., Goetghebeur, E., Kenward, M. G. and Molenberghs, G. (2006). Ignorance and uncertainty regions as inferential tools in a sensitivity analysis. Statist. Sinica 16 953–979.
  • van der Laan, M. J. and Robins, J. M. (2003). Unified Methods for Censored Longitudinal Data and Causality. Springer, New York.
  • Ver Steeg, G. and Galstyan, A. (2010). Ruling out latent homophily in social networks. In NIPS Workshop on Social Computing.
  • Zhang, J. L. and Rubin, D. B. (2003). Estimation of causal effects via principal stratification when some outcomes are truncated by “death.” J. Educational and Behavioral Statistics 28 353–368.