Electronic Journal of Statistics

False discovery rate control for effect modification in observational studies

Bikram Karmakar, Ruth Heller, and Dylan S. Small

Full-text: Open access


In an observational study, a difference between the treatment and control group’s outcome might reflect the bias in treatment assignment rather than a true treatment effect. A sensitivity analysis determines the magnitude of this bias that would be needed to explain away as non-causal a significant treatment effect from a naive analysis that assumed no bias. Effect modification is the interaction between a treatment and a pretreatment covariate. In an observational study, there are often many possible effect modifiers and it is desirable to be able to look at the data to identify the effect modifiers that will be tested. For observational studies, we address simultaneously the problem of accounting for the multiplicity involved in choosing effect modifiers to test among many possible effect modifiers by looking at the data and conducting a proper sensitivity analysis. We develop an approach that provides finite sample false discovery rate control for a collection of adaptive hypotheses identified from the data on matched-pairs design. Along with simulation studies, an empirical study is presented on the effect of cigarette smoking on lead level in the blood using data from the U.S. National Health and Nutrition Examination Survey. Other applications of the suggested method are briefly discussed.

Article information

Electron. J. Statist., Volume 12, Number 2 (2018), 3232-3253.

Received: May 2017
First available in Project Euclid: 2 October 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Classification and regression trees effect modification sensitivity analysis simultaneous testing treatment effect

Creative Commons Attribution 4.0 International License.


Karmakar, Bikram; Heller, Ruth; Small, Dylan S. False discovery rate control for effect modification in observational studies. Electron. J. Statist. 12 (2018), no. 2, 3232--3253. doi:10.1214/18-EJS1476. https://projecteuclid.org/euclid.ejs/1538445643

Export citation


  • Aakvik, A. (2001). Bounding a Matching Estimator: The Case of a Norwegian Training Program., Oxford Bulletin of Economics and Statistics 63 115–143.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing., Journal of the Royal Statistical Society. Series B (Methodological) 57 289–300.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency., Ann. Statist. 29 1165–1188.
  • Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984)., Classification and regression trees. Wadsworth Statistics/Probability Series. Wadsworth Advanced Books and Software, Belmont, CA.
  • Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B. and Wynder, E. L. (2009). Smoking and lung cancer: recent evidence and a discussion of some questions., International Journal of Epidemiology 38 1175–1191.
  • Fisher, R. A. (1935)., The design of experiments. Oliver and Boyd, Edinburgh.
  • Freidlin, B. and Korn, E. L. (2014). Biomarker enrichment strategies: matching trial design to biomarker credentials., Nature Reviews Clinical Oncology 11 81–90.
  • Gastwirth, J. L. (1992). Methods for assessing the sensitivity of statistical comparisons used in title vii cases to omitted variables., Jurimetrics 33 19–34.
  • Gemenisa, K. and Rosemab, M. (2014). Voting Advice Applications and electoral turnout., Electoral Studies 36 281–289.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 64 499–517.
  • Glickman, M. E., Rao, S. R. and Schultz, M. R. (2014). False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies., Journal of Clinical Epidemiology 67 850–857.
  • Handorf, E. A., Bekelman, J. E., Heitjan, D. F. and Mitra, N. (2013). Evaluating costs with unmeasured confounding: A sensitivity analysis for the treatment effect., Ann. Appl. Stat. 7 2062–2080.
  • Hansen, B. B. (2007). Optmatch: Flexible, optimal matching for observational studies., R News 7 18–24.
  • Hansen, B. B., Rosenbaum, P. R. and Small, D. S. (2014). Clustered treatment assignments and sensitivity to unmeasured biases in observational studies., Journal of the American Statistical Association 109 133–144.
  • Hosman, C. A., Hansen, B. B. and Holland, P. W. (2010). The sensitivity of linear regression coefficients’ confidence limits to the omission of a confounder., Ann. Appl. Stat. 4 849–870.
  • Hsu, J. Y. and Small, D. S. (2013). Calibrating sensitivity analyses to observed covariates in observational studies., Biometrics 69 803–811.
  • Hsu, J. Y., Small, D. S. and Rosenbaum, P. R. (2013). Effect modification and design sensitivity in observational studies., Journal of the American Statistical Association 108 135–148.
  • Hsu, J. Y., Zubizarreta, J. R., Small, D. S. and Rosenbaum, P. R. (2015). Strong control of the familywise error rate in observational studies that discover effect modification by exploratory methods., Biometrika 102 767–782.
  • Huber, P. (1981)., Robust Statistics. New York: Wiley.
  • Imbens, G. W. (2003). Sensitivity to exogeneity assumptions in program evaluation., The American Economic Review 93 126–132.
  • Jogdeo, K. (1977). Association and Probability Inequalities., Ann. Statist. 5 495–504.
  • Keele, L. and Minozzi, W. (2013). How Much Is Minnesota Like Wisconsin? Assumptions and Counterfactuals in Causal Inference with Observational Data., Political Analysis 21 193–216.
  • Letham, B., Rudin, C., McCormick, T. H. and Madigan, D. (2015). Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model., Ann. Appl. Stat. 9 1350–1371.
  • Liu, W., Kuramoto, J. and Stuart, E. (2013). Sensitivity Analysis for Unobserved Confounding in Nonexperimental Prevention Research., Prevention Science 14 570–580.
  • Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance., Biometrika 63 655–660.
  • Maritz, J. S. (1979). A note on exact robust confidence intervals for location., Biometrika 66 163–166.
  • McCandless, L. C., Gustafson, P. and Levy, A. (2007). Bayesian sensitivity analysis for unmeasured confounding in observational studies., Statistics in Medicine 26 2331–2347.
  • Neyman, J. (1923). On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9. Translated and edited by D. M. Dabrowska and T. P. Speed., Statist. Sci. 5 465–472.
  • Pimentel, S. D., Yoon, F. and Keele, L. (2016). Variable ratio matching with fine balance in a study of the Peer Health Exchange., Statistics in medicine 34 4070–4082.
  • Rosenbaum, P. R. (1987). Sensitivity analysis for certain permutation inferences in matched observational studies., Biometrika 74 13–26.
  • Rosenbaum, P. R. (2002). Observational Studies. In, Observational Studies. Springer Series in Statistics Springer, New York.
  • Rosenbaum, P. R. (2004). Design sensitivity in observational studies., Biometrika 91 153–164.
  • Rosenbaum, P. R. (2007a). Confidence intervals for uncommon but dramatic responses to treatment., Biometrics 63 1164–1171.
  • Rosenbaum, P. R. (2007b). Sensitivity analysis for m-estimates, tests, and confidence intervals in matched observational studies., Biometrics 63 456–464.
  • Rosenbaum, P. R. (2010). Observational Studies. In, Design of Observational Studies. Springer Series in Statistics Springer, New York.
  • Rosenbaum, P. R. (2011). A new u-statistic with superior design sensitivity in matched observational studies., Biometrics 67 1017–1027.
  • Rosenbaum, P. R. (2014). Weighted m-statistics with superior design sensitivity in matched observational studies with multiple controls., Journal of the American Statistical Association 109 1145–1158.
  • Rosenbaum, P. R. (2015). Two r packages for sensitivity analysis in observational studies., Observational Studies 1 1–17.
  • Rosenbaum, P. R. (2017). The general structure of evidence factors in observational studies., Statistical Science 32 514–530.
  • Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies., Journal of Educational Psychology 66 688–701.
  • Sammarco, P., Kolian, S., Warby, R., Bouldin, J., Subra, W. and Porter, S. (2016). Concentrations in human blood of petroleum hydrocarbons associated with the BP/Deepwater Horizon oil spill, Gulf of Mexico., Archives of Toxicology 90 829–837.
  • Simes, R. J. (1986). An improved bonferroni procedure for multiple tests of significance., Biometrika 73 751–754.
  • Wang, L. and Krieger, A. M. (2006). Causal conclusions are most sensitive to unobserved binary covariates., Statistics in Medicine 25 2257–2271.
  • Yanagawa, T. (1984). Case-control studies: assessing the effect of a confounding factor., Biometrika 71 191–194.
  • Yu, B. and Gastwirth, J. L. (2005). Sensitivity analysis for trend tests: application to the risk of radiation exposure., Biostatistics 6 201–209.
  • Zaykin, D. V., Zhivotovsky, L. A., Westfall, P. H. and Weir, B. S. (2002). Truncated product method for combining p-values., Genetic Epidemiology 22 170–185.
  • Zubizarreta, J. R., Neuman, M., Silber, J. H. and Rosenbaum, P. R. (2012). Contrasting Evidence Within and Between Institutions That Provide Treatment in an Observational Study of Alternate Forms of Anesthesia., Journal of the American Statistical Association 107 901–915.