The Annals of Applied Statistics

Using Scheffé projections for multiple outcomes in an observational study of smoking and periodontal disease

Paul R. Rosenbaum

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


In an observational study of the effects caused by treatments, a sensitivity analysis asks about the magnitude of bias from unmeasured covariates that would need to be present to alter the conclusions of a naive analysis that presumes adjustments for measured covariates remove all biases. When there are two or more outcomes in an observational study, these outcomes may be unequally sensitive to unmeasured biases, and the least sensitive finding may concern a combination of several outcomes. A method of sensitivity analysis is proposed using Scheffé projections that permits the investigator to consider all linear contrasts in two or more scored outcomes while controlling the family-wise error rate. In sufficiently large samples, the method will exhibit insensitivity to bias that is greater than or equal to methods, such as the Bonferroni–Holm procedure, that focus on individual outcomes; that is, Scheffé projections have larger design sensitivities. More precisely, if the least sensitive linear combination is a single one of the several outcomes, then the design sensitivity using Scheffé projections equals that using a Bonferroni correction, but if the least sensitive combination is a nontrivial combination of two or more outcomes, then Scheffé projections have larger design sensitivities. This asymptotic property is examined in terms of finite sample power of sensitivity analyses using simulation. The method is applied to a replication with recent data of a well-known study of the effects of smoking on periodontal disease. In the example, the comparison that is least sensitive to bias from unmeasured covariates combines results for lower and upper teeth, but emphasizes lower teeth. This pattern would be difficult to anticipate prior to examining the data, but Scheffé’s method permits use of this unanticipated pattern without fear of capitalizing on chance.

Article information

Ann. Appl. Stat., Volume 10, Number 3 (2016), 1447-1471.

Received: August 2015
Revised: January 2016
First available in Project Euclid: 28 September 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Causal inference design sensitivity observational study Scheffe projection sensitivity analysis


Rosenbaum, Paul R. Using Scheffé projections for multiple outcomes in an observational study of smoking and periodontal disease. Ann. Appl. Stat. 10 (2016), no. 3, 1447--1471. doi:10.1214/16-AOAS942.

Export citation


  • Billingsley, P. (1979). Probability and Measure. Wiley, New York.
  • Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B. and Wynder, E. L. (1959). Smoking and lung cancer: Recent evidence and a discussion of some questions. J. Natl. Cancer Inst. 22 173–203.
  • Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd, Edinburgh.
  • Fogarty, C. B. and Small, D. S. (2016). Sensitivity analysis for multiple comparisons in matched observational studies through quadratically constrained linear programming. J. Amer. Statist. Assoc. To appear, DOI:10.1080/01621459.2015.1120675.
  • Gastwirth, J. L., Krieger, A. M. and Rosenbaum, P. R. (2000). Asymptotic separability in sensitivity analysis. J. R. Stat. Soc. Ser. B. Stat. Methodol. 62 545–555.
  • Genz, A. and Bretz, F. (2009). Computation of Multivariate Normal and $t$ Probabilities. Lecture Notes in Statistics 195. Springer, Dordrecht.
  • Hansen, B. B. (2007). Optmatch (R package optmatch). R News 7 18–24.
  • Heller, R., Rosenbaum, P. R. and Small, D. S. (2009). Split samples and design sensitivity in observational studies. J. Amer. Statist. Assoc. 104 1090–1101.
  • Hodges, J. L. Jr. and Lehmann, E. L. (1962). Rank methods for combination of independent experiments in analysis of variance. Ann. Math. Stat. 33 482–497.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 65–70.
  • Hosman, C. A., Hansen, B. B. and Holland, P. W. (2010). The sensitivity of linear regression coefficients’ confidence limits to the omission of a confounder. Ann. Appl. Stat. 4 849–870.
  • Hsu, J. Y. and Small, D. S. (2013). Calibrating sensitivity analyses to observed covariates in observational studies. Biometrics 69 803–811.
  • Huber, P. J. (1981). Robust Statistics. Wiley, New York.
  • Karlin, S. (1992). Mathematical Methods and Theory in Games, Programming, and Economics. Dover Publications, New York. Vol. I: Matrix games, programming, and mathematical economics, Vol. II: The theory of infinite games, Reprint of the 1959 original.
  • Lehmacher, W., Wassmer, G. and Reitmeir, P. (1991). Procedures for two-sample comparisons with multiple endpoints controlling the experimentwise error rate. Biometrics 47 511–521.
  • Lehmann, E. L. (1975). Nonparametrics. Holden Day, San Francisco.
  • Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses, 3rd ed. Springer, New York.
  • Liu, W., Kuramoto, S. J. and Stuart, E. A. (2013). An introduction to sensitivity analysis for unobserved confounding in nonexperimental prevention research. Prev. Sci. 14 570–580.
  • Manski, C. F. (1990). Nonparametric bounds on treatment effects. Am. Econ. Rev. 80 319–323.
  • Manski, C. F. and Nagin, D. S. (1990). Bounding disagreements about treatment effects: A case study of sentencing and recidivism. Sociol. Method. 28 99–137.
  • Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660.
  • Maritz, J. S. (1979). A note on exact robust confidence intervals for location. Biometrika 66 163–166.
  • McCandless, L. C., Gustafson, P. and Levy, A. (2007). Bayesian sensitivity analysis for unmeasured confounding in observational studies. Stat. Med. 26 2331–2347.
  • McKillip, J. (1992). Research without control groups: A control construct design. In Methodological Issues in Applied Social Psychology (F. B. Bryant et al., eds.) 159–175. Plenum Press, New York.
  • Neyman, J. (1923, 1990). On the application of probability theory to agricultural experiments. Essay on principles. Section 9 [Originally published in Polish in Ann. Agric. Sci. 10 (1923), 1–51]. Reprinted in 1990 in Statist. Sci. 5 463–464.
  • Pitman, E. J. G. (1937). Significance tests that may be applied to samples from any population, I. Supp. J. Roy. Satist. Soc. 4 119–130.
  • Rao, C. R. (1973). Linear Statistical Inference and Its Applications, 2nd ed. Wiley, New York.
  • Rosenbaum, P. R. (1995). Quantiles in nonrandom samples and observational studies. J. Amer. Statist. Assoc. 90 1424–1431.
  • Rosenbaum, P. R. (2004). Design sensitivity in observational studies. Biometrika 91 153–164.
  • Rosenbaum, P. R. (2007). Sensitivity analysis for $m$-estimates, tests, and confidence intervals in matched observational studies. Biometrics 63 456–464.
  • Rosenbaum, P. R. (2010). Design of Observational Studies. Springer, New York.
  • Rosenbaum, P. R. (2013). Impact of multiple matched controls on design sensitivity in observational studies. Biometrics 69 118–127.
  • Rosenbaum, P. R. (2014). Weighted $M$-statistics with superior design sensitivity in matched observational studies with multiple controls. J. Amer. Statist. Assoc. 109 1145–1158.
  • Rosenbaum, P. R. (2015). Bahadur efficiency of sensitivity analyses in observational studies. J. Amer. Statist. Assoc. 110 205–217.
  • Rosenbaum, P. R. and Rubin, D. B. (1983). Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J. Roy. Satist. Soc. B 45 212–218.
  • Rosenbaum, P. R. and Silber, J. H. (2009a). Amplification of sensitivity analysis in matched observational studies. J. Amer. Statist. Assoc. 104 1398–1405.
  • Rosenbaum, P. R. and Silber, J. H. (2009b). Sensitivity analysis for equivalence and difference in an observational study of neonatal intensive care units. J. Amer. Statist. Assoc. 104 501–511.
  • Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. J. Ed. Psych. 66 688–701.
  • Scheffé, H. (1953). A method for judging all contrasts in the analysis of variance. Biometrika 40 87–104.
  • Shepherd, B. E., Gilbert, P. B., Jemiai, Y. and Rotnitzky, A. (2006). Sensitivity analyses comparing outcomes only existing in a subset selected post-randomization, conditional on covariates, with application to HIV vaccine trials. Biometrics 62 332–342.
  • Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statist. Sci. 25 1–21.
  • Tomar, S. L. and Asma, S. (2000). Smoking-attributable periodontitis in the United States: Findings from NHANES III. J. Periodont. 71 743–751.
  • van Elteren, Ph. (1960). On the combination of independent two sample test of Wilcoxon. Bull. Inst. Internat. Statist. 37 351–361.
  • Wei, L., Barker, L. and Eke, P. (2013). Array applications in determining periodontal disease measurement. SouthEast SAS User’s Group. (SESUG2013) Paper CC-15,
  • Weiss, N. (2002). Can the ‘specificity’ of an association be rehabilitated as a basis for supporting a causal hypothesis? Epidemiology 13 6–8.
  • Welch, B. L. (1937). On the z-test in randomized blocks and latin squares. Biometrika 29 21–52.
  • Yu, B. B. and Gastwirth, J. L. (2005). Sensitivity analysis for trend tests: Application to the risk of radiation exposure. Biostatistics 6 201–209.
  • Zubizarreta, J. R. (2012). Using mixed integer programming for matching in an observational study of kidney failure after surgery. J. Amer. Statist. Assoc. 107 1360–1371.