The Annals of Statistics

A new multiple testing method in the dependent case

Arthur Cohen, Harold B. Sackrowitz, and Minya Xu

Full-text: Open access


The most popular multiple testing procedures are stepwise procedures based on P-values for individual test statistics. Included among these are the false discovery rate (FDR) controlling procedures of Benjamini–Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289–300] and their offsprings. Even for models that entail dependent data, P-values based on marginal distributions are used. Unlike such methods, the new method takes dependency into account at all stages. Furthermore, the P-value procedures often lack an intuitive convexity property, which is needed for admissibility. Still further, the new methodology is computationally feasible. If the number of tests is large and the proportion of true alternatives is less than say 25 percent, simulations demonstrate a clear preference for the new methodology. Applications are detailed for models such as testing treatments against control (or any intraclass correlation model), testing for change points and testing means when correlation is successive.

Article information

Ann. Statist., Volume 37, Number 3 (2009), 1518-1544.

First available in Project Euclid: 10 April 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62F03: Hypothesis testing
Secondary: 62J15: Paired and multiple comparisons

Admissibility change point problem false discovery rate likelihood ratio residuals step-down procedure step-up procedure successive correlation model treatments vs. control two-sided alternatives vector risk


Cohen, Arthur; Sackrowitz, Harold B.; Xu, Minya. A new multiple testing method in the dependent case. Ann. Statist. 37 (2009), no. 3, 1518--1544. doi:10.1214/08-AOS616.

Export citation


  • Anderson, T. W. (1984). An Introduction to Multivariate Statistical Analysis, 2nd ed. Wiley, New York.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Cai, G. and Sarkar, S. K. (2006). Modified Simes critical values under positive dependence. J. Statist. Plann. Inference 136 4129–4146.
  • Carey, G. (2005). The intraclass covariance matrix. Behavior Genetics 35 667–670.
  • Casella, G. and Berger, R. L. (2002). Statistical Inference, 2nd ed. Duxbury, Pacific Grove, CA.
  • Chen, J. and Gupta, A. K. (2000). Parametric Statistical Change Point Analysis. Birkhäuser, Boston.
  • Cohen, A., Kolassa, J. and Sackrowitz, H. B. (2007). A smooth version of the step-up procedure for multiple tests of hypotheses. J. Statist. Plann. Inference 137 3352–3360.
  • Cohen, A. and Sackrowitz, H. B. (2005). Characterization of Bayes procedures for multiple endpoint problems and inadmissibility of the step-up procedure. Ann. Statist. 33 145–158.
  • Cohen, A. and Sackrowitz, H. B. (2007). More on the inadmissibility of step-up. J. Multivariate Anal. 98 481–492.
  • Cohen, A. and Sackrowitz, H. B. (2008). Multiple testing of two-sided alternatives with dependent data. Statist. Sinica. 18 1593–1602.
  • Dudoit, S., Shaffer, J. P. and Boldrick, J. C. (2003). Multiple hypothesis testing in microarray experiments. Statist. Sci. 18 71–103.
  • Dudoit, S. and van der Laan, M. (2008). Multiple Testing Procedures with Applications to Genomics. Springer. New York.
  • Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc. 96 1151–1160.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extension of the FDR procedure. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 499–518.
  • Ishwaran, H. and Rao, J. S. (2003). Detecting differentially expressed genes in microarrays using Bayesian model selection. J. Amer. Statist. Assoc. 98 438–455.
  • Hochberg, Y. and Tamhane, A. C. (1987). Multiple Comparison Procedure. Wiley, New York.
  • Hommel, G. and Bernhard, G. (1999). Bonferroni procedures for logically related hypotheses. J. Statist. Plann. Inference 82 119–128.
  • Krishnaiah, P. R. and Pathak, P. K. (1967). Tests for equality of covariance matrices under the intraclass correlation model. Ann. Math. Statist. 38 1226–1288.
  • Lehmann, E. L. (1957). A theory of some multiple decision problems. I. Ann. Math. Statist. 28 1–25.
  • Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660.
  • Matthes, T. K. and Truax, D. R. (1967). Tests of composite hypotheses for the multivariate exponential family. Ann. Math. Statist. 38 681–697.
  • Muller, P., Parmigiani, G., Robert, C. and Rousseau, J. (2004). Optimal sample size for multiple testing: The case of gene expression microarrays. J. Amer. Statist. Assoc. 99 990–1001.
  • Rao, C. R. (1945). Familial correlations for the multivariate generalizations of the intraclass correlation. Current Science 14 66–67.
  • Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
  • Scheffé, H. (1959). The Analysis of Variance. Wiley, New York.
  • Sidák, Z. (1968). On multivariate normal probabilities of rectangles. Ann. Math. Statist. 39 1425–1434.
  • Storey, J. D. and Tibshirani, R. (2003). Statistical significance for genome-wide studies. Proc. Natl. Acad. Sci. USA 100 9440–9445.
  • Vostrikova, L. J. (1981). Detecting “disorder” in multidimensional random processes. Soviet Mathematics Doklady 24 55–59.
  • Westall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing. Wiley, New York.