The Annals of Statistics

The control of the false discovery rate in multiple testing under dependency

Yoav Benjamini and Daniel Yekutieli

Full-text: Open access

Abstract

Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate $t$. Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which a procedure with proven FDR control can be offered is greatly increased.

Article information

Source
Ann. Statist. Volume 29, Number 4 (2001), 1165-1188.

Dates
First available in Project Euclid: 14 February 2002

Permanent link to this document
http://projecteuclid.org/euclid.aos/1013699998

Digital Object Identifier
doi:10.1214/aos/1013699998

Mathematical Reviews number (MathSciNet)
MR1869245

Zentralblatt MATH identifier
01829051

Subjects
Primary: 62J15: Paired and multiple comparisons 62G30: Order statistics; empirical distribution functions 47N30: Applications in probability theory and statistics

Keywords
Multiple comparisons procedures FDR Simes’equality Hochberg’s procedure MTP2 densities positive regression dependency unidimensional latent variables discrete test statistics multiple endpoints many-to-one comparisons comparisons with control

Citation

Benjamini, Yoav; Yekutieli, Daniel. The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 (2001), no. 4, 1165--1188. doi:10.1214/aos/1013699998. http://projecteuclid.org/euclid.aos/1013699998.


Export citation

References

  • Abramovich, F. and Benjamini, Y. (1996). Adaptive thresholding of wavelet coefficients. Comput. Statist. Data Anal. 22 351-361.
  • Barinaga, M. (1994). From fruit flies, rats, mice: evidence of genetic influence. Science 264 1690-1693.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289-300.
  • Benjamini, Y. and Hochberg, Y. (1997). Multiple hypotheses testing with weights. Scand. J. Statist. 24 407-418.
  • Benjamini, Y. and Hochberg, Y. (2000). The adaptive control of the false discovery rate in multiple hypotheses testing. J. Behav. Educ. Statist. 25 60-83.
  • Benjamini, Y., Hochberg, Y. and Kling, Y. (1993). False discovery rate control in pairwise comparisons. Working Paper 93-2, Dept. Statistics and O.R., Tel Aviv Univ.
  • Benjamini, Y., Hochberg, Y. and Kling, Y. (1997). False discovery rate control in multiple hypotheses testing using dependent test statistics. Research Paper 97-1, Dept. Statistics and O.R., Tel Aviv Univ.
  • Benjamini, Y. and Wei, L. (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Statist. Plann. Inference 82 163-170.
  • Chang, C. K., Rom, D. M. and Sarkar, S. K. (1996). A modified Bonferroni procedure for repeated significance testing. Technical Report 96-01, Temple Univ.
  • Eaton, M. L. (1986). Lectures on topics in probability inequalities. CWI Tract 35.
  • Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800-803.
  • Hochberg, Y. and Hommel, G. (1998). Step-up multiple testing procedures. Encyclopedia Statist. Sci. (Supp.) 2.
  • Hochberg, Y. and Rom, D. (1995). Extensions of multiple testing procedures based on Simes' test. J. Statist. Plann. Inference 48 141-152.
  • Hochberg, Y. and Tamhane, A. (1987). Multiple Comparison Procedures. Wiley, New York.
  • Holland, P. W. and Rosenbaum, P. R. (1986). Conditional association and unidimensionality in monotone latent variable models. Ann. Statist. 14 1523-1543.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist 6 65-70.
  • Hommel, G. (1988). A stage-wise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383-386.
  • Hsu, J. (1996). Multiple Comparisons Procedures. Chapman and Hall, London.
  • Karlin, S. and Rinott, Y. (1980). Classes of orderings of measures and related correlation inequalities I. Multivariate totally positive distributions. J. Multivariate Statist. 10 467-498.
  • Karlin, S. and Rinott, Y. (1981). Total positivity properties of absolute value multinormal variable with applications to confidence interval estimates and related probabilistic inequalities. Ann. Statist. 9 1035-1049.
  • Lander E. S. and Botstein D. (1989). Mapping Mendelian factors underlying quantitative traits using RFLP linkage maps. Genetics 121 185-190.
  • Lander, E. S. and Kruglyak L. (1995). Genetic dissection of complex traits: guidelines for interpreting and reporting linkage results. Nature Genetics 11 241-247.
  • Lehmann, E. L. (1966). Some concepts of dependence. Ann. Math. Statist. 37 1137-1153.
  • Needleman, H., Gunnoe, C., Leviton, A., Reed, R., Presie, H., Maher, C. and Barret, P. (1979). Deficits in psychologic and classroom performance of children with elevated dentine lead levels. New England J. Medicine 300 689-695.
  • Paterson, A. H. G., Powles, T. J., Kanis, J. A., McCloskey, E., Hanson, J. and Ashley, S. (1993). Double-blind controlled trial of oral clodronate in patients with bone metastases from breastcancer. J. Clinical Oncology 1 59-65.
  • Rosenbaum, P. R. (1984). Testing the conditional independence and monotonicity assumptions of item response theory. Psychometrika 49 425-436.
  • Sarkar, T. K. (1969). Some lower bounds of reliability. Technical Report, 124, Dept. Operation Research and Statistics, Stanford Univ.
  • Sarkar, S. K. (1998). Some probability inequalities for ordered MTP2 random variables: a proof of Simes' conjecture. Ann. Statist. 26 494-504.
  • Sarkar, S. K. and Chang, C. K. (1997). The Simes method for multiple hypotheses testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601-1608.
  • Seeger, (1968). A note on a method for the analysis of significances en mass. Technometrics 10 586-593. Sen, P. K. (1999a). Some remarks on Simes-type multiple tests of significance. J. Statist. Plann. Inference, 82 139-145. Sen, P. K. (1999b). Multiple comparisons in interim analysis. J. Statist. Plann. Inference 82 5-23.
  • Shaffer, J. P. (1995). Multiple hypotheses-testing. Ann. Rev. Psychol. 46 561-584.
  • Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751-754.
  • Steel, R. G. D. and Torrie, J. H. (1980). Principles and Procedures of Statistics: A Biometrical Approach, 2nd ed. McGraw-Hill, New York.
  • Tamhane, A. C. (1996). Multiple comparisons. In Handbook of Statistics (S. Ghosh and C. R. Rao, eds.) 13 587-629. North-Holland, Amsterdam.
  • Tamhane, A. C. and Dunnett, C. W. (1999). Stepwise multiple test procedures with biometric applications. J. Statist. Plann. Inference 82 55-68.
  • Troendle, J. (2000). Stepwise normal theory tests procedures controlling the false discovery rate. J. Statist. Plann. Inference 84 139-158.
  • Wassmer, G., Reitmer, P., Kieser, M. and Lehmacher, W. (1999). Procedures for testing multiple endpoints in clinical trials: an overview. J. Statist. Plann. Inference 82 69-81.
  • Weller, J. I., Song, J. Z., Heyen, D. W., Lewin, H. A. and Ron, M. (1998). A new approach to the problem of multiple comparison in the genetic dissection of complex traits. Genetics 150 1699-1706.
  • Westfall, P. H. and Young, S. S. (1993). Resampling Based Multiple Testing, Wiley, New York.
  • Williams, V. S. L., Jones, L. V. and Tukey, J. W. (1999). Controlling error in multiple comparisons, with special attention to the National Assessment of Educational Progress. J. Behav. Educ. Statist. 24 42-69.
  • Yekutieli, D. and Benjamini, Y. (1999). A resampling based false discovery rate controlling multiple test procedure. J. Statist. Plann. Inference 82 171-196.