Annals of Statistics

The control of the false discovery rate in multiple testing under dependency

Yoav Benjamini and Daniel Yekutieli

Full-text: Open access

PDF File (166 KB)

Abstract

Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate $t$. Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which a procedure with proven FDR control can be offered is greatly increased.

Article information

Source
Ann. Statist., Volume 29, Number 4 (2001), 1165-1188.

Dates
First available in Project Euclid: 14 February 2002

Permanent link to this document
https://projecteuclid.org/euclid.aos/1013699998

Digital Object Identifier
doi:10.1214/aos/1013699998

Mathematical Reviews number (MathSciNet)
MR1869245

Zentralblatt MATH identifier
1041.62061

Subjects
Primary: 62J15: Paired and multiple comparisons 62G30: Order statistics; empirical distribution functions 47N30: Applications in probability theory and statistics

Keywords
Multiple comparisons procedures FDR Simes’equality Hochberg’s procedure MTP2 densities positive regression dependency unidimensional latent variables discrete test statistics multiple endpoints many-to-one comparisons comparisons with control

Citation

Benjamini, Yoav; Yekutieli, Daniel. The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 (2001), no. 4, 1165--1188. doi:10.1214/aos/1013699998. https://projecteuclid.org/euclid.aos/1013699998


Export citation

References

  • Abramovich, F. and Benjamini, Y. (1996). Adaptive thresholding of wavelet coefficients. Comput. Statist. Data Anal. 22 351-361.
    Mathematical Reviews (MathSciNet): MR97e:62052
  • Barinaga, M. (1994). From fruit flies, rats, mice: evidence of genetic influence. Science 264 1690-1693.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289-300.
    Mathematical Reviews (MathSciNet): MR96d:62143
    JSTOR: links.jstor.org
  • Benjamini, Y. and Hochberg, Y. (1997). Multiple hypotheses testing with weights. Scand. J. Statist. 24 407-418.
    Mathematical Reviews (MathSciNet): MR99d:62046
    Digital Object Identifier: doi:10.1111/1467-9469.00072
  • Benjamini, Y. and Hochberg, Y. (2000). The adaptive control of the false discovery rate in multiple hypotheses testing. J. Behav. Educ. Statist. 25 60-83.
  • Benjamini, Y., Hochberg, Y. and Kling, Y. (1993). False discovery rate control in pairwise comparisons. Working Paper 93-2, Dept. Statistics and O.R., Tel Aviv Univ.
  • Benjamini, Y., Hochberg, Y. and Kling, Y. (1997). False discovery rate control in multiple hypotheses testing using dependent test statistics. Research Paper 97-1, Dept. Statistics and O.R., Tel Aviv Univ.
  • Benjamini, Y. and Wei, L. (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Statist. Plann. Inference 82 163-170.
    Mathematical Reviews (MathSciNet): MR1736441
    Zentralblatt MATH: 1063.62558
    Digital Object Identifier: doi:10.1016/S0378-3758(99)00040-3
  • Chang, C. K., Rom, D. M. and Sarkar, S. K. (1996). A modified Bonferroni procedure for repeated significance testing. Technical Report 96-01, Temple Univ.
  • Eaton, M. L. (1986). Lectures on topics in probability inequalities. CWI Tract 35.
    Mathematical Reviews (MathSciNet): MR889671
    Zentralblatt MATH: 0622.60024
  • Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800-803.
    Mathematical Reviews (MathSciNet): MR90d:62037
    Zentralblatt MATH: 0661.62067
    Digital Object Identifier: doi:10.1093/biomet/75.4.800
    JSTOR: links.jstor.org
  • Hochberg, Y. and Hommel, G. (1998). Step-up multiple testing procedures. Encyclopedia Statist. Sci. (Supp.) 2.
  • Hochberg, Y. and Rom, D. (1995). Extensions of multiple testing procedures based on Simes' test. J. Statist. Plann. Inference 48 141-152.
    Mathematical Reviews (MathSciNet): MR97c:62038
    Zentralblatt MATH: 0851.62054
    Digital Object Identifier: doi:10.1016/0378-3758(95)00005-T
  • Hochberg, Y. and Tamhane, A. (1987). Multiple Comparison Procedures. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR89c:62125
  • Holland, P. W. and Rosenbaum, P. R. (1986). Conditional association and unidimensionality in monotone latent variable models. Ann. Statist. 14 1523-1543.
    Mathematical Reviews (MathSciNet): MR88g:62115
    Zentralblatt MATH: 0625.62102
    Digital Object Identifier: doi:10.1214/aos/1176350174
    Project Euclid: euclid.aos/1176350174
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist 6 65-70.
    Mathematical Reviews (MathSciNet): MR81i:62042
  • Hommel, G. (1988). A stage-wise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383-386.
  • Hsu, J. (1996). Multiple Comparisons Procedures. Chapman and Hall, London.
    Mathematical Reviews (MathSciNet): MR1629127
  • Karlin, S. and Rinott, Y. (1980). Classes of orderings of measures and related correlation inequalities I. Multivariate totally positive distributions. J. Multivariate Statist. 10 467-498.
    Zentralblatt MATH: 0469.60006
    Mathematical Reviews (MathSciNet): MR599685
    Digital Object Identifier: doi:10.1016/0047-259X(80)90065-2
  • Karlin, S. and Rinott, Y. (1981). Total positivity properties of absolute value multinormal variable with applications to confidence interval estimates and related probabilistic inequalities. Ann. Statist. 9 1035-1049.
    Mathematical Reviews (MathSciNet): MR83c:62081
    Zentralblatt MATH: 0477.62035
    Digital Object Identifier: doi:10.1214/aos/1176345583
    Project Euclid: euclid.aos/1176345583
  • Lander E. S. and Botstein D. (1989). Mapping Mendelian factors underlying quantitative traits using RFLP linkage maps. Genetics 121 185-190.
  • Lander, E. S. and Kruglyak L. (1995). Genetic dissection of complex traits: guidelines for interpreting and reporting linkage results. Nature Genetics 11 241-247.
  • Lehmann, E. L. (1966). Some concepts of dependence. Ann. Math. Statist. 37 1137-1153.
    Mathematical Reviews (MathSciNet): MR34:2101
    Zentralblatt MATH: 0146.40601
    Digital Object Identifier: doi:10.1214/aoms/1177699260
    Project Euclid: euclid.aoms/1177699260
  • Needleman, H., Gunnoe, C., Leviton, A., Reed, R., Presie, H., Maher, C. and Barret, P. (1979). Deficits in psychologic and classroom performance of children with elevated dentine lead levels. New England J. Medicine 300 689-695.
  • Paterson, A. H. G., Powles, T. J., Kanis, J. A., McCloskey, E., Hanson, J. and Ashley, S. (1993). Double-blind controlled trial of oral clodronate in patients with bone metastases from breastcancer. J. Clinical Oncology 1 59-65.
  • Rosenbaum, P. R. (1984). Testing the conditional independence and monotonicity assumptions of item response theory. Psychometrika 49 425-436.
    Mathematical Reviews (MathSciNet): MR86f:62187
    Zentralblatt MATH: 0569.62097
    Digital Object Identifier: doi:10.1007/BF02306030
  • Sarkar, T. K. (1969). Some lower bounds of reliability. Technical Report, 124, Dept. Operation Research and Statistics, Stanford Univ.
  • Sarkar, S. K. (1998). Some probability inequalities for ordered MTP2 random variables: a proof of Simes' conjecture. Ann. Statist. 26 494-504.
    Mathematical Reviews (MathSciNet): MR99m:62024
    Zentralblatt MATH: 0929.62065
    Digital Object Identifier: doi:10.1214/aos/1028144846
    Project Euclid: euclid.aos/1028144846
  • Sarkar, S. K. and Chang, C. K. (1997). The Simes method for multiple hypotheses testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601-1608.
    Mathematical Reviews (MathSciNet): MR1615269
    Zentralblatt MATH: 0912.62079
    Digital Object Identifier: doi:10.2307/2965431
    JSTOR: links.jstor.org
  • Seeger, (1968). A note on a method for the analysis of significances en mass. Technometrics 10 586-593. Sen, P. K. (1999a). Some remarks on Simes-type multiple tests of significance. J. Statist. Plann. Inference, 82 139-145. Sen, P. K. (1999b). Multiple comparisons in interim analysis. J. Statist. Plann. Inference 82 5-23.
  • Shaffer, J. P. (1995). Multiple hypotheses-testing. Ann. Rev. Psychol. 46 561-584.
  • Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751-754.
    Mathematical Reviews (MathSciNet): MR88d:62043
    Zentralblatt MATH: 0613.62067
    Digital Object Identifier: doi:10.1093/biomet/73.3.751
    JSTOR: links.jstor.org
  • Steel, R. G. D. and Torrie, J. H. (1980). Principles and Procedures of Statistics: A Biometrical Approach, 2nd ed. McGraw-Hill, New York.
  • Tamhane, A. C. (1996). Multiple comparisons. In Handbook of Statistics (S. Ghosh and C. R. Rao, eds.) 13 587-629. North-Holland, Amsterdam.
    Mathematical Reviews (MathSciNet): MR1492580
  • Tamhane, A. C. and Dunnett, C. W. (1999). Stepwise multiple test procedures with biometric applications. J. Statist. Plann. Inference 82 55-68.
    Mathematical Reviews (MathSciNet): MR1736432
    Zentralblatt MATH: 0979.62049
    Digital Object Identifier: doi:10.1016/S0378-3758(99)00031-2
  • Troendle, J. (2000). Stepwise normal theory tests procedures controlling the false discovery rate. J. Statist. Plann. Inference 84 139-158.
    Mathematical Reviews (MathSciNet): MR1747501
    Zentralblatt MATH: 1131.62310
    Digital Object Identifier: doi:10.1016/S0378-3758(99)00145-7
  • Wassmer, G., Reitmer, P., Kieser, M. and Lehmacher, W. (1999). Procedures for testing multiple endpoints in clinical trials: an overview. J. Statist. Plann. Inference 82 69-81.
    Mathematical Reviews (MathSciNet): MR1736433
    Zentralblatt MATH: 0979.62090
    Digital Object Identifier: doi:10.1016/S0378-3758(99)00032-4
  • Weller, J. I., Song, J. Z., Heyen, D. W., Lewin, H. A. and Ron, M. (1998). A new approach to the problem of multiple comparison in the genetic dissection of complex traits. Genetics 150 1699-1706.
  • Westfall, P. H. and Young, S. S. (1993). Resampling Based Multiple Testing, Wiley, New York.
  • Williams, V. S. L., Jones, L. V. and Tukey, J. W. (1999). Controlling error in multiple comparisons, with special attention to the National Assessment of Educational Progress. J. Behav. Educ. Statist. 24 42-69.
  • Yekutieli, D. and Benjamini, Y. (1999). A resampling based false discovery rate controlling multiple test procedure. J. Statist. Plann. Inference 82 171-196.
    Mathematical Reviews (MathSciNet): MR1736442
    Zentralblatt MATH: 1063.62563
    Digital Object Identifier: doi:10.1016/S0378-3758(99)00041-5

The Institute of Mathematical Statistics

Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more