Annals of Statistics

Generalizations of the familywise error rate

E. L. Lehmann and Joseph P. Romano

Full-text: Open access

Enhanced PDF (120 KB)

Abstract

Consider the problem of simultaneously testing null hypotheses H1,…,Hs. The usual approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, particularly if s is large, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which we call the k-FWER. We derive both single-step and stepdown procedures that control the k-FWER, without making any assumptions concerning the dependence structure of the p-values of the individual tests. In particular, we derive a stepdown procedure that is quite simple to apply, and prove that it cannot be improved without violation of control of the k-FWER. We also consider the false discovery proportion (FDP) defined by the number of false rejections divided by the total number of rejections (defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289–300] controls E(FDP). Here, we construct methods such that, for any γ and α, P{FDP>γ}≤α. Two stepdown methods are proposed. The first holds under mild conditions on the dependence structure of p-values, while the second is more conservative but holds without any dependence assumptions.

Article information

Source
Ann. Statist., Volume 33, Number 3 (2005), 1138-1154.

Dates
First available in Project Euclid: 1 July 2005

Permanent link to this document
https://projecteuclid.org/euclid.aos/1120224098

Digital Object Identifier
doi:10.1214/009053605000000084

Mathematical Reviews number (MathSciNet)
MR2195631

Zentralblatt MATH identifier
1072.62060

Subjects
Primary: 62J15: Paired and multiple comparisons
Secondary: 62G10: Hypothesis testing

Keywords
Familywise error rate multiple testing p-value stepdown procedure

Citation

Lehmann, E. L.; Romano, Joseph P. Generalizations of the familywise error rate. Ann. Statist. 33 (2005), no. 3, 1138--1154. doi:10.1214/009053605000000084. https://projecteuclid.org/euclid.aos/1120224098


Export citation

References

  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
    Zentralblatt MATH: 0809.62014
    Digital Object Identifier: doi:10.1111/j.2517-6161.1995.tb02031.x
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
    Zentralblatt MATH: 1041.62061
    Digital Object Identifier: doi:10.1214/aos/1013699998
    Project Euclid: euclid.aos/1013699998
  • Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
    Mathematical Reviews (MathSciNet): MR2065197
    Zentralblatt MATH: 1092.62065
    Digital Object Identifier: doi:10.1214/009053604000000283
    Project Euclid: euclid.aos/1085408494
  • Hochberg, Y. and Tamhane, A. (1987). Multiple Comparison Procedures. Wiley, New York.
    Zentralblatt MATH: 0731.62125
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70.
    Zentralblatt MATH: 0402.62058
  • Hommel, G. (1983). Tests of the overall hypothesis for arbitrary dependence structures. Biometrical J. 25 423–430.
    Zentralblatt MATH: 0524.62027
    Digital Object Identifier: doi:10.1002/bimj.19830250502
  • Hommel, G. and Hoffman, T. (1988). Controlled uncertainty. In Multiple Hypotheses Testing (P. Bauer, G. Hommel and E. Sonnemann, eds.) 154–161. Springer, Heidelberg.
  • Korn, E., Troendle, J., McShane, L. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data. J. Statist. Plann. Inference 124 379–398.
    Zentralblatt MATH: 1074.62070
    Digital Object Identifier: doi:10.1016/S0378-3758(03)00211-8
  • Perone Pacifico, M., Genovese, C., Verdinelli, I. and Wasserman, L. (2004). False discovery control for random fields. J. Amer. Statist. Assoc. 99 1002–1014.
    Zentralblatt MATH: 1055.62105
    Digital Object Identifier: doi:10.1198/0162145000001655
  • Sarkar, S. (1998). Some probability inequalities for ordered $\mm{MTP}_2$ random variables: A proof of the Simes conjecture. Ann. Statist. 26 494–504.
  • Sarkar, S. and Chang, C. (1997). The Simes method for multiple hypothesis testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601–1608.
    Zentralblatt MATH: 0912.62079
    Digital Object Identifier: doi:10.1080/01621459.1997.10473682
  • Simes, R. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
    Zentralblatt MATH: 0613.62067
    Digital Object Identifier: doi:10.1093/biomet/73.3.751
  • van der Laan, M., Dudoit, S. and Pollard, K. (2004). Multiple testing. Part III. Procedures for control of the generalized family-wise error rate and proportion of false positives. Working Paper Series, Paper 141, Div. Biostatistics, Univ. California, Berkeley.
    Mathematical Reviews (MathSciNet): MR2101463
    Zentralblatt MATH: 1166.62379
  • Victor, N. (1982). Exploratory data analysis and clinical research. Methods of Information in Medicine 21 53–54.

The Institute of Mathematical Statistics

Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more