Annals of Statistics

Stepup procedures controlling generalized FWER and generalized FDR

Sanat K. Sarkar

Full-text: Open access


In many applications of multiple hypothesis testing where more than one false rejection can be tolerated, procedures controlling error rates measuring at least k false rejections, instead of at least one, for some fixed k≥1 can potentially increase the ability of a procedure to detect false null hypotheses. The k-FWER, a generalized version of the usual familywise error rate (FWER), is such an error rate that has recently been introduced in the literature and procedures controlling it have been proposed. A further generalization of a result on the k-FWER is provided in this article. In addition, an alternative and less conservative notion of error rate, the k-FDR, is introduced in the same spirit as the k-FWER by generalizing the usual false discovery rate (FDR). A k-FWER procedure is constructed given any set of increasing constants by utilizing the kth order joint null distributions of the p-values without assuming any specific form of dependence among all the p-values. Procedures controlling the k-FDR are also developed by using the kth order joint null distributions of the p-values, first assuming that the sets of null and nonnull p-values are mutually independent or they are jointly positively dependent in the sense of being multivariate totally positive of order two (MTP2) and then discarding that assumption about the overall dependence among the p-values.

Article information

Ann. Statist., Volume 35, Number 6 (2007), 2405-2420.

First available in Project Euclid: 22 January 2008

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J15: Paired and multiple comparisons 62H15: Hypothesis testing
Secondary: 62H99: None of the above, but in this section

Generalized Holm procedure generalized Hochberg procedure generalized BH procedure generalized BY procedure equicorrelated multivariate normal


Sarkar, Sanat K. Stepup procedures controlling generalized FWER and generalized FDR. Ann. Statist. 35 (2007), no. 6, 2405--2420. doi:10.1214/009053607000000398.

Export citation


  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Benjamini, Y. and Yekutieli, D. (2005). False discovery rate-adjusted multiple confidence intervals for selected parameters (with discussion). J. Amer. Statist. Assoc. 100 71–93.
  • Fan, J., Hall, P. and Yao, Q. (2006). To how many simultaneous hypothesis tests can normal, Student's t or bootstrap calibration be applied? Unpublished manuscript.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 499–517.
  • Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • Hochberg, Y. (1988). A sharper $\rmB$onferroni procedure for multiple tests of significance. Biometrika 75 800–802.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70.
  • Karlin, S. and Rinott, Y. (1980). Classes of orderings of measures and related correlation inequalities. $\textscI$. Multivariate totally positive distributions. J. Multivariate Anal. 10 467–498.
  • Korn, E., Troendle, J., McShane, L. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data. J. Statist. Plann. Inference 124 379–398.
  • Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • Meinshausen, N. and Rice, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 373–393.
  • Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34 1850–1873.
  • Sarkar, S. K. (1998). Some probability inequalities for ordered MTP$_2$ random variables: A proof of the Simes conjecture. Ann. Statist. 26 494–504.
  • Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
  • Sarkar, S. K. (2004). FDR-controlling stepwise procedures and their false negatives rates. J. Statist. Plann. Inference 125 119–137.
  • Sarkar, S. K. (2006). False discovery and false nondiscovery rates in single-step multiple testing procedures. Ann. Statist. 34 394–415.
  • Sarkar, S. K. (2007). Generalizing Simes' test and Hochberg's stepup procedure. Ann. Statist. To appear.
  • Sarkar, S. K. and Chang, C.-K. (1997). The Simes method for multiple hypothesis testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601–1608.
  • Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479–498.
  • Storey, J. D. (2003). The positive false discovery rate: A Bayesian interpretation and the $q$-value. Ann. Statist. 31 2013–2035.
  • van der Laan, M., Dudoit, S. and Pollard, K. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat. App. Gen. Mol. Biol. 3 Article 15.