Electronic Journal of Statistics

Asymptotic properties of false discovery rate controlling procedures under independence

Pierre Neuvial

Full-text: Open access


We investigate the performance of a family of multiple comparison procedures for strong control of the False Discovery Rate ($\mathsf{FDR}$). The $\mathsf{FDR}$ is the expected False Discovery Proportion ($\mathsf{FDP}$), that is, the expected fraction of false rejections among all rejected hypotheses. A number of refinements to the original Benjamini-Hochberg procedure [1] have been proposed, to increase power by estimating the proportion of true null hypotheses, either implicitly, leading to one-stage adaptive procedures [4, 7] or explicitly, leading to two-stage adaptive (or plug-in) procedures [2, 21].

We use a variant of the stochastic process approach proposed by Genovese and Wasserman [11] to study the fluctuations of the $\mathsf{FDP}$ achieved with each of these procedures around its expectation, for independent tested hypotheses.

We introduce a framework for the derivation of generic Central Limit Theorems for the $\mathsf{FDP}$ of these procedures, characterizing the associated regularity conditions, and comparing the asymptotic power of the various procedures. We interpret recently proposed one-stage adaptive procedures [4, 7] as fixed points in the iteration of well known two-stage adaptive procedures [2, 21].

Article information

Electron. J. Statist., Volume 2 (2008), 1065-1110.

First available in Project Euclid: 21 November 2008

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G10: Hypothesis testing 62H15: Hypothesis testing 60F05: Central limit and other weak theorems

Multiple hypothesis testing Benjamini-Hochberg procedure FDP FDR


Neuvial, Pierre. Asymptotic properties of false discovery rate controlling procedures under independence. Electron. J. Statist. 2 (2008), 1065--1110. doi:10.1214/08-EJS207. https://projecteuclid.org/euclid.ejs/1227287693

Export citation


  • [1] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing., J. R. Stat. Soc. Ser. B Stat. Methodol. 57, 1, 289–300.
  • [2] Benjamini, Y., Krieger, A. M., and Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate., Biometrika 93, 3, 491–507.
  • [3] Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency., Ann. Statist. 29, 4, 1165–1188.
  • [4] Blanchard, G. and Roquain, E. (2008). Adaptive FDR control under independence and dependence. arXiv preprint, math.ST/0707.0536v2.
  • [5] Chi, Z. (2007). On the performance of FDR control: constraints and a partial solution., Ann. Statist. 35, 4, 1409–1431.
  • [6] Donsker, M. D. (1951). An invariance principle for certain probability limit theorems., Mem. Amer. Math. Soc. 6, 12.
  • [7] Finner, H., Dickhaus, T., and Roters, M. (2008). On the false discovery rate and an asymptotically optimal rejection curve., Ann. Statist. (to appear).
  • [8] Finner, H. and Roters, M. (2001). On the false discovery rate and expected type I errors., Biom. J. 43, 8, 985–1005.
  • [9] Finner, H. and Roters, M. (2002). Multiple hypotheses testing and expected number of type I errors., Ann. Statist. 30, 1, 220–238.
  • [10] Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate., J. R. Stat. Soc. Ser. B Stat. Methodol. 64, 3, 499–517.
  • [11] Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control., Ann. Statist. 32, 3, 1035–1061.
  • [12] Genovese, C. and Wasserman, L. (2006). Exceedance control of the false discovery proportion., J. Amer. Statist. Assoc. 101, 476, 1408–1417.
  • [13] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate., Ann. Statist 33, 3, 1138–1154.
  • [14] Neuvial, P. (2008). Intrinsic bounds and false discovery rate control in multiple testing problems. In revision for, J. Mach. Learn. Res.
  • [15] Perone Pacifico, M., Genovese, C., Verdinelli, I., and Wasserman, L. (2004). False discovery control for random fields., J. Amer. Statist. Assoc. 99, 468, 1002–1014.
  • [16] Romano, J. P. and Shaikh, A. M. (2006a). On stepdown control of the false discovery proportion. In, Optimality: The Second Erich L. Lehmann Symposium, J. Rojo, Ed. Vol. 33. Institute of Mathematical Statistics, Beachwood, Ohio, USA.
  • [17] Romano, J. P. and Shaikh, A. M. (2006b). Step-up procedures for control of generalizations of the family-wise error rate., Ann. Statist. 34, 4, 1850–1873.
  • [18] Romano, J. P. and Wolf, M. (2007). Control of generalized error rates in multiple testing., Ann. Statist. 35, 4, 1378–1408.
  • [19] Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures., Ann. Statist. 30, 1, 239–257.
  • [20] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance., Biometrika 73, 3, 751-754.
  • [21] Storey, J. D. (2002). A direct approach to false discovery rates., J. R. Stat. Soc. Ser. B Stat. Methodol. 64, 3, 479–498.
  • [22] Storey, J. D. (2003). The positive false discovery rate: a bayesian interpretation and the q-value., Ann. Statist. 31, 6, 2013–2035.
  • [23] Storey, J. D., Taylor, J. E., and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach., J. R. Stat. Soc. Ser. B Stat. Methodol. 66, 1, 187–205.
  • [24] Storey, J. D. and Tibshirani, R. (2003). SAM thresholding and false discovery rates for detecting differential gene expression in DNA microarrays. In, The Analysis of Gene Expression Data: Methods and Software. Springer, New York, 272–290.
  • [25] Sun, W. and Cai, T. T. (2007). Oracle and adaptive compound decision rules for false discovery rate control., J. Amer. Statist. Assoc. 102, 479, 901–912.
  • [26] van der Laan, M. J., Dudoit, S., and Pollard, K. S. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives., Stat. Appl. Genet. Mol. Biol. 3, Art. 15, 27 pp.
  • [27] van der Vaart, A. W. (1998)., Asymptotic Statistics. Cambridge University Press.
  • [28] Wu, W. B. (2008). On false discovery control under dependence., Ann. Statist. 36, 1, 364–380.