The Annals of Statistics

On the performance of FDR control: Constraints and a partial solution

Zhiyi Chi

Full-text: Open access


The False Discovery Rate (FDR) paradigm aims to attain certain control on Type I errors with relatively high power for multiple hypothesis testing. The Benjamini–Hochberg (BH) procedure is a well-known FDR controlling procedure. Under a random effects model, we show that, in general, unlike the FDR, the positive FDR (pFDR) of the BH procedure cannot be controlled at an arbitrarily low level due to the limited evidence provided by the observations to separate false and true nulls. This results in a criticality phenomenon, which is characterized by a transition of the procedure’s power from being positive to asymptotically 0 without any reduction in the pFDR, once the target FDR control level is below a positive critical value. To address the constraints on the power and pFDR control imposed by the criticality phenomenon, we propose a procedure which applies BH-type procedures at multiple locations in the domain of $p$-values. Both analysis and simulations show that the proposed procedure can attain substantially improved power and pFDR control.

Article information

Ann. Statist., Volume 35, Number 4 (2007), 1409-1431.

First available in Project Euclid: 29 August 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G10: Hypothesis testing 62H15: Hypothesis testing
Secondary: 60G35: Signal detection and filtering [See also 62M20, 93E10, 93E11, 94Axx]

multiple hypothesis testing Benjamini–Hochberg FDR pFDR power Bahadur representation


Chi, Zhiyi. On the performance of FDR control: Constraints and a partial solution. Ann. Statist. 35 (2007), no. 4, 1409--1431. doi:10.1214/009053607000000037.

Export citation


  • Abramovich, F., Benjamini, Y., Donoho, D. and Johnstone, I. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584–653.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. J. Educational and Behavioral Statistics 25 60–83.
  • Chi, Z. and Tan, Z. (2007). Positive false discovery proportions for multiple testing: Intrinsic bounds and adaptive control. Statist. Sinica. To appear.
  • Donoho, D. and Jin, J. (2006). Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data. Ann. Statist. 34 2980–3018.
  • Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. G. (2001). Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc. 96 1151–1160.
  • Finner, H. and Roters, M. (2002). Multiple hypotheses testing and expected number of type I errors. Ann. Statist. 30 220–238.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 499–517.
  • Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • Meinshausen, N. and Rice, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 373–393.
  • R Development Core Team (2005). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. Available at
  • Sarkar, S. K. (2006). False discovery and false nondiscovery rates in single-step multiple testing procedures. Ann. Statist. 34 394–415.
  • Shorack, G. R. and Wellner, J. A. (1986). Empirical Processes with Applications to Statistics. Wiley, New York.
  • Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479–498.
  • Storey, J. D. (2003). The positive false discovery rate: A Bayesian interpretation and the $q$-value. Ann. Statist. 31 2013–2035.
  • Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 66 187–205.
  • Yekutieli, D. and Benjamini, Y. (1999). Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Statist. Plann. Inference 82 171–196.

Supplemental materials