The Annals of Statistics

Exact calculations for false discovery proportion with application to least favorable configurations

Etienne Roquain and Fanny Villers

Full-text: Open access


In a context of multiple hypothesis testing, we provide several new exact calculations related to the false discovery proportion (FDP) of step-up and step-down procedures. For step-up procedures, we show that the number of erroneous rejections conditionally on the rejection number is simply a binomial variable, which leads to explicit computations of the c.d.f., the sth moment and the mean of the FDP, the latter corresponding to the false discovery rate (FDR). For step-down procedures, we derive what is to our knowledge the first explicit formula for the FDR valid for any alternative c.d.f. of the p-values. We also derive explicit computations of the power for both step-up and step-down procedures. These formulas are “explicit” in the sense that they only involve the parameters of the model and the c.d.f. of the order statistics of i.i.d. uniform variables. The p-values are assumed either independent or coming from an equicorrelated multivariate normal model and an additional mixture model for the true/false hypotheses is used. Our approach is then used to investigate new results which are of interest in their own right, related to least/most favorable configurations for the FDR and the variance of the FDP.

Article information

Ann. Statist., Volume 39, Number 1 (2011), 584-612.

First available in Project Euclid: 15 February 2011

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J15: Paired and multiple comparisons
Secondary: 62G10: Hypothesis testing 60C05: Combinatorial probability

False discovery rate false discovery proportion multiple testing least favorable configuration power equicorrelated multivariate normal distribution step-up step-down


Roquain, Etienne; Villers, Fanny. Exact calculations for false discovery proportion with application to least favorable configurations. Ann. Statist. 39 (2011), no. 1, 584--612. doi:10.1214/10-AOS847.

Export citation


  • [1] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • [2] Benjamini, Y., Krieger, A. M. and Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93 491–507.
  • [3] Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • [4] Blanchard, G. and Roquain, E. (2008). Two simple sufficient conditions for FDR control. Electron. J. Stat. 2 963–992.
  • [5] Blanchard, G. and Roquain, E. (2009). Adaptive false discovery rate control under independence and dependence. J. Mach. Learn. Res. 10 2837–2871.
  • [6] Chi, Z. (2007). On the performance of FDR control: Constraints and a partial solution. Ann. Statist. 35 1409–1431.
  • [7] Chi, Z. and Tan, Z. (2008). Positive false discovery proportions: Intrinsic bounds and adaptive control. Statist. Sinica 18 837–860.
  • [8] Dickhaus, T. (2008). False discovery rate and asymptotics. Ph.D. thesis, Heinrich-Heine Univ., Düsseldorf.
  • [9] Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • [10] Efron, B. (2008). Microarrays, empirical Bayes and the two-groups model. Statist. Sci. 23 1–22.
  • [11] Efron, B. (2010). Correlated z-values and the accuracy of large-scale statistical estimates. J. Amer. Statist. Assoc. 105 1042–1055.
  • [12] Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc. 96 1151–1160.
  • [13] Ferreira, J. A. and Zwinderman, A. H. (2006). On the Benjamini–Hochberg method. Ann. Statist. 34 1827–1849.
  • [14] Finner, H., Dickhaus, R. and Roters, M. (2009). On the false discovery rate and an asymptotically optimal rejection curve. Ann. Statist. 37 596–618.
  • [15] Finner, H., Dickhaus, T. and Roters, M. (2007). Dependency and false discovery rate: Asymptotics. Ann. Statist. 35 1432–1455.
  • [16] Finner, H. and Roters, M. (2002). Multiple hypotheses testing and expected number of type I errors. Ann. Statist. 30 220–238.
  • [17] Gavrilov, Y., Benjamini, Y. and Sarkar, S. K. (2009). An adaptive step-down procedure with proven FDR control under independence. Ann. Statist. 37 619–629.
  • [18] Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 499–517.
  • [19] Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • [20] Glueck, D. H., Mandel, J., Karimpour-Fard, A., Hunter, L. and Muller, K. E. (2008). Exact calculations of average power for the Benjamini–Hochberg procedure. Int. J. Biostat. 4 1103.
  • [21] Guo, W. and Rao, M. B. (2008). On control of the false discovery rate under no assumption of dependency. J. Statist. Plann. Inference 138 3176–3188.
  • [22] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • [23] Neuvial, P. (2008). Asymptotic properties of false discovery rate controlling procedures under independence. Electron. J. Stat. 2 1065–1110.
  • [24] Owen, D. B. and Steck, G. P. (1962). Moments of order statistics from the equicorrelated multivariate normal distribution. Ann. Math. Statist. 33 1286–1291.
  • [25] Reiner-Benaim, A. (2007). FDR control by the BH procedure for two-sided correlated tests with implications to gene expression data analysis. Biom. J. 49 107–126.
  • [26] Roquain, E. (2007). Exceptional motifs in heterogeneous sequences. Contributions to theory and methodology of multiple testing. Ph.D. thesis, Univ. Paris XI.
  • [27] Roquain, E. and van de Wiel, M. (2009). Optimal weighting for false discovery rate control. Electron. J. Stat. 3 678–711.
  • [28] Roquain, E. and Villers, F. (2010). Supplement to “Exact calculations for false discovery proportion with application to least favorable configurations.” DOI: 10.1214/10-AOS847SUPP.
  • [29] Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
  • [30] Sarkar, S. K. (2008). On methods controlling the false discovery rate. Sankhyā 70 135–168.
  • [31] Seeger, P. (1968). A note on a method for the analysis of significances en masse. Technometrics 10 586–593.
  • [32] Shorack, G. R. and Wellner, J. A. (1986). Empirical Processes With Applications to Statistics. Wiley, New York.
  • [33] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • [34] Storey, J. D. (2003). The positive false discovery rate: A Bayesian interpretation and the q-value. Ann. Statist. 31 2013–2035.
  • [35] Stuart, A. (1958). Equally correlated variates and the multinormal integral. J. Roy. Statist. Soc. Ser. B 20 373–378.
  • [36] Zeisel, A., Zuk, O. and Domany, E. (2011). FDR control with adaptive procedures and FDR monotonicity. Ann. Appl. Statist. To appear. Available at ArXiv:0909.3704v1.

Supplemental materials

  • Supplementary material: Supplement to “Exact calculations for false discovery proportion with application to least favorable configurations”. Supplement which provides some proofs for the present paper.