Annals of Statistics

Asymptotic Bayes-optimality under sparsity of some multiple testing procedures

Małgorzata Bogdan, Arijit Chakrabarti, Florian Frommlet, and Jayanta K. Ghosh

Full-text: Open access


Within a Bayesian decision theoretic framework we investigate some asymptotic optimality properties of a large class of multiple testing rules. A parametric setup is considered, in which observations come from a normal scale mixture model and the total loss is assumed to be the sum of losses for individual tests. Our model can be used for testing point null hypotheses, as well as to distinguish large signals from a multitude of very small effects. A rule is defined to be asymptotically Bayes optimal under sparsity (ABOS), if within our chosen asymptotic framework the ratio of its Bayes risk and that of the Bayes oracle (a rule which minimizes the Bayes risk) converges to one. Our main interest is in the asymptotic scheme where the proportion p of “true” alternatives converges to zero.

We fully characterize the class of fixed threshold multiple testing rules which are ABOS, and hence derive conditions for the asymptotic optimality of rules controlling the Bayesian False Discovery Rate (BFDR). We finally provide conditions under which the popular Benjamini–Hochberg (BH) and Bonferroni procedures are ABOS and show that for a wide class of sparsity levels, the threshold of the former can be approximated by a nonrandom threshold.

It turns out that while the choice of asymptotically optimal FDR levels for BH depends on the relative cost of a type I error, it is almost independent of the level of sparsity. Specifically, we show that when the number of tests m increases to infinity, then BH with FDR level chosen in accordance with the assumed loss function is ABOS in the entire range of sparsity parameters pmβ, with β ∈ (0, 1].

Article information

Ann. Statist., Volume 39, Number 3 (2011), 1551-1579.

First available in Project Euclid: 7 June 2011

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62C25: Compound decision problems 62F05: Asymptotic properties of tests
Secondary: 62C10: Bayesian problems; characterization of Bayes procedures

Multiple testing FDR Bayes oracle asymptotic optimality


Bogdan, Małgorzata; Chakrabarti, Arijit; Frommlet, Florian; Ghosh, Jayanta K. Asymptotic Bayes-optimality under sparsity of some multiple testing procedures. Ann. Statist. 39 (2011), no. 3, 1551--1579. doi:10.1214/10-AOS869.

Export citation


  • [1] Abramovich, F., Benjamini, Y., Donoho, D. L. and Johnstone, I. M. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584–653.
  • [2] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • [3] Bogdan, M., Chakrabarti, A., Frommlet, F. and Ghosh, J. K. (2010). Supplement to “Asymptotic Bayes-optimality under sparsity of some multiple testing procedures.” DOI:10.1214/10-AOS869SUPP.
  • [4] Bogdan, M., Chakrabarti, A., Frommlet, F. and Ghosh, J. K. (2010). The Bayes oracle and asymptotic optimality of multiple testing procedures under sparsity. Available at arXiv:1002.3501.
  • [5] Bogdan, M., Ghosh, J. K., Ochman, A. and Tokdar, S. T. (2007). On the Empirical Bayes approach to the problem of multiple testing. Quality and Reliability Engineering International 23 727–739.
  • [6] Bogdan, M., Ghosh, J. K. and Tokdar, S. T. (2008). A comparison of the Benjamini–Hochberg procedure with some Bayesian rules for multiple testing. In Beyond Parametrics in Interdisciplinary Research: Festschrift in Honor of Professor Pranab K. Sen. Inst. Math. Stat. Collect. 1 211–230. IMS, Beachwood, OH.
  • [7] Cai, T. T. and Jin, J. (2010). Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing. Ann. Statist. 38 100–145.
  • [8] Cai, T. T., Jin, J. and Low, M. G. (2007). Estimation and confidence sets for sparse normal mixtures. Ann. Statist. 35 2421–2449.
  • [9] Carvalho, C. M., Polson, N. and Scott, J. G. (2008). The horseshoe estimator for sparse signals. Technical Report 2008-31, Dept. Statistical Science, Duke Univ.
  • [10] Chi, Z. (2007). On the performance of FDR control: Constraints and a partial solution. Ann. Statist. 35 1409–1431.
  • [11] Chi, Z. (2008). False discovery rate control with multivariate p-values. Electron. J. Stat. 2 368–411.
  • [12] Do, K.-A., Müller, P. and Tang, F. (2005). A Bayesian mixture model for differential gene expression. J. Roy. Statist. Soc. Ser. C 54 627–644.
  • [13] Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • [14] Donoho, D. and Jin, J. (2006). Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data. Ann. Statist. 34 2980–3018.
  • [15] Donoho, D. L. and Johnstone, I. M. (1994). Minimax risk over lp-balls for lq-error. Probab. Theory Related Fields 99 277–303.
  • [16] Efron, B. (2004). Large-scale simultaneous hypothesis testing: The choice of a null hypothesis. J. Amer. Statist. Assoc. 99 96–104.
  • [17] Efron, B. (2008). Microarrays, empirical Bayes and the two-groups model. Statist. Sci. 23 1–22.
  • [18] Efron, B. and Tibshirani, R. (2002). Empirical Bayes methods and false discovery rates for microarrays. Genet. Epidemiol. 23 70–86.
  • [19] Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc. 96 1151–1160.
  • [20] Finner, H., Dickhaus, T. and Roters, M. (2009). On the false discovery rate and an asymptotically optimal rejection curve. Ann. Statist. 37 596–618.
  • [21] Finner, H. and Roters, M. (2002). Multiple hypotheses testing and expected number of type I errors. Ann. Statist. 30 220–238.
  • [22] Frommlet, F., Bogdan, M. and Chakrabarti, A. (2010). Asymptotic Bayes optimality under sparsity for general priors under the alternative. Available at arXiv:1005.4753v1.
  • [23] Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 499–517.
  • [24] Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • [25] Guo, W. and Rao, M. B. (2008). On optimality of the Benjamini–Hochberg procedure for the false discovery rate. Statist. Probab. Lett. 78 2024–2030.
  • [26] Jin, J. and Cai, T. T. (2007). Estimating the null and the proportional of nonnull effects in large-scale multiple comparisons. J. Amer. Statist. Assoc. 102 495–506.
  • [27] Lehmann, E. L. (1957). A theory of some multiple decision problems. I. Ann. Math. Statist. 28 1–25.
  • [28] Lehmann, E. L. (1957). A theory of some multiple decision problems. II. Ann. Math. Statist. 28 547–572.
  • [29] Lehmann, E. L., Romano, J. P. and Shaffer, J. P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108.
  • [30] Meinshausen, N. and Rice, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 373–393.
  • [31] Müller, P., Parmigiani, G., Robert, C. and Rousseau, J. (2004). Optimal sample size for multiple testing: The case of gene expression microarrays. J. Amer. Statist. Assoc. 99 990–1001.
  • [32] Peña, E. A., Habiger, J. D. and Wu, W. (2011). Power-enhanced multiple decision functions controlling family-wise error and false discovery rates. Ann. Statist. 39 556–583.
  • [33] Roquain, E. and van de Wiel, M. A. (2009). Optimal weighting for false discovery rate control. Electron. J. Stat. 3 678–711.
  • [34] Sarkar, S. K. (2006). False discovery and false nondiscovery rates in single-step multiple testing procedures. Ann. Statist. 34 394–415.
  • [35] Scott, J. G. and Berger, J. O. (2006). An exploration of aspects of Bayesian multiple testing. J. Statist. Plann. Inference 136 2144–2162.
  • [36] Scott, J. G. and Berger, J. O. (2010). Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann. Statist. 38 2587–2619.
  • [37] Seeger, P. (1968). A note on a method for the analysis of significance en masse. Technometrics 10 586–593.
  • [38] Sen, P. K. (1999). Some remarks on Simes-type multiple tests of significance. J. Statist. Plann. Inference 82 139–145.
  • [39] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • [40] Storey, J. D. (2003). The positive false discovery rate: A Bayesian interpretation and the q-value. Ann. Statist. 31 2013–2035.
  • [41] Storey, J. D. (2007). The optimal discovery procedure: A new approach to simultaneous significance testing. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 347–368.
  • [42] Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 66 187–205.
  • [43] Sun, W. and Cai, T. T. (2007). Oracle and adaptive compound decision rules for false discovery rate control. J. Amer. Statist. Assoc. 102 901–912.
  • [44] Tai, Y. C. and Speed, T. P. (2006). A multivariate empirical Bayes statistic for replicated microarray time course data. Ann. Statist. 34 2387–2412.

Supplemental materials

  • Supplementary material: Supplement to “Asymptotic Bayes-optimality under sparsity of some multiple testing procedures”. Analysis of behavior of BFDR for scale mixtures of normal distributions and proofs of Theorems 3.2, 4.1 and 5.1.