The Annals of Statistics

Family-Wise Separation Rates for multiple testing

Magalie Fromont, Matthieu Lerasle, and Patricia Reynaud-Bouret

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Starting from a parallel between some minimax adaptive tests of a single null hypothesis, based on aggregation approaches, and some tests of multiple hypotheses, we propose a new second kind error-related evaluation criterion, as the core of an emergent minimax theory for multiple tests. Aggregation-based tests, proposed for instance by Baraud [Bernoulli 8 (2002) 577–606], Baraud, Huet and Laurent [Ann. Statist. 31 (2003) 225–251] or Fromont and Laurent [Ann. Statist. 34 (2006) 680–720], are justified through their first kind error rate, which is controlled by the prescribed level on the one hand, and through their separation rates over various classes of alternatives, rates which are minimax on the other hand. We show that some of these tests can be viewed as the first steps of classical step-down multiple testing procedures, and accordingly be evaluated from the multiple testing point of view also, through a control of their Family-Wise Error Rate (FWER). Conversely, many multiple testing procedures, from the historical ones of Bonferroni and Holm, to more recent ones like min-$p$ procedures or randomized procedures such as the ones proposed by Romano and Wolf [J. Amer. Statist. Assoc. 100 (2005) 94–108], can be investigated from the minimax adaptive testing point of view. To this end, we extend the notion of separation rate to the multiple testing field, by defining the weak Family-Wise Separation Rate and its stronger counterpart, the Family-Wise Separation Rate (FWSR). As for nonparametric tests of a single null hypothesis, we prove that these new concepts allow an accurate analysis of the second kind error of a multiple testing procedure, leading to clear definitions of minimax and minimax adaptive multiple tests. Some illustrations in classical Gaussian frameworks corroborate several expected results under particular conditions on the tested hypotheses, but also lead to new questions and perspectives.

Article information

Ann. Statist., Volume 44, Number 6 (2016), 2533-2563.

Received: January 2015
Revised: November 2015
First available in Project Euclid: 23 November 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G10: Hypothesis testing 62H15: Hypothesis testing 62C20: Minimax procedures

Multiple testing Family-Wise Error Rate step-down procedure separation rate minimax test adaptive test


Fromont, Magalie; Lerasle, Matthieu; Reynaud-Bouret, Patricia. Family-Wise Separation Rates for multiple testing. Ann. Statist. 44 (2016), no. 6, 2533--2563. doi:10.1214/15-AOS1418.

Export citation


  • [1] Baraud, Y. (2002). Non-asymptotic minimax rates of testing in signal detection. Bernoulli 8 577–606.
  • [2] Baraud, Y., Huet, S. and Laurent, B. (2003). Adaptive tests of linear hypotheses by model selection. Ann. Statist. 31 225–251.
  • [3] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 289–300.
  • [4] Birgé, L. (2001). An alternative point of view on Lepski’s method. In State of the Art in Probability and Statistics (Leiden, 1999). Institute of Mathematical Statistics Lecture Notes—Monograph Series 36 113–133. IMS, Beachwood, OH.
  • [5] Collier, O. and Dalalyan, A. S. (2015). Curve registration by nonparametric goodness-of-fit testing. J. Statist. Plann. Inference 162 20–42.
  • [6] Dudoit, S. and van Der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genomics. Springer, New York.
  • [7] Dümbgen, L. and Spokoiny, V. G. (2001). Multiscale testing of qualitative hypotheses. Ann. Statist. 29 124–152.
  • [8] Dümbgen, L. and Walther, G. (2008). Multiscale inference about a density. Ann. Statist. 36 1758–1785.
  • [9] Fromont, M. and Laurent, B. (2006). Adaptive goodness-of-fit tests in a density model. Ann. Statist. 34 680–720.
  • [10] Fromont, M., Laurent, B., Lerasle, M. and Reynaud-Bouret, P. (2012). Kernels based tests with non-asymptotic bootstrap approaches for two-sample problems. In Journal of Machine Learning Research: Workshop and Conference Proceedings, COLT 2012 23 23.1–23.22. Edimbourg, UK.
  • [11] Fromont, M., Laurent, B. and Reynaud-Bouret, P. (2011). Adaptive tests of homogeneity for a Poisson process. Ann. Inst. Henri Poincaré Probab. Stat. 47 176–213.
  • [12] Fromont, M., Laurent, B. and Reynaud-Bouret, P. (2013). The two-sample problem for Poisson processes: Adaptive tests with a nonasymptotic wild bootstrap approach. Ann. Statist. 41 1431–1461.
  • [13] Fromont, M., Lerasle, M. and Reynaud-Bouret, P. (2016). Supplement to “Family-Wise Separation Rates for multiple testing.” DOI:10.1214/15-AOS1418SUPP.
  • [14] Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • [15] Goeman, J. J. and Solari, A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810.
  • [16] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 65–70.
  • [17] Hommel, G. and Hoffmann, T. (1988). Controlled uncertainty. In Multiple Hypothesenprüfung/Multiple Hypotheses Testing 154–161. Springer, Berlin.
  • [18] Inglot, T. (2010). Inequalities for quantiles of the chi-square distribution. Probab. Math. Statist. 30 339–351.
  • [19] Ingster, Y. I. (1993). Asymptotically minimax hypothesis testing for nonparametric alternatives. I, II, III. Math. Methods Statist. 2 85–114, 171–189, 249–268.
  • [20] Ingster, Y. I. (2000). Adaptive chi-square tests. J. Math. Sci. 99 1110–1119.
  • [21] Korn, E. L., Troendle, J. F., McShane, L. M. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data. J. Statist. Plann. Inference 124 379–398.
  • [22] Laurent, B., Loubes, J.-M. and Marteau, C. (2012). Non asymptotic minimax rates of testing in signal detection with heterogeneous variances. Available at
  • [23] Laurent, B., Loubes, J.-M. and Marteau, C. (2012). Non asymptotic minimax rates of testing in signal detection with heterogeneous variances. Electron. J. Stat. 6 91–122.
  • [24] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • [25] Lehmann, E. L., Romano, J. P. and Shaffer, J. P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108.
  • [26] Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660.
  • [27] Romano, J. P., Shaikh, A. and Wolf, M. (2011). Consonance and the closure method in multiple testing. Int. J. Biostat. 7 1–27.
  • [28] Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34 1850–1873.
  • [29] Romano, J. P. and Wolf, M. (2005). Exact and approximate stepdown methods for multiple hypothesis testing. J. Amer. Statist. Assoc. 100 94–108.
  • [30] Romano, J. P. and Wolf, M. (2007). Control of generalized error rates in multiple testing. Ann. Statist. 35 1378–1408.
  • [31] Romano, J. P. and Wolf, M. (2010). Balanced control of generalized error rates. Ann. Statist. 38 598–633.
  • [32] Schmidt-Hieber, J., Munk, A. and Dümbgen, L. (2013). Multiscale methods for shape constraints in deconvolution: Confidence statements for qualitative features. Ann. Statist. 41 1299–1328.
  • [33] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • [34] Spjøtvoll, E. (1972). On the optimality of some multiple comparison procedures. Ann. Math. Stat. 43 398–411.
  • [35] Spokoiny, V. G. (1996). Adaptive hypothesis testing using wavelets. Ann. Statist. 24 2477–2498.

Supplemental materials

  • Additional results and proofs for “Family-Wise Separation Rates for multiple testing”. This supplement contains additional results on the cumulative distribution functions and the proof of Lemma 1.