## The Annals of Statistics

### Family-Wise Separation Rates for multiple testing

#### Abstract

Starting from a parallel between some minimax adaptive tests of a single null hypothesis, based on aggregation approaches, and some tests of multiple hypotheses, we propose a new second kind error-related evaluation criterion, as the core of an emergent minimax theory for multiple tests. Aggregation-based tests, proposed for instance by Baraud [Bernoulli 8 (2002) 577–606], Baraud, Huet and Laurent [Ann. Statist. 31 (2003) 225–251] or Fromont and Laurent [Ann. Statist. 34 (2006) 680–720], are justified through their first kind error rate, which is controlled by the prescribed level on the one hand, and through their separation rates over various classes of alternatives, rates which are minimax on the other hand. We show that some of these tests can be viewed as the first steps of classical step-down multiple testing procedures, and accordingly be evaluated from the multiple testing point of view also, through a control of their Family-Wise Error Rate (FWER). Conversely, many multiple testing procedures, from the historical ones of Bonferroni and Holm, to more recent ones like min-$p$ procedures or randomized procedures such as the ones proposed by Romano and Wolf [J. Amer. Statist. Assoc. 100 (2005) 94–108], can be investigated from the minimax adaptive testing point of view. To this end, we extend the notion of separation rate to the multiple testing field, by defining the weak Family-Wise Separation Rate and its stronger counterpart, the Family-Wise Separation Rate (FWSR). As for nonparametric tests of a single null hypothesis, we prove that these new concepts allow an accurate analysis of the second kind error of a multiple testing procedure, leading to clear definitions of minimax and minimax adaptive multiple tests. Some illustrations in classical Gaussian frameworks corroborate several expected results under particular conditions on the tested hypotheses, but also lead to new questions and perspectives.

#### Article information

Source
Ann. Statist., Volume 44, Number 6 (2016), 2533-2563.

Dates
Revised: November 2015
First available in Project Euclid: 23 November 2016

https://projecteuclid.org/euclid.aos/1479891627

Digital Object Identifier
doi:10.1214/15-AOS1418

Mathematical Reviews number (MathSciNet)
MR3576553

Zentralblatt MATH identifier
1360.62214

#### Citation

Fromont, Magalie; Lerasle, Matthieu; Reynaud-Bouret, Patricia. Family-Wise Separation Rates for multiple testing. Ann. Statist. 44 (2016), no. 6, 2533--2563. doi:10.1214/15-AOS1418. https://projecteuclid.org/euclid.aos/1479891627

#### References

• [1] Baraud, Y. (2002). Non-asymptotic minimax rates of testing in signal detection. Bernoulli 8 577–606.
• [2] Baraud, Y., Huet, S. and Laurent, B. (2003). Adaptive tests of linear hypotheses by model selection. Ann. Statist. 31 225–251.
• [3] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 289–300.
• [4] Birgé, L. (2001). An alternative point of view on Lepski’s method. In State of the Art in Probability and Statistics (Leiden, 1999). Institute of Mathematical Statistics Lecture Notes—Monograph Series 36 113–133. IMS, Beachwood, OH.
• [5] Collier, O. and Dalalyan, A. S. (2015). Curve registration by nonparametric goodness-of-fit testing. J. Statist. Plann. Inference 162 20–42.
• [6] Dudoit, S. and van Der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genomics. Springer, New York.
• [7] Dümbgen, L. and Spokoiny, V. G. (2001). Multiscale testing of qualitative hypotheses. Ann. Statist. 29 124–152.
• [8] Dümbgen, L. and Walther, G. (2008). Multiscale inference about a density. Ann. Statist. 36 1758–1785.
• [9] Fromont, M. and Laurent, B. (2006). Adaptive goodness-of-fit tests in a density model. Ann. Statist. 34 680–720.
• [10] Fromont, M., Laurent, B., Lerasle, M. and Reynaud-Bouret, P. (2012). Kernels based tests with non-asymptotic bootstrap approaches for two-sample problems. In Journal of Machine Learning Research: Workshop and Conference Proceedings, COLT 2012 23 23.1–23.22. Edimbourg, UK.
• [11] Fromont, M., Laurent, B. and Reynaud-Bouret, P. (2011). Adaptive tests of homogeneity for a Poisson process. Ann. Inst. Henri Poincaré Probab. Stat. 47 176–213.
• [12] Fromont, M., Laurent, B. and Reynaud-Bouret, P. (2013). The two-sample problem for Poisson processes: Adaptive tests with a nonasymptotic wild bootstrap approach. Ann. Statist. 41 1431–1461.
• [13] Fromont, M., Lerasle, M. and Reynaud-Bouret, P. (2016). Supplement to “Family-Wise Separation Rates for multiple testing.” DOI:10.1214/15-AOS1418SUPP.
• [14] Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
• [15] Goeman, J. J. and Solari, A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810.
• [16] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 65–70.
• [17] Hommel, G. and Hoffmann, T. (1988). Controlled uncertainty. In Multiple Hypothesenprüfung/Multiple Hypotheses Testing 154–161. Springer, Berlin.
• [18] Inglot, T. (2010). Inequalities for quantiles of the chi-square distribution. Probab. Math. Statist. 30 339–351.
• [19] Ingster, Y. I. (1993). Asymptotically minimax hypothesis testing for nonparametric alternatives. I, II, III. Math. Methods Statist. 2 85–114, 171–189, 249–268.
• [20] Ingster, Y. I. (2000). Adaptive chi-square tests. J. Math. Sci. 99 1110–1119.
• [21] Korn, E. L., Troendle, J. F., McShane, L. M. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data. J. Statist. Plann. Inference 124 379–398.
• [22] Laurent, B., Loubes, J.-M. and Marteau, C. (2012). Non asymptotic minimax rates of testing in signal detection with heterogeneous variances. Available at http://arxiv.org/abs/0912.2423v1.
• [23] Laurent, B., Loubes, J.-M. and Marteau, C. (2012). Non asymptotic minimax rates of testing in signal detection with heterogeneous variances. Electron. J. Stat. 6 91–122.
• [24] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
• [25] Lehmann, E. L., Romano, J. P. and Shaffer, J. P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108.
• [26] Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660.
• [27] Romano, J. P., Shaikh, A. and Wolf, M. (2011). Consonance and the closure method in multiple testing. Int. J. Biostat. 7 1–27.
• [28] Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34 1850–1873.
• [29] Romano, J. P. and Wolf, M. (2005). Exact and approximate stepdown methods for multiple hypothesis testing. J. Amer. Statist. Assoc. 100 94–108.
• [30] Romano, J. P. and Wolf, M. (2007). Control of generalized error rates in multiple testing. Ann. Statist. 35 1378–1408.
• [31] Romano, J. P. and Wolf, M. (2010). Balanced control of generalized error rates. Ann. Statist. 38 598–633.
• [32] Schmidt-Hieber, J., Munk, A. and Dümbgen, L. (2013). Multiscale methods for shape constraints in deconvolution: Confidence statements for qualitative features. Ann. Statist. 41 1299–1328.
• [33] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
• [34] Spjøtvoll, E. (1972). On the optimality of some multiple comparison procedures. Ann. Math. Stat. 43 398–411.
• [35] Spokoiny, V. G. (1996). Adaptive hypothesis testing using wavelets. Ann. Statist. 24 2477–2498.

#### Supplemental materials

• Additional results and proofs for “Family-Wise Separation Rates for multiple testing”. This supplement contains additional results on the cumulative distribution functions and the proof of Lemma 1.