The Annals of Statistics

Some nonasymptotic results on resampling in high dimension, II: Multiple tests

Sylvain Arlot, Gilles Blanchard, and Etienne Roquain

Full-text: Open access


In the context of correlated multiple tests, we aim to nonasymptotically control the family-wise error rate (FWER) using resampling-type procedures. We observe repeated realizations of a Gaussian random vector in possibly high dimension and with an unknown covariance matrix, and consider the one- and two-sided multiple testing problem for the mean values of its coordinates. We address this problem by using the confidence regions developed in the companion paper [Ann. Statist. (2009), to appear], which lead directly to single-step procedures; these can then be improved using step-down algorithms, following an established general methodology laid down by Romano and Wolf [J. Amer. Statist. Assoc. 100 (2005) 94–108]. This gives rise to several different procedures, whose performances are compared using simulated data.

Article information

Ann. Statist. Volume 38, Number 1 (2010), 83-99.

First available in Project Euclid: 31 December 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G10: Hypothesis testing
Secondary: 62G09: Resampling methods

Family-wise error multiple testing high-dimensional data nonasymptotic error control resampling resampled quantile


Arlot, Sylvain; Blanchard, Gilles; Roquain, Etienne. Some nonasymptotic results on resampling in high dimension, II: Multiple tests. Ann. Statist. 38 (2010), no. 1, 83--99. doi:10.1214/08-AOS668.

Export citation


  • [1] Arlot, S., Blanchard, G. and Roquain, É. (2010). Some nonasymptotic results on resampling in high dimension, I: Confidence regions. Ann. Statist. 38 51–82.
  • [2] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • [3] Blanchard, G. and Roquain, É. (2008). Two simple sufficient conditions for FDR control. Electron. J. Stat. 2 963–992.
  • [4] Blanchard, G. and Roquain, É. (2009). Adaptive FDR control under independence and dependence. J. Mach. Learn. Res. To appear.
  • [5] Darvas, F., Rautiainen, M., Pantazis, D., Baillet, S., Benali, H., Mosher, J., Garnero, L. and Leahy, R. (2005). Investigations of dipole localization accuracy in MEG using the bootstrap. NeuroImage 25 355–368.
  • [6] Durot, C. and Rozenholc, Y. (2006). An adaptive test for zero mean. Math. Methods Statist. 15 26–60.
  • [7] Fisher, R. A. (1935). The Design of Experiments. Oliver and Boyd, Edinburgh.
  • [8] Ge, Y., Dudoit, S. and Speed, T. P. (2003). Resampling-based multiple testing for microarray data analysis. Test 12 1–77.
  • [9] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70.
  • [10] Jerbi, K., Lachaux, J.-P., N’Diaye, K., Pantazis, D., Leahy, R. M., Garnero, L. and Baillet, S. (2007). Coherent neural representation of hand speed in humans revealed by MEG imaging. PNAS 104 7676–7681.
  • [11] Pantazis, D., Nichols, T. E., Baillet, S. and Leahy, R. M. (2005). A comparison of random field theory and permutation methods for statistical analysis of MEG data. NeuroImage 25 383–394.
  • [12] Pacifico, M. P., Genovese, I., Verdinelli, I. and Wasserman, L. (2004). False discovery control for random fields. J. Amer. Statist. Assoc. 99 1002–1014.
  • [13] Pollard, K. S. and van der Laan, M. J. (2004). Choice of a null distribution in resampling-based multiple testing. J. Statist. Plann. Inference 125 85–100.
  • [14] Romano, J. P. (1989). Bootstrap and randomization tests of some nonparametric hypotheses. Ann. Statist. 17 141–159.
  • [15] Romano, J. P. (1990). On the behavior of randomization tests without a group invariance assumption. J. Amer. Statist. Assoc. 85 686–692.
  • [16] Romano, J. P. and Wolf, M. (2005). Exact and approximate stepdown methods for multiple hypothesis testing. J. Amer. Statist. Assoc. 100 94–108.
  • [17] Romano, J. P. and Wolf, M. (2007). Control of generalized error rates in multiple testing. Ann. Statist. 35 1378–1408.
  • [18] Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. Wiley, New York.
  • [19] Yekutieli, D. and Benjamini, Y. (1999). Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Statist. Plann. Inference 82 171–196.