Electronic Journal of Statistics

Coarse-to-fine multiple testing strategies

Kamel Lahouel, Donald Geman, and Laurent Younes

Full-text: Open access


We analyze control of the familywise error rate (FWER) in a multiple testing scenario with a great many null hypotheses about the distribution of a high-dimensional random variable among which only a very small fraction are false, or “active”. In order to improve power relative to conservative Bonferroni bounds, we explore a coarse-to-fine procedure adapted to a situation in which tests are partitioned into subsets, or “cells”, and active hypotheses tend to cluster within cells. We develop procedures for a non-parametric case based on generalized permutation testing and a linear Gaussian model, and demonstrate higher power than Bonferroni estimates at the same FWER when the active hypotheses do cluster. The main technical difficulty arises from the correlation between the test statistics at the individual and cell levels, which increases the likelihood of a hypothesis being falsely discovered when the cell that contains it is falsely discovered (survivorship bias). This requires sharp estimates of certain quadrant probabilities when a cell is inactive.

Article information

Electron. J. Statist., Volume 13, Number 1 (2019), 1292-1328.

Received: January 2018
First available in Project Euclid: 5 April 2019

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G10: Hypothesis testing
Secondary: 62G09: Resampling methods

Multiple testing FWER hierarchical testing permutation tests

Creative Commons Attribution 4.0 International License.


Lahouel, Kamel; Geman, Donald; Younes, Laurent. Coarse-to-fine multiple testing strategies. Electron. J. Statist. 13 (2019), no. 1, 1292--1328. doi:10.1214/19-EJS1536. https://projecteuclid.org/euclid.ejs/1554451243

Export citation


  • [1] David J Aldous. Exchangeability and related topics. In, École d’Été de Probabilités de Saint-Flour XIII, 1983, pages 1–198. Springer, 1985.
  • [2] Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al. Gene ontology: tool for the unification of biology., Nature Genetics, 25(1):25–29, 2000.
  • [3] David J Balding. A tutorial on statistical methods for population association studies., Nature Reviews Genetics, 7(10):781–791, 2006.
  • [4] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing., Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289–300, 1995.
  • [5] Yoav Benjamini, Abba M. Krieger, and Daniel Yekutieli. Adaptive linear step-up procedures that control the false discovery rate., Biometrika, 93(3):491, 2006.
  • [6] Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency., Annals of Statistics, pages 1165–1188, 2001.
  • [7] Gilles Blanchard and Donald Geman. Hierarchical testing designs for pattern recognition., Annals of Statistics, pages 1155–1202, 2005.
  • [8] Aiden Corvin, Nick Craddock, and Patrick F Sullivan. Genome-wide association studies: a primer., Psychological Medicine, 40(07) :1063–1077, 2010.
  • [9] Thorsten Dickhaus, Klaus Straßburger, Daniel Schunk, Carlos Morcillo-Suarez, Thomas Illig, and Arcadi Navarro. How to analyze many contingency tables simultaneously in genetic association studies., Statistical applications in genetics and molecular biology, 11(4), 2012.
  • [10] Brooke L Fridley. Bayesian variable and model selection methods for genetic association studies., Genetic Epidemiology, 33(1):27–37, 2009.
  • [11] K. Ruben Gabriel. Simultaneous test procedures–some theory of multiple comparisons., The Annals of Mathematical Statistics, 40(1):pp. 224–250, 1969.
  • [12] Phillip Good., Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Science & Business Media, 2013.
  • [13] Yosef Hochberg. A sharper Bonferroni procedure for multiple tests of significance., Biometrika, 75(4):800, 1988.
  • [14] Sture Holm. A simple sequentially rejective multiple test procedure., Scandinavian Journal of Statistics, 6(2):65–70, 1979.
  • [15] Hailiang Huang, Pritam Chanda, Alvaro Alonso, Joel S Bader, and Dan E Arking. Gene-based tests of association., PLoS Genetics, 7(7): e1002177, 2011.
  • [16] Nicolai Meinshausen. Hierarchical testing of variable importance., Biometrika, 95(2):265–278, 2008.
  • [17] Shaun Purcell, Benjamin Neale, Kathe Todd-Brown, Lori Thomas, Manuel AR Ferreira, David Bender, Julian Maller, Pamela Sklar, Paul IW De Bakker, Mark J Daly, et al. Plink: a tool set for whole-genome association and population-based linkage analyses., The American Journal of Human Genetics, 81(3):559–575, 2007.