• Bernoulli
  • Volume 16, Number 2 (2010), 418-434.

Strong approximations of level exceedences related to multiple hypothesis testing

Peter Hall and Qiying Wang

Full-text: Open access


Particularly in genomics, but also in other fields, it has become commonplace to undertake highly multiple Student’s t-tests based on relatively small sample sizes. The literature on this topic is continually expanding, but the main approaches used to control the family-wise error rate and false discovery rate are still based on the assumption that the tests are independent. The independence condition is known to be false at the level of the joint distributions of the test statistics, but that does not necessarily mean, for the small significance levels involved in highly multiple hypothesis testing, that the assumption leads to major errors. In this paper, we give conditions under which the assumption of independence is valid. Specifically, we derive a strong approximation that closely links the level exceedences of a dependent “studentized process” to those of a process of independent random variables. Via this connection, it can be seen that in high-dimensional, low sample-size cases, provided the sample size diverges faster than the logarithm of the number of tests, the assumption of independent t-tests is often justified.

Article information

Bernoulli, Volume 16, Number 2 (2010), 418-434.

First available in Project Euclid: 25 May 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

false discovery rate family-wise error rate genomic data large deviation probability moving average Poisson approximation Student’s t-statistic upper tail dependence upper tail independence


Hall, Peter; Wang, Qiying. Strong approximations of level exceedences related to multiple hypothesis testing. Bernoulli 16 (2010), no. 2, 418--434. doi:10.3150/09-BEJ220.

Export citation


  • Almirantis, Y. and Provata, A. (1999). Long- and short-range correlations in genome organization. J. Statist. Phys. 97 233–262.
  • Amaratunga, D. and Cabrera, J. (2004). Exploration and Analysis of DNA Microarray and Protein Array Data. New York: Wiley.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Benjamini, Y. and Yekutieli, D. (2005). False discovery rate-adjusted multiple confidence intervals for selected parameters (with discussion). J. Amer. Statist. Assoc. 100 71–93.
  • Bernhard, G., Klein, M. and Hommel, G. (2004). Global and multiple test procedures using ordered p-values – a review. Statist. Papers 45 1–14.
  • Blair, R.C., Troendle, J.F. and Beck, R.W. (1996). Control of familywise errors in multiple endpoint assessments via stepwise permutation tests. Statist. Med. 15 1107–1121.
  • Clarke, S. and Hall, P. (2009). Robustness of multiple testing procedures against dependence. Ann. Statist. 37 332–358.
  • Cui, X. and Churchill, G.A. (2003). Statistical tests for differential expression in cDNA microarray experiments. Genome Biol. 4 210 (electronic).
  • Dudoit, S., Shaffer, J.P. and Boldrick, J.C. (2003). Multiple hypothesis testing in microarray experiments. Statist. Sci. 18 73–103.
  • Efron, B. (2007a). Size, power and false discovery rates. Ann. Statist. 35 1351–1377.
  • Efron, B. (2007b). Correlation and large-scale simultaneous significance testing. J. Amer. Statist. Assoc. 102 93–103.
  • Efron, B. and Tibshirani, R. (2002). Empirical Bayes methods and false discovery rates for microarrays. Genetic Epidemiol. 23 70–86.
  • Falk, M. and Reiss, R.-D. (2001). Estimation of canonical dependence parameters in a class of bivariate peaks-over-threshold models. Statist. Probab. Lett. 52 233–242.
  • Fox, R.J. and Dimmic, M.W. (2006). A two-sample Bayesian t-test for microarray data. BMC Bioinformatics 7 126 (electronic).
  • Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800–802.
  • Hochberg, Y. and Benjamini, Y. (1990). More powerful procedures for multiple testing. Statist. Med. 9 811–818.
  • Hochberg, Y. and Tamhane, A.C. (1987). Multiple Comparison Procedures. New York: Wiley.
  • Holland, B. and Cheung, S.H. (2002). Familywise robustness criteria for multiple-comparison procedures. J. Roy. Statist. Soc. Ser. B 64 63–77.
  • Hommel, G. (1988). A comparison of two modified Bonferroni procedures. Biometrika 76 624–625.
  • Hu, Z. and Willsky, G.R. (2006). Utilization of two sample T-test statistics from redundant probe sets to evaluate different probe set algorithms in GeneChip studies. BMC Bioinformatics 7 12 (electronic).
  • Lehmann, E.L. and Romano, J.P. (2005). Testing Statistical Hypotheses, 3rd ed. New York: Springer.
  • Lehmann, E.L., Romano, J.P. and Shaffer, J.P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108.
  • Li, J. (2006). Modelling dependency between different lines of business with copulas. Research Paper 146, Centre for Actuarial Studies, Department of Economics, The University of Melbourne.
  • Liu, P. and Hwang, J.T.G. (2007). Quick calculation for sample size while controlling false discovery rate with application to microarray analysis. Bioinformatics 23 739–746.
  • Mansilla, R., De Castillo, N., Govezensky, T., Miramontes, P., José, M. and Coho, G. (2004). Long-range correlation in the whole human genome. Available at
  • Messer, P.W. and Arndt, P.F. (2006). CorGen-measuring and generating long-range correlations for DNA sequence analysis. Nucleic Acids Research 34 W692–W695.
  • Pawluk-Kolc, M., Zieba-Palus, J. and Parczewski, A. (2006). Application of false discovery rate procedure to pairwise comparisons of refractive index of glass fragments. Forensic Sci. Internat. 160 53–58.
  • Perneger, T.V. (1998). What’s wrong with Bonferroni adjustments. BMJ 316 1236–1238.
  • Pigeot, I. (2000). Basic concepts of multiple tests – A survey. Statist. Papers 41 3–36.
  • Qiu, X. and Yakovlev, A. (2006). Some comments on instability of false discovery rate estimation. J. Bioinform. Comput. Biol. 4 1057–1068.
  • Sarkar, S.K. (1998). Some probability inequalities for ordered MTP2 random variables: A proof of the Simes conjecture. Ann. Statist. 26 494–504.
  • Sarkar, S.K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
  • Sarkar, S.K. and Chang, C.K. (1997). The Simes method for multiple hypothesis testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601–1608.
  • Scheid, S. and Spang, R. (2005). Twilight; a Bioconductor package for estimating the local false discovery rate. Bioinformatics 21 2921–2922.
  • Schmidt, R. (2002). Tail dependence for elliptically contoured distributions. Math. Methods Oper. Res. 55 301–327.
  • Schmidt, R. and Stadtmüller, U. (2006). Nonparametric estimation of tail dependence. Scand. J. Statist. 33 307–335.
  • Schmidt, T. (2007). Coping with copulas. In Copulas – From Theory to Applications in Finance (J. Rank, ed.). Risk Books.
  • Sen, P.K. (1999). Some remarks on Simes-type multiple tests of significance. J. Statist. Plann. Inference 82 139–145.
  • Shaffer, J.P. (2005). Comment on “False discovery rate-adjusted multiple confidence intervals for selected parameters” by Y. Benjamini and D. Yekutieli. J. Amer. Statist. Assoc. 100 82–84.
  • Shao, Q.-M. (1999). A Cramér type large deviation result for Student’s t-statistic. J. Theoret. Probab. 12 385–398.
  • Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751–754.
  • van de Wiel, M.A. and Kim, K.I. (2007). Estimating the false discovery rate using nonparametric deconvolution. Biometrics 63 806–815.
  • Wang, Q. (2005). Limit Theorems for self-normalized large deviation. Electronic J. Probab. 10 1260–1285.
  • Wang, Q. and Hall, P. (2009). Relative errors in central limit theorem for Student’s t statistic, with applications. Statist. Sinica 19 343–354.
  • Wu, G. (1994). An empirical test of ordinal independence. J. Risk and Uncertainty 9 39–60.