The Annals of Applied Statistics

Simultaneous inference: When should hypothesis testing problems be combined?

Bradley Efron

Full-text: Access has been disabled (more information)

Abstract

Modern statisticians are often presented with hundreds or thousands of hypothesis testing problems to evaluate at the same time, generated from new scientific technologies such as microarrays, medical and satellite imaging devices, or flow cytometry counters. The relevant statistical literature tends to begin with the tacit assumption that a single combined analysis, for instance, a False Discovery Rate assessment, should be applied to the entire set of problems at hand. This can be a dangerous assumption, as the examples in the paper show, leading to overly conservative or overly liberal conclusions within any particular subclass of the cases. A simple Bayesian theory yields a succinct description of the effects of separation or combination on false discovery rate analyses. The theory allows efficient testing within small subclasses, and has applications to “enrichment,” the detection of multi-case effects.

Article information

Source
Ann. Appl. Stat. Volume 2, Number 1 (2008), 197-223.

Dates
First available in Project Euclid: 24 March 2008

Permanent link to this document
http://projecteuclid.org/euclid.aoas/1206367818

Digital Object Identifier
doi:10.1214/07-AOAS141

Mathematical Reviews number (MathSciNet)
MR2415600

Zentralblatt MATH identifier
1137.62010

Keywords
False discovery rates separate-class model enrichment

Citation

Efron, Bradley. Simultaneous inference: When should hypothesis testing problems be combined?. Ann. Appl. Stat. 2 (2008), no. 1, 197--223. doi:10.1214/07-AOAS141. http://projecteuclid.org/euclid.aoas/1206367818.


Export citation

References

  • Benjamini, Y. and Hochberg (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing., J. Roy. Statist. Soc. Ser. B 57 289–300.
    Mathematical Reviews (MathSciNet): MR1325392
    JSTOR: links.jstor.org
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate under dependency., Ann. Statist. 29 1165–1188.
    Mathematical Reviews (MathSciNet): MR1869245
    Zentralblatt MATH: 1041.62061
    Digital Object Identifier: doi:10.1214/aos/1013699998
    Project Euclid: euclid.aos/1013699998
  • Efron, B. (2004a). Large-scale simultaneous hypothesis testing: The choice of a null hypothesis., J. Amer. Statist. Assoc. 99 96–104.
    Mathematical Reviews (MathSciNet): MR2054289
    Zentralblatt MATH: 1089.62502
    Digital Object Identifier: doi:10.1198/016214504000000089
  • Efron, B. (2004b). The estimation of prediction error: Covariance penalties and cross-validation (with discussion)., J. Amer. Statist. Assoc. 99 619–642.
    Mathematical Reviews (MathSciNet): MR2090899
    Zentralblatt MATH: 1117.62324
    Digital Object Identifier: doi:10.1198/016214504000000692
  • Efron, B. (2005). Local false discovery rates. Available at, http://www-stat.stanford.edu/~brad/papers/False.pdf.
  • Efron, B. (2007a). Correlation and large-scale significance testing., J. Amer. Statist. Assoc. 102 93–103.
    Mathematical Reviews (MathSciNet): MR2293302
    Zentralblatt MATH: 05191552
    Digital Object Identifier: doi:10.1198/016214506000001211
  • Efron, B. (2007b). Size, power and false discovery rates., Ann. Statist. 35 1351–1377.
    Mathematical Reviews (MathSciNet): MR2351089
    Zentralblatt MATH: 1123.62008
    Digital Object Identifier: doi:10.1214/009053606000001460
    Project Euclid: euclid.aos/1188405614
  • Efron, B. and Tibshirani, R. (2007). On testing the significance of sets of genes., Ann. Appl. Statist. 1 107–129.
    Mathematical Reviews (MathSciNet): MR2393843
    Zentralblatt MATH: 1129.62102
    Digital Object Identifier: doi:10.1214/07-AOAS101
    Project Euclid: euclid.aoas/1183143731
  • Ferkinstad, E., Frigessi, A., Thorleifsson, G. and Kong, A. (2007). Covariate-modulated false discovery rates. Available at, http://folk.uio.no/egilfe/cmfdr-ims.pdf.
  • Genovese, C., Roeder, K. and Wasserman, L. (2006). False discovery control with, p-value weighting. Biometrika 93 509–524.
    Mathematical Reviews (MathSciNet): MR2261439
    Zentralblatt MATH: 1108.62070
    Digital Object Identifier: doi:10.1093/biomet/93.3.509
  • Lehmann, E. and Romano, J. (2005)., Testing Statistical Hypotheses, 3rd ed. Springer, New York.
    Mathematical Reviews (MathSciNet): MR2135927
  • Newton, M., Quintana, F., den Boon, J., Sengupta, S. and Ahlquist, P. (2007). Random-set methods identify distinct aspects of the enrichment signal in gene-set analysis., Ann. Appl. Statist. 1 85–106.
    Mathematical Reviews (MathSciNet): MR2393842
    Zentralblatt MATH: 1129.62103
    Digital Object Identifier: doi:10.1214/07-AOAS104
    Project Euclid: euclid.aoas/1183143730
  • Schwartzman, A., Dougherty, R. F. and Taylor, J. E. (2005). Cross-subject comparison of principal diffusion direction maps., Magn. Reson. Med. 53 1423–1431.
  • Smyth, G. (2004). Linear models and empirical Bayes methods for assessing differential expression in microarray experiments., Stat. Appl. Genet. Mol. Biol. 3 (1). Available at http://www.bepress.com/sagmb/vol3/iss1/art3.
    Mathematical Reviews (MathSciNet): MR2101454
    Zentralblatt MATH: 1038.62110
    Digital Object Identifier: doi:10.2202/1544-6115.1027
  • Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S. and Mesirov, J. P. (2005). Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles., Proc. Natl. Acad. Sci. 102 15545–15550.

The Institute of Mathematical Statistics

The Annals of Applied Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more