The Annals of Applied Statistics

Statistical tests for the intersection of independent lists of genes: Sensitivity, FDR, and type I error control

Loki Natarajan, Minya Pu, and Karen Messer

Full-text: Access has been disabled (more information)

Abstract

Public data repositories have enabled researchers to compare results across multiple genomic studies in order to replicate findings. A common approach is to first rank genes according to an hypothesis of interest within each study. Then, lists of the top-ranked genes within each study are compared across studies. Genes recaptured as highly ranked (usually above some threshold) in multiple studies are considered to be significant. However, this comparison strategy often remains informal, in that type I error and false discovery rate (FDR) are usually uncontrolled. In this paper, we formalize an inferential strategy for this kind of list-intersection discovery test. We show how to compute a $p$-value associated with a “recaptured” set of genes, using a closed-form Poisson approximation to the distribution of the size of the recaptured set. We investigate operating characteristics of the test as a function of the total number of studies considered, the rank threshold within each study, and the number of studies within which a gene must be recaptured to be declared significant. We investigate the trade off between FDR control and expected sensitivity (the expected proportion of true-positive genes identified as significant). We give practical guidance on how to design a bioinformatic list-intersection study with maximal expected sensitivity and prespecified control of type I error (at the set level) and false discovery rate (at the gene level). We show how optimal choice of parameters may depend on particular alternative hypothesis which might hold. We illustrate our methods using prostate cancer gene-expression datasets from the curated Oncomine database, and discuss the effects of dependence between genes on the test.

Article information

Source
Ann. Appl. Stat. Volume 6, Number 2 (2012), 521-541.

Dates
First available in Project Euclid: 11 June 2012

Permanent link to this document
http://projecteuclid.org/euclid.aoas/1339419606

Digital Object Identifier
doi:10.1214/11-AOAS510

Mathematical Reviews number (MathSciNet)
MR2976481

Zentralblatt MATH identifier
1243.62135

Keywords
Concordance validation gene-ranking meta-analysis rank based methods gene expression analysis microarray next generation sequencing cancer

Citation

Natarajan, Loki; Pu, Minya; Messer, Karen. Statistical tests for the intersection of independent lists of genes: Sensitivity, FDR, and type I error control. Ann. Appl. Stat. 6 (2012), no. 2, 521--541. doi:10.1214/11-AOAS510. http://projecteuclid.org/euclid.aoas/1339419606.


Export citation

References

  • Benjamini, Y. and Heller, R. (2008). Screening for partial conjunction hypotheses. Biometrics 64 1215–1222.
    Mathematical Reviews (MathSciNet): MR2522270
    Digital Object Identifier: doi:10.1111/j.1541-0420.2007.00984.x
  • Benjamini, Y., Heller, R. and Yekutieli, D. (2009). Selective inference in complex research. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4255–4271.
    Mathematical Reviews (MathSciNet): MR2546387
    Digital Object Identifier: doi:10.1098/rsta.2009.0127
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
    Mathematical Reviews (MathSciNet): MR1869245
    Zentralblatt MATH: 1041.62061
    Digital Object Identifier: doi:10.1214/aos/1013699998
    Project Euclid: euclid.aos/1013699998
  • Chan, S. K., Griffith, O. L. et al. (2008). Meta-analysis of colorectal cancer gene expression profiling studies identifies consistently reported candidate biomarkers. Cancer Epidemiology Biomarkers and Prevention 17 543–552.
  • Culhane, A. C., Schwarzl, T., Sultana, R., Picard, K. C., Picard, S. C., Lu, T. H., Franklin, K. R., French, S. J., Papenhausen, G., Correll, M. and Quackenbush, J. (2010). GeneSigDB—a curated database of gene expression signatures. Nucleic Acids Res. 38 D716–D725.
  • Deng, X. T., Xu, J. et al. (2008). Improving the power for detecting overlapping genes from multiple DNA microarray-derived gene lists. BMC Bioinformatics 9 8.
  • Dhanasekaran, S. M., Dash, A., Yu, N., Maine, I. P., Laxman, B., Tomlins, S. A., Creighton, C. N., Menon, A., Rubin, M. A. and Chinnaiyan, A. M. (2004). Molecular profiling of human prostate tissues: Insights into gene expression patterns of prostate development during puberty. FASEB N 9 243–245.
  • Efron, B. (2010). Correlated $z$-values and the accuracy of large-scale statistical estimates. J. Amer. Statist. Assoc. 105 1042–1055.
    Mathematical Reviews (MathSciNet): MR2752597
    Digital Object Identifier: doi:10.1198/jasa.2010.tm09129
  • Feller, W. (1957). An Introduction to Probability Theory and Its Applications. Vol. I, 2nd ed. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR88081
  • Futreal, P. A., Coin, L., Marshall, M., Down, T., Hubbard, T., Wooster, R., Rahman, N. and Stratton, M. R. (2004). A census of human cancer genes. Nat. Rev. Cancer 4 177–183.
  • Garrett-Mayer, E., Parmigiani, G., Zhong, X. G., Cope, L. and Gabrielson, E. (2008). Cross-study validation and combined analysis of gene expression microarray data. Biostatistics 9 333–354.
  • Glez-Pena, D., Gomez-Lopez, G., Pisano, D. G. and Fdez-Riverola, F. (2008). WhichGenes: A web-based tool for gathering, building, storing and exporting gene sets. Biostatistics 9 333–354.
  • Gupta, R. C. and Tao, H. (2010). A generalized correlated binomial distribution with application in multiple testing problems. Metrika 71 59–77.
    Mathematical Reviews (MathSciNet): MR2562928
    Digital Object Identifier: doi:10.1007/s00184-008-0202-7
  • Hong, F. X. and Breitling, R. (2008). A comparison of meta-analysis methods for detecting differentially expressed genes in microarray experiments. Bioinformatics 24 374–382.
  • Irizarry, R. A., Warren, D. et al. (2005). Multiple-laboratory comparison of microarray platforms. Nature Methods 5 345–349.
  • Jeffries, C. D., Ward, W. O. et al. (2009). Discovering collectively informative descriptors from high-throughput experiments. BMC Bioinformatics 10 9.
  • Lapointe, N., Li, C., Higgins, N. P., van de Rijn, M., Bair, E., Montgomery, K., Ferrari, M., Egevad, L., Rayford, W., Bergerheim, U., Ekman, P., DeMarzo, A. M., Tibshirani, R., Botstein, D., Brown, P. O., Brooks, N. D. and Pollack, N. R. (2004). Gene expression profiling identifies clinically relevant subtypes of prostate cancer. Proc. Natl. Acad. Sci. USA 101 811–816.
  • Lu, X., Gamst, A. and Xu, R. (2008). RDCurve: A nonparametric method to evaluate the stability of ranking procedures. Trans. Comput. Biol. Bioinf. 138 719–726.
  • Miller, B. G. and Stamatoyannopoulos, J. A. (2010). Integrative meta-analysis of differential gene expression in acute myeloid leukemia. PLoS One 5 13.
  • Natarajan, L., Pu, M. and Messer, K. (2011). Supplement to “Statistical tests for the intersection of independent lists of genes: Sensitivity, FDR, and type I error control.” DOI:10.1214/11-AOAS510SUPP.
    Mathematical Reviews (MathSciNet): MR2976481
    Zentralblatt MATH: 1243.62135
    Digital Object Identifier: doi:10.1214/11-AOAS510
  • Pyne, S., Futcher, B. and Skiena, S. (2006). Meta-analysis based on control of false discovery rate: Combining yeast ChIP-chip datasets. Bioinformatics 22 2516–2522.
  • Rhodes, D. R., Kalyana-Sundaram, S., Mahavisno, V. et al. (2007). Oncomine 3.0: Genes, pathways, and networks in a collection of 18,000 cancer gene expression profiles. Neoplasia 9 166–180.
  • Sun, W. and Cai, T. T. (2009). Large-scale multiple testing under dependence. J. R. Stat. Soc. Ser. B Stat. Methodol. 71 393–424.
    Mathematical Reviews (MathSciNet): MR2649603
    Zentralblatt MATH: 1248.62005
    Digital Object Identifier: doi:10.1111/j.1467-9868.2008.00694.x
  • Tiwari, G., Sakaue, H., Pollack, N. R. and Roth, R. A. (2003). Gene expression profiling in prostate cancer cells with Akt activation reveals Fra-1 as an Akt-inducible gene. Molecular Cancer Research 1 475–484.
  • Tomlins, S. A., Rhodes, D. A., Perner, S., Dhanasekaran, S. M., Mehra, R., Sun, X., Varambally, S., Cao, X., Tchinda, N., Kuefer, R. et al. (2005). Recurrent fusion of TMPRSS2 and ETS transcription factor gens in prostate cancer. Science 310 644–648.
  • Tomlins, S. A., Mehra, R., Rhodes, D. R., Cao, X., Wang, L., Dhanasekaran, S. M., Kalyana-Sundaram, S., Wei, N. T., Rubin, M. A., Pienta, K. N., Shah, R. B. and Chinnaiyan, A. M. (2006). Integrative molecular concept modeling of prostate cancer progression. Nature Genetics 39 41–51.
  • Yu, C. and Zelterman, D. (2002). Sums of dependent Bernoulli random variables and disease clustering. Statist. Probab. Lett. 57 363–373.
    Mathematical Reviews (MathSciNet): MR1914014
  • Zaykin, D. V., Zhivotovsky, L. A., Westfall, P. H. and Weir, B. S. (2002). Truncated product method for combining $\mathrm{P}$-values. Genet. Epidemiol. 22 170–185.

Supplemental materials

  • Supplementary material: Online supplement: Statistical tests for the intersection of independent lists of genes. Simulation studies and proofs are in the online supplement. In Section S1 we show by simulation that the Poisson approximation to the null distribution of the test statistic gives reliable $p$-values under a wide range of parameters, both for the independent case (Section S1.1) and under a range of moderate positive correlation structures (Section S1.2). We confirm that the Poisson approximation computed under assumed independence yields conservative $p$-values under examples of extreme positive correlation, as conjectured in the text (Section 6.1). In Section S2 we derive the alternative distribution of the test statistic for some useful special cases, using combinatorial results Feller (1957).
    Digital Object Identifier: doi:10.1214/11-AOAS510SUPP
    Supplemental files available for subscribers.

The Institute of Mathematical Statistics

The Annals of Applied Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more