Annals of Statistics

Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Nicolai Meinshausen and John Rice

Full-text: Open access

Enhanced PDF (293 KB)

Abstract

We consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses, focusing on the situation in which the proportion of false null hypotheses is very small. We propose a family of methods for establishing lower 100(1−α)% confidence bounds for this proportion, based on the empirical distribution of the p-values of the tests. Methods in this family are then compared in terms of ability to consistently estimate the proportion by letting α→0 as the number of hypothesis tests increases and the proportion decreases. This work is motivated by a signal detection problem that occurs in astronomy.

Article information

Source
Ann. Statist., Volume 34, Number 1 (2006), 373-393.

Dates
First available in Project Euclid: 2 May 2006

Permanent link to this document
https://projecteuclid.org/euclid.aos/1146576267

Digital Object Identifier
doi:10.1214/009053605000000741

Mathematical Reviews number (MathSciNet)
MR2275246

Zentralblatt MATH identifier
1091.62059

Subjects
Primary: 62H15: Hypothesis testing
Secondary: 62J15: Paired and multiple comparisons 62P35: Applications to physics

Keywords
Hypothesis testing multiple comparisons sparsity

Citation

Meinshausen, Nicolai; Rice, John. Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 (2006), no. 1, 373--393. doi:10.1214/009053605000000741. https://projecteuclid.org/euclid.aos/1146576267


Export citation

References

  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
    Mathematical Reviews (MathSciNet): MR1325392
    Zentralblatt MATH: 0809.62014
    Digital Object Identifier: doi:10.1111/j.2517-6161.1995.tb02031.x
  • Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple hypothesis testing with independent statistics. J. Educational and Behavioral Statistics 25 60–83.
  • Chen, W., Zhang, Z., King, S., Alcock, C., Byun, Y., Cook, K., Dave, R., Giammarco, J., Lee, T., Lehner, M., Liang, C., Lissauer, J., Marshall, S., de Pater, I., Porrata, R., Rice, J., Wang, A., Wang, S. and Wen, C. (2003). Fast CCD photometry in the Taiwanese–American Occultation Survey. Baltic Astronomy 12 568–573.
  • Csörgő, M. and Horvath, L. (1993). Weighted Approximations in Probability and Statistics. Wiley, Chichester.
    Mathematical Reviews (MathSciNet): MR1215046
  • Daniels, H. (1945). The statistical theory of the strength of bundles of threads. I. Proc. Roy. Soc. London Ser. A 183 405–435.
    Mathematical Reviews (MathSciNet): MR0012388
    Zentralblatt MATH: 0063.01035
    Digital Object Identifier: doi:10.1098/rspa.1945.0011
  • Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
    Mathematical Reviews (MathSciNet): MR2065195
    Zentralblatt MATH: 1092.62051
    Digital Object Identifier: doi:10.1214/009053604000000265
    Project Euclid: euclid.aos/1085408492
  • Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
    Mathematical Reviews (MathSciNet): MR2065197
    Zentralblatt MATH: 1092.62065
    Digital Object Identifier: doi:10.1214/009053604000000283
    Project Euclid: euclid.aos/1085408494
  • Liang, C.-L., Rice, J., de Pater, I., Alcock, C., Axelrod, T., Wang, A. and Marshall, S. (2004). Statistical methods for detecting stellar occultations by Kuiper Belt objects: The Taiwanese–American Occultation Survey. Statist. Sci. 19 265–274.
    Mathematical Reviews (MathSciNet): MR2146947
    Zentralblatt MATH: 1100.85501
    Digital Object Identifier: doi:10.1214/088342304000000378
    Project Euclid: euclid.ss/1105714162
  • Meinshausen, N. and Bühlmann, P. (2005). Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures. Biometrika 92 893–907.
  • Nettleton, D. and Hwang, J. (2003). Estimating the number of false null hypotheses when conducting many tests. Technical report, Dept. Statistics, Iowa State Univ.
  • Schweder, T. and Spjøtvoll, E. (1982). Plots of $p$-values to evaluate many tests simultaneously. Biometrika 69 493–502.
  • Shorack, G. and Wellner, J. (1986). Empirical Processes with Applications to Statistics. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR0838963
    Zentralblatt MATH: 1170.62365
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479–498.
    Mathematical Reviews (MathSciNet): MR1924302
    Zentralblatt MATH: 1090.62073
    Digital Object Identifier: doi:10.1111/1467-9868.00346

The Institute of Mathematical Statistics

Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more