The Annals of Statistics

Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Abstract

We consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses, focusing on the situation in which the proportion of false null hypotheses is very small. We propose a family of methods for establishing lower 100(1−α)% confidence bounds for this proportion, based on the empirical distribution of the p-values of the tests. Methods in this family are then compared in terms of ability to consistently estimate the proportion by letting α→0 as the number of hypothesis tests increases and the proportion decreases. This work is motivated by a signal detection problem that occurs in astronomy.

Article information

Source
Ann. Statist., Volume 34, Number 1 (2006), 373-393.

Dates
First available in Project Euclid: 2 May 2006

https://projecteuclid.org/euclid.aos/1146576267

Digital Object Identifier
doi:10.1214/009053605000000741

Mathematical Reviews number (MathSciNet)
MR2275246

Zentralblatt MATH identifier
1091.62059

Subjects
Primary: 62H15: Hypothesis testing
Secondary: 62J15: Paired and multiple comparisons 62P35: Applications to physics

Citation

Meinshausen, Nicolai; Rice, John. Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 (2006), no. 1, 373--393. doi:10.1214/009053605000000741. https://projecteuclid.org/euclid.aos/1146576267

References

• Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289--300.
• Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple hypothesis testing with independent statistics. J. Educational and Behavioral Statistics 25 60--83.
• Chen, W., Zhang, Z., King, S., Alcock, C., Byun, Y., Cook, K., Dave, R., Giammarco, J., Lee, T., Lehner, M., Liang, C., Lissauer, J., Marshall, S., de Pater, I., Porrata, R., Rice, J., Wang, A., Wang, S. and Wen, C. (2003). Fast CCD photometry in the Taiwanese--American Occultation Survey. Baltic Astronomy 12 568--573.
• Csörgő, M. and Horvath, L. (1993). Weighted Approximations in Probability and Statistics. Wiley, Chichester.
• Daniels, H. (1945). The statistical theory of the strength of bundles of threads. I. Proc. Roy. Soc. London Ser. A 183 405--435.
• Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962--994.
• Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035--1061.
• Liang, C.-L., Rice, J., de Pater, I., Alcock, C., Axelrod, T., Wang, A. and Marshall, S. (2004). Statistical methods for detecting stellar occultations by Kuiper Belt objects: The Taiwanese--American Occultation Survey. Statist. Sci. 19 265--274.
• Meinshausen, N. and Bühlmann, P. (2005). Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures. Biometrika 92 893--907.
• Nettleton, D. and Hwang, J. (2003). Estimating the number of false null hypotheses when conducting many tests. Technical report, Dept. Statistics, Iowa State Univ.
• Schweder, T. and Spjøtvoll, E. (1982). Plots of $p$-values to evaluate many tests simultaneously. Biometrika 69 493--502.
• Shorack, G. and Wellner, J. (1986). Empirical Processes with Applications to Statistics. Wiley, New York.
• Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479--498.