The Annals of Applied Statistics

Multiple testing under dependence via graphical models

Jie Liu, Chunming Zhang, and David Page

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Large-scale multiple testing tasks often exhibit dependence. Leveraging the dependence between individual tests is still one challenging and important problem in statistics. With recent advances in graphical models, it is feasible to use them to capture the dependence among multiple hypotheses. We propose a multiple testing procedure which is based on a Markov-random-field-coupled mixture model. The underlying true states of hypotheses are represented by a latent binary Markov random field, and the observed test statistics appear as the coupled mixture variables. The model can be learned by a novel EM algorithm. The next step is to infer the posterior probability that each hypothesis is null (termed local index of significance), and the false discovery rate can be controlled accordingly. We also provide a semiparametric variation of the graphical model which is useful in the situation where $f_{1}$ (the density function of the test statistic under the alternative hypothesis) is heterogeneous among multiple hypotheses. This semiparametric approach exactly generalizes the local FDR procedure [J. Amer. Statist. Assoc. 96 (2001) 1151–1160] and connects with the BH procedure [J. Roy. Statist. Soc. Ser. B 57 (1995) 289–300]. Simulations show that the numerical performance of multiple testing can be improved substantially by using our procedure. We apply the procedure to a real-world genome-wide association study on breast cancer, and we identify several SNPs with strong association evidence.

Article information

Ann. Appl. Stat., Volume 10, Number 3 (2016), 1699-1724.

Received: September 2014
Revised: May 2016
First available in Project Euclid: 28 September 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Multiple testing under dependence graphical models Markov random field local index of significance genome-wide association study


Liu, Jie; Zhang, Chunming; Page, David. Multiple testing under dependence via graphical models. Ann. Appl. Stat. 10 (2016), no. 3, 1699--1724. doi:10.1214/16-AOAS956.

Export citation


  • Baum, L. E., Petrie, T., Soules, G. and Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. Ann. Math. Stat. 41 164–171.
  • Benjamini, Y. and Heller, R. (2007). False discovery rates for spatial signals. J. Amer. Statist. Assoc. 102 1272–1281.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25 60–83.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Besag, J. (1975). Statistical analysis of non-lattice data. J. R. Stat. Soc., Ser. D Stat. 24 179–195.
  • Blanchard, G. and Roquain, É. (2009). Adaptive false discovery rate control under independence and dependence. J. Mach. Learn. Res. 10 2837–2871.
  • Celeux, G., Forbes, F. and Peyrard, N. (2003). EM procedures using mean field-like approximations for Markov model-based image segmentation. Pattern Recogn. 36 131–144.
  • Crouse, M. S., Nowak, R. D. and Baraniuk, R. G. (1998). Wavelet-based statistical signal processing using hidden Markov models. IEEE Trans. Signal Process. 46 886–902.
  • Efron, B. (2007). Correlation and large-scale simultaneous significance testing. J. Amer. Statist. Assoc. 102 93–103.
  • Efron, B. (2010). Large-Scale Inference. Institute of Mathematical Statistics (IMS) Monographs 1. Cambridge Univ. Press, Cambridge.
  • Efron, B. and Tibshirani, R. (2002). Empirical Bayes methods and false discovery rates for microarrays. Genet. Epidemiol. 23 70–86.
  • Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc. 96 1151–1160.
  • Epanechnikov, V. A. (1969). Non-parametric estimation of a multivariate probability density. Theory Probab. Appl. 14 153–158.
  • Fan, J., Han, X. and Gu, W. (2012). Control of the false discovery rate under arbitrary covariance dependence. J. Amer. Statist. Assoc. 107 1019–1045.
  • Farcomeni, A. (2007). Some results on the control of the false discovery rate under dependence. Scand. J. Stat. 34 275–297.
  • Finner, H. and Roters, M. (2002). Multiple hypotheses testing and expected number of type I errors. Ann. Statist. 30 220–238.
  • Friguet, C., Kloareg, M. and Causeur, D. (2009). A factor model approach to multiple testing under dependence. J. Amer. Statist. Assoc. 104 1406–1415.
  • Gelfand, A. E. and Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities. J. Amer. Statist. Assoc. 85 398–409.
  • Genovese, C. R., Roeder, K. and Wasserman, L. (2006). False discovery control with $p$-value weighting. Biometrika 93 509–524.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Ser. B. Stat. Methodol. 64 499–517.
  • Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • Guedj, M., Robin, S., Celisse, A. and Nuel, G. (2009). Kerfdr: A semi-parametric kernel-based approach to local false discovery rate estimation. BMC Bioinformatics 10 84.
  • Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Comput. 14 1771–1800.
  • Hunter, D. J., Kraft, P., Jacobs, K. B., Cox, D. G., Yeager, M., Hankinson, S. E., Wacholder, S., Wang, Z., Welch, R., Hutchinson, A., Wang, J., Yu, K., Chatterjee, N., Orr, N., Willett, W. C., Colditz, G. A., Ziegler, R. G., Berg, C. D., Buys, S. S., Mccarty, C. A., Feigelson, H. S., Calle, E. E., Thun, M. J., Hayes, R. B., Tucker, M., Gerhard, D. S., Fraumeni, J. F., Hoover, R. N., Thomas, G. and Chanock, S. J. (2007). A genome-wide association study identifies alleles in FGFR2 associated with risk of sporadic postmenopausal breast cancer. Nat. Genet. 39 870–874.
  • International HapMap Consortium (2003). The international HapMap project. Nature 426 789–796.
  • Jordan, M. I., Ghahramani, Z., Jaakkola, T. and Saul, L. K. (1999). An introduction to variational methods for graphical models. Mach. Learn. 37 183–233.
  • Kim, D. and Zhang, C. (2014). Adaptive linear step-up multiple testing procedure with the bias-reduced estimator. Statist. Probab. Lett. 87 31–39.
  • Kschischang, F. R., Frey, B. J. and Loeliger, H.-A. (2001). Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory 47 498–519.
  • Lauritzen, S. L. and Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems. J. Roy. Statist. Soc. Ser. B 50 157–224.
  • Leek, J. T. and Storey, J. D. (2008). A general framework for multiple testing dependence. Proc. Natl. Acad. Sci. USA 105 18718–18723.
  • Liang, K. and Nettleton, D. (2012). Adaptive and dynamic adaptive procedures for false discovery rate control and estimation. J. R. Stat. Soc. Ser. B. Stat. Methodol. 74 163–182.
  • Manolio, T. A., Collins, F. S., Cox, N. J., Goldstein, D. B., Hindorff, L. A., Hunter, D. J., McCarthy, M. I., Ramos, E. M., Cardon, L. R., Chakravarti, A. et al. (2009). Finding the missing heritability of complex diseases. Nature 461 747–753.
  • McCarty, C. A., Wilke, R. A., Giampietro, P. F., Wesbrook, S. D. and Caldwell, M. D. (2005). Marshfield clinic personalized medicine research project (PMRP): Design, methods and recruitment for a large population-based biobank. Personalized Medicine 2 49–79.
  • McCarty, C. A., Chisholm, R. L., Chute, C. G., Kullo, I. J., Jarvik, G. P., Larson, E. B., Li, R., Masys, D. R., Ritchie, M. D., Roden, D. M. et al. (2011). The eMERGE network: A consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Medical Genomics 4 13.
  • Murphy, K. P., Weiss, Y. and Jordan, M. I. (1999). Loopy belief propagation for approximate inference: An empirical study. In UAI 467–475.
  • Nguyen, V. H. and Matias, C. (2014). Nonparametric estimation of the density of the alternative hypothesis in a multiple testing setup. Application to local false discovery rate estimation. ESAIM Probab. Stat. 18 584–612.
  • Owen, A. B. (2005). Variance of the number of false discoveries. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 411–426.
  • Robbins, H. (1951). Asymptotically subminimax solutions of compound statistical decision problems. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950 131–148. Univ. California Press, Berkeley.
  • Robbins, H. (1956). An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 19541955, Vol. i 157–163. Univ. California Press, Berkeley.
  • Robin, S., Bar-Hen, A., Daudin, J.-J. and Pierre, L. (2007). A semi-parametric approach for mixture models: Application to local false discovery rate estimation. Comput. Statist. Data Anal. 51 5483–5493.
  • Romano, J. P., Shaikh, A. M. and Wolf, M. (2008). Control of the false discovery rate under dependence using the bootstrap and subsampling. TEST 17 417–442.
  • Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. Ann. Math. Stat. 27 832–837.
  • Sarkar, S. K. (2006). False discovery and false nondiscovery rates in single-step multiple testing procedures. Ann. Statist. 34 394–415.
  • Satrom, P., Biesinger, J., Li, S. M., Smith, D., Thomas, L. F., Majzoub, K., Rivas, G. E., Alluin, J., Rossi, J. J., Krontiris, T. G., Weitzel, J., Daly, M. B., Benson, A. B., Kirkwood, J. M., ODwyer, P. J., Sutphen, R., Stewart, J. A., Johnson, D. and Larson, G. P. (2009). A risk variant in an miR-125b binding site in BMPR1B is associated with breast cancer pathogenesis. Cancer Res. 69 7459–7465.
  • Schraudolph, N. N. (2010). Polynomial-time exact inference in NP-hard binary MRFs via reweighted perfect matching. In AISTATS.
  • Schraudolph, N. N. and Kamenetsky, D. (2009). Efficient exact inference in planar Ising models. In NIPS.
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B. Stat. Methodol. 64 479–498.
  • Storey, J. D. (2003). The positive false discovery rate: A Bayesian interpretation and the $q$-value. Ann. Statist. 31 2013–2035.
  • Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. R. Stat. Soc. Ser. B. Stat. Methodol. 66 187–205.
  • Sun, W. and Cai, T. T. (2007). Oracle and adaptive compound decision rules for false discovery rate control. J. Amer. Statist. Assoc. 102 901–912.
  • Sun, W. and Cai, T. T. (2009). Large-scale multiple testing under dependence. J. R. Stat. Soc. Ser. B. Stat. Methodol. 71 393–424.
  • Svenson, U., Nordfjäll, K., Stegmayr, B., Manjer, J., Nilsson, P., Tavelin, B., Henriksson, R., Lenner, P. and Roos, G. (2008). Breast cancer survival is associated with telomere length in peripheral blood cells. Cancer Res. 68 3618–3623.
  • Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In ICML 1064–1071.
  • Wainwright, M. J., Jaakkola, T. S. and Willsky, A. S. (2003a). Tree-reweighted belief propagation algorithms and approximate ML estimation via pseudo-moment matching. In AISTATS.
  • Wainwright, M. J., Jaakkola, T. S. and Willsky, A. S. (2003b). Tree-based reparameterization framework for analysis of sum-product and related algorithms. IEEE Trans. Inform. Theory 49 1120–1146.
  • Wei, Z., Sun, W., Wang, K. and Hakonarson, H. (2009). Multiple testing in genome-wide association studies via hidden Markov models. Bioinformatics 25 2802–2808.
  • Weiss, Y. (2000). Correctness of local probability propagation in graphical models with loops. Neural Comput. 12 1–41.
  • Welling, M. and Sutton, C. (2005). Learning in Markov random fields with contrastive free energies. In AISTATS.
  • Wu, W. B. (2008). On false discovery control under dependence. Ann. Statist. 36 364–380.
  • Xiao, J., Zhu, W. and Guo, J. (2013). Large-scale multiple testing in genome-wide association studies via region-specific hidden Markov models. BMC Bioinformatics 14 282.
  • Yedidia, J. S., Freeman, W. T. and Weiss, Y. (2000). Generalized belief propagation. In NIPS 689–695. MIT Press, Cambridge.
  • Yekutieli, D. and Benjamini, Y. (1999). Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Statist. Plann. Inference 82 171–196.
  • Zhang, Y., Brady, M. and Smith, S. (2001). Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Trans. Med. Imag. 20 45–57.
  • Zhang, C., Fan, J. and Yu, T. (2011). Multiple testing via $\mathrm{FDR}_{L}$ for large-scale imaging data. Ann. Statist. 39 613–642.