The Annals of Statistics

Global testing against sparse alternatives under Ising models

Rajarshi Mukherjee, Sumit Mukherjee, and Ming Yuan

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

In this paper, we study the effect of dependence on detecting sparse signals. In particular, we focus on global testing against sparse alternatives for the means of binary outcomes following an Ising model, and establish how the interplay between the strength and sparsity of a signal determines its detectability under various notions of dependence. The profound impact of dependence is best illustrated under the Curie–Weiss model where we observe the effect of a “thermodynamic” phase transition. In particular, the critical state exhibits a subtle “blessing of dependence” phenomenon in that one can detect much weaker signals at criticality than otherwise. Furthermore, we develop a testing procedure that is broadly applicable to account for dependence and show that it is asymptotically minimax optimal under fairly general regularity conditions.

Article information

Source
Ann. Statist., Volume 46, Number 5 (2018), 2062-2093.

Dates
Received: November 2016
Revised: May 2017
First available in Project Euclid: 17 August 2018

Permanent link to this document
https://projecteuclid.org/euclid.aos/1534492829

Digital Object Identifier
doi:10.1214/17-AOS1612

Mathematical Reviews number (MathSciNet)
MR3845011

Zentralblatt MATH identifier
06964326

Subjects
Primary: 62G10: Hypothesis testing 62G20: Asymptotic properties 62C20: Minimax procedures

Keywords
Detection boundary Ising models phase transitions sparse signals

Citation

Mukherjee, Rajarshi; Mukherjee, Sumit; Yuan, Ming. Global testing against sparse alternatives under Ising models. Ann. Statist. 46 (2018), no. 5, 2062--2093. doi:10.1214/17-AOS1612. https://projecteuclid.org/euclid.aos/1534492829


Export citation

References

  • Addario-Berry, L., Broutin, N., Devroye, L. and Lugosi, G. (2010). On combinatorial testing problems. Ann. Statist. 38 3063–3092.
  • Arias-Castro, E., Candès, E. J. and Plan, Y. (2011). Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism. Ann. Statist. 39 2533–2556.
  • Arias-Castro, E., Donoho, D. L. and Huo, X. (2005). Near-optimal detection of geometric objects by fast multiscale methods. IEEE Trans. Inform. Theory 51 2402–2425.
  • Arias-Castro, E. and Wang, M. (2015). The sparse Poisson means model. Electron. J. Stat. 9 2170–2201.
  • Arias-Castro, E., Candès, E. J., Helgason, H. and Zeitouni, O. (2008). Searching for a trail of evidence in a maze. Ann. Statist. 36 1726–1757.
  • Besag, J. (1974). Spatial interaction and the statistical analysis of lattice systems. J. Roy. Statist. Soc. Ser. B 36 192–236.
  • Besag, J. (1975). Statistical analysis of non-lattice data. Amer. Statist. 179–195.
  • Bhattacharya, B. B. and Mukherjee, S. (2018). Inference in Ising models. Bernoulli 24 493–525.
  • Burnašev, M. V. (1979). Minimax detection of an imperfectly known signal against a background of Gaussian white noise. Teor. Veroyatn. Primen. 24 106–118.
  • Cai, T. T. and Yuan, M. (2014). Rate-optimal detection of very short signal segments. Preprint. Available at arXiv:1407.2812.
  • Chatterjee, S. (2005). Concentration Inequalities with Exchangeable Pairs. Ph.D. thesis, Stanford University. Available at arXiv:math/0507526.
  • Chatterjee, S. (2007a). Estimation in spin glasses: A first step. Ann. Statist. 35 1931–1946.
  • Chatterjee, S. (2007b). Stein’s method for concentration inequalities. Probab. Theory Related Fields 138 305–321.
  • Chatterjee, S. and Dey, P. S. (2010). Applications of Stein’s method for concentration inequalities. Ann. Probab. 38 2443–2485.
  • Comets, F. and Gidas, B. (1991). Asymptotics of maximum likelihood estimators for the Curie–Weiss model. Ann. Statist. 19 557–578.
  • Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • Ellis, R. S. and Newman, C. M. (1978). The statistics of Curie–Weiss models. J. Stat. Phys. 19 149–161.
  • Grimmett, G. (2006). The Random-Cluster Model. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 333. Springer, Berlin.
  • Guyon, X. (1995). Random Fields on a Network: Modeling, Statistics, and Applications. Springer, New York.
  • Hall, P. and Jin, J. (2008). Properties of higher criticism under strong dependence. Ann. Statist. 36 381–402.
  • Hall, P. and Jin, J. (2010). Innovated higher criticism for detecting sparse signals in correlated noise. Ann. Statist. 38 1686–1732.
  • Ingster, Y. I. (1994). Minimax detection of a signal in $l_{p}$ metrics. J. Math. Sci. 68 503–515.
  • Ingster, Y. I. (1998). Minimax detection of a signal for $l^{n}$-balls. Math. Methods Statist. 7 401–428.
  • Ingster, Y. I. and Suslina, I. A. (2003). Nonparametric Goodness-of-Fit Testing Under Gaussian Models. Lecture Notes in Statistics 169. Springer, New York.
  • Ingster, Y. I., Tsybakov, A. B. and Verzelen, N. (2010). Detection boundary in sparse regression. Electron. J. Stat. 4 1476–1526.
  • Ising, E. (1925). Beitrag zur theorie des ferromagnetismus. Zeitschrift Für Physik A Hadrons and Nuclei 31 253–258.
  • Jin, J. and Ke, Z. T. (2016). Rare and weak effects in large-scale inference: Methods and phase diagrams. Statist. Sinica 26 1–34.
  • Kac, M. (1959). On the Partition Function of a One-Dimensional Gas. Phys. Fluids 2 8–12.
  • Kac, M. (1969). Mathematical Mechanisms of Phase Transitions. Technical Report, Rockefeller Univ., New York.
  • Majewski, J., Li, H. and Ott, J. (2001). The Ising model in physics and statistical genetics. Am. J. Hum. Genet. 69 853–862.
  • Mézard, M. and Montanari, A. (2009). Information, Physics, and Computation. Oxford Univ. Press, Oxford.
  • Mukherjee, S. (2013). Consistent estimation in the two star exponential random graph model. Preprint. Available at arXiv:1310.4526.
  • Mukherjee, R., Mukherjee, S. and Yuan, M. (2018). Supplement to “Global testing against sparse alternatives under Ising models.” DOI:10.1214/17-AOS1612SUPP.
  • Mukherjee, R., Pillai, N. S. and Lin, X. (2015). Hypothesis testing for high-dimensional sparse binary regression. Ann. Statist. 43 352–381.
  • Nishimori, H. (2001). Statistical Physics of Spin Glasses and Information Processing: An Introduction. Oxford Univ. Press, New York.
  • Onsager, L. (1944). Crystal statistics. I. A two-dimensional model with an order-disorder transition. Phys. Rep. 65 117–149.
  • Park, J. and Newman, M. E. J. (2004). Solution of the two-star model of a network. Phys. Rev. E (3) 70 066146.
  • Stauffer, D. (2008). Social applications of two-dimensional Ising models. Am. J. Phys. 76 470–473.
  • Wu, Z., Sun, Y., He, S., Cho, J., Zhao, H. and Jin, J. (2014). Detection boundary and higher criticism approach for rare and weak genetic effects. Ann. Appl. Stat. 8 824–851.

Supplemental materials

  • Supplement to “Global testing against sparse alternatives under Ising models”. The supplementary material contain the proofs of additional technical results.