The Annals of Statistics

Computational barriers in minimax submatrix detection

Zongming Ma and Yihong Wu

Full-text: Open access

Abstract

This paper studies the minimax detection of a small submatrix of elevated mean in a large matrix contaminated by additive Gaussian noise. To investigate the tradeoff between statistical performance and computational cost from a complexity-theoretic perspective, we consider a sequence of discretized models which are asymptotically equivalent to the Gaussian model. Under the hypothesis that the planted clique detection problem cannot be solved in randomized polynomial time when the clique size is of smaller order than the square root of the graph size, the following phase transition phenomenon is established: when the size of the large matrix $p\to\infty$, if the submatrix size $k=\Theta(p^{\alpha})$ for any $\alpha\in(0,{2}/{3})$, computational complexity constraints can incur a severe penalty on the statistical performance in the sense that any randomized polynomial-time test is minimax suboptimal by a polynomial factor in $p$; if $k=\Theta(p^{\alpha})$ for any $\alpha\in({2}/{3},1)$, minimax optimal detection can be attained within constant factors in linear time. Using Schatten norm loss as a representative example, we show that the hardness of attaining the minimax estimation rate can crucially depend on the loss function. Implications on the hardness of support recovery are also obtained.

Article information

Source
Ann. Statist., Volume 43, Number 3 (2015), 1089-1116.

Dates
Received: August 2014
Revised: December 2014
First available in Project Euclid: 15 May 2015

Permanent link to this document
https://projecteuclid.org/euclid.aos/1431695639

Digital Object Identifier
doi:10.1214/14-AOS1300

Mathematical Reviews number (MathSciNet)
MR3346698

Zentralblatt MATH identifier
1328.62354

Subjects
Primary: 62H15: Hypothesis testing
Secondary: 62C20: Minimax procedures

Keywords
Asymptotic equivalence high-dimensional statistics computational complexity minimax rate planted clique submatrix detection

Citation

Ma, Zongming; Wu, Yihong. Computational barriers in minimax submatrix detection. Ann. Statist. 43 (2015), no. 3, 1089--1116. doi:10.1214/14-AOS1300. https://projecteuclid.org/euclid.aos/1431695639


Export citation

References

  • [1] Addario-Berry, L., Broutin, N., Devroye, L. and Lugosi, G. (2010). On combinatorial testing problems. Ann. Statist. 38 3063–3092.
  • [2] Alon, N., Andoni, A., Kaufman, T., Matulef, K., Rubinfeld, R. and Xie, N. (2007). Testing $k$-wise and almost $k$-wise independence. In STOC’07—Proceedings of the 39th Annual ACM Symposium on Theory of Computing 496–505. ACM, New York.
  • [3] Alon, N., Krivelevich, M. and Sudakov, B. (1998). Finding a large hidden clique in a random graph. In Proceedings of the Ninth Annual ACM–SIAM Symposium on Discrete Algorithms (San Francisco, CA, 1998) 594–598. ACM, New York.
  • [4] Ames, B. P. W. and Vavasis, S. A. (2011). Nuclear norm minimization for the planted clique and biclique problems. Math. Program. 129 69–89.
  • [5] Applebaum, B., Barak, B. and Wigderson, A. (2010). Public-key cryptography from different assumptions. In STOC’10—Proceedings of the 2010 ACM International Symposium on Theory of Computing 171–180. ACM, New York.
  • [6] Arias-Castro, E. and Verzelen, N. (2013). Community detection in random networks. Preprint. Available at arXiv:1302.7099.
  • [7] Arora, S. and Barak, B. (2009). Computational Complexity: A Modern Approach. Cambridge Univ. Press, Cambridge.
  • [8] Balakrishnan, S., Kolar, M., Rinaldo, A., Singh, A. and Wasserman, L. (2011). Statistical and computational tradeoffs in biclustering. In NIPS 2011 Workshop on Computational Trade-Offs in Statistical Learning.
  • [9] Berthet, Q. and Rigollet, P. (2013). Complexity theoretic lower bounds for sparse principal component detection. Journal of Machine Learning Research: Workshop and Conference Proceedings 30 1–21.
  • [10] Berthet, Q. and Rigollet, P. (2013). Optimal detection of sparse principal components in high dimension. Ann. Statist. 41 1780–1815.
  • [11] Bhamidi, S., Dey, P. S. and Nobel, A. B. (2012). Energy landscape for large average submatrix detection problems in Gaussian random matrices. Preprint. Available at arXiv:1211.2284.
  • [12] Bhatia, R. (1997). Matrix Analysis. Graduate Texts in Mathematics 169. Springer, New York.
  • [13] Butucea, C. and Ingster, Y. I. (2013). Detection of a sparse submatrix of a high-dimensional noisy matrix. Bernoulli 19 2652–2688.
  • [14] Candès, E. J. and Recht, B. (2009). Exact matrix completion via convex optimization. Found. Comput. Math. 9 717–772.
  • [15] Chandrasekaran, V. and Jordan, M. I. (2013). Computational and statistical tradeoffs via convex relaxation. Proc. Natl. Acad. Sci. USA 110 E1181–E1190.
  • [16] Cover, T. M. and Thomas, J. A. (2006). Elements of Information Theory, 2nd ed. Wiley, Hoboken, NJ.
  • [17] Csiszár, I. (1967). Information-type measures of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar. 2 299–318.
  • [18] Dekel, Y., Gurel-Gurevich, O. and Peres, Y. (2011). Finding hidden cliques in linear time with high probability. In ANALCO11—Workshop on Analytic Algorithmics and Combinatorics 67–75. SIAM, Philadelphia, PA.
  • [19] Deshpande, Y. and Montanari, A. (2013). Finding hidden cliques of size $\sqrt{N/e}$ in nearly linear time. Preprint. Available at arXiv:1304.7047.
  • [20] Feige, U. and Krauthgamer, R. (2000). Finding and certifying a large hidden clique in a semirandom graph. Random Structures Algorithms 16 195–208.
  • [21] Feige, U. and Ron, D. (2010). Finding hidden cliques in linear time. In 21st International Meeting on Probabilistic, Combinatorial, and Asymptotic Methods in the Analysis of Algorithms (AofA’10) 189–203. Assoc. Discrete Math. Theor. Comput. Sci., Nancy.
  • [22] Feldman, V., Grigorescu, E., Reyzin, L., Vempala, S. S. and Xiao, Y. (2013). Statistical algorithms and a lower bound for detecting planted cliques. In STOC’13—Proceedings of the 2013 ACM Symposium on Theory of Computing 655–664. ACM, New York.
  • [23] Hazan, E. and Krauthgamer, R. (2011). How hard is it to approximate the best Nash equilibrium? SIAM J. Comput. 40 79–91.
  • [24] Jerrum, M. (1992). Large cliques elude the Metropolis process. Random Structures Algorithms 3 347–359.
  • [25] Juels, A. and Peinado, M. (2000). Hiding cliques for cryptographic security. Des. Codes Cryptogr. 20 269–280.
  • [26] Knuth, D. E. (1969). The Art of Computer Programming. Vol. 2: Seminumerical Algorithms. Addison-Wesley, Reading, MA.
  • [27] Koiran, P. and Zouzias, A. (2014). Hidden cliques and the certification of the restricted isometry property. IEEE Trans. Inform. Theory 60 4999–5006.
  • [28] Kolar, M., Balakrishnan, S., Rinaldo, A. and Singh, A. (2011). Minimax localization of structural information in large noisy matrices. Adv. Neural Inf. Process. Syst. 24 909–917.
  • [29] Koshy, T. (2009). Catalan Numbers with Applications. Oxford Univ. Press, Oxford.
  • [30] Krauthgamer, R., Nadler, B. and Vilenchik, D. (2013). Do semidefinite relaxations really solve sparse PCA? Preprint. Available at arXiv:1306.3690.
  • [31] Kuvcera, L. (1992). A generalized encryption scheme based on random graphs. In Graph-Theoretic Concepts in Computer Science (Fischbachau, 1991). Lecture Notes in Computer Science 570 180–186. Springer, Berlin.
  • [32] Kuvcera, L. (1995). Expected complexity of graph partitioning problems. Discrete Appl. Math. 57 193–212.
  • [33] Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer, New York.
  • [34] Ma, Z. and Wu, Y. (2013). Volume ratio, sparsity, and minimaxity under unitarily invariant norms. Preprint. Available at arXiv:1306.3609.
  • [35] Ma, Z. and Wu, Y. (2015). Supplement to “Computational barriers in minimax submatrix detection.” DOI:10.1214/14-AOS1300SUPP.
  • [36] Rényi, A. (1959). On the dimension and entropy of probability distributions. Acta Math. Acad. Sci. Hungar. 10 193–215 (unbound insert).
  • [37] Rossman, B. (2010). Average-case complexity of detecting cliques. Ph.D. thesis, Massachusetts Institute of Technology.
  • [38] Shabalin, A. A., Weigman, V. J., Perou, C. M. and Nobel, A. B. (2009). Finding large average submatrices in high dimensional data. Ann. Appl. Stat. 3 985–1012.
  • [39] Shiryaev, A. N. and Spokoiny, V. G. (2000). Statistical Experiments and Decisions: Asymptotic Theory. Advanced Series on Statistical Science & Applied Probability 8. World Scientific, River Edge, NJ.
  • [40] Strassen, V. (1965). The existence of probability measures with given marginals. Ann. Math. Statist. 36 423–439.
  • [41] Sun, X. and Nobel, A. B. (2013). On the maximal size of large-average and ANOVA-fit submatrices in a Gaussian random matrix. Bernoulli 19 275–294.
  • [42] Verzelen, N. and Arias-Castro, E. (2013). Community detection in sparse random networks. Preprint. Available at arXiv:1308.2955.

Supplemental materials