Annals of Probability

Algorithmic thresholds for tensor PCA

Abstract

We study the algorithmic thresholds for principal component analysis of Gaussian $k$-tensors with a planted rank-one spike, via Langevin dynamics and gradient descent. In order to efficiently recover the spike from natural initializations, the signal-to-noise ratio must diverge in the dimension. Our proof shows that the mechanism for the success/failure of recovery is the strength of the “curvature” of the spike on the maximum entropy region of the initial data. To demonstrate this, we study the dynamics on a generalized family of high-dimensional landscapes with planted signals, containing the spiked tensor models as specific instances. We identify thresholds of signal-to-noise ratios above which order 1 time recovery succeeds; in the case of the spiked tensor model, these match the thresholds conjectured for algorithms such as approximate message passing. Below these thresholds, where the curvature of the signal on the maximal entropy region is weak, we show that recovery from certain natural initializations takes at least stretched exponential time. Our approach combines global regularity estimates for spin glasses with pointwise estimates to study the recovery problem by a perturbative approach.

Article information

Source
Ann. Probab., Volume 48, Number 4 (2020), 2052-2087.

Dates
Revised: November 2019
First available in Project Euclid: 20 July 2020

https://projecteuclid.org/euclid.aop/1595232103

Digital Object Identifier
doi:10.1214/19-AOP1415

Mathematical Reviews number (MathSciNet)
MR4124533

Zentralblatt MATH identifier
07224968

Citation

Ben Arous, Gérard; Gheissari, Reza; Jagannath, Aukosh. Algorithmic thresholds for tensor PCA. Ann. Probab. 48 (2020), no. 4, 2052--2087. doi:10.1214/19-AOP1415. https://projecteuclid.org/euclid.aop/1595232103

References

• [1] Abbe, E. and Sandon, C. (2018). Proof of the achievability conjectures for the general stochastic block model. Comm. Pure Appl. Math. 71 1334–1406.
• [2] Auffinger, A. and Ben Arous, G. (2013). Complexity of random smooth functions on the high-dimensional sphere. Ann. Probab. 41 4214–4247.
• [3] Auffinger, A., Ben Arous, G. and Cerný, J. (2013). Random matrices and complexity of spin glasses. Comm. Pure Appl. Math. 66 165–201.
• [4] Baik, J., Ben Arous, G. and Péché, S. (2005). Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab. 33 1643–1697.
• [5] Barak, B., Hopkins, S. B., Kelner, J., Kothari, P., Moitra, A. and Potechin, A. (2016). A nearly tight sum-of-squares lower bound for the planted clique problem. In 57th Annual IEEE Symposium on Foundations of Computer Science—FOCS 2016 428–437. IEEE Computer Soc., Los Alamitos, CA.
• [6] Barak, B. and Moitra, A. (2016). Noisy tensor completion via the sum-of-squares hierarchy. In Conference on Learning Theory 417–445.
• [7] Ben Arous, G., Gheissari, R. and Jagannath, A. (2020). Bounding flows for spherical spin glass dynamics. Comm. Math. Phys. 373 1011–1048.
• [8] Ben Arous, G. and Jagannath, A. (2018). Spectral gap estimates in mean field spin glasses. Comm. Math. Phys. 361 1–52.
• [9] Ben Arous, G., Mei, S., Montanari, A. and Nica, M. (2019). The landscape of the spiked tensor model. Comm. Pure Appl. Math. 72 2282–2330.
• [10] Benaych-Georges, F., Guionnet, A. and Maida, M. (2012). Large deviations of the extreme eigenvalues of random deformations of matrices. Probab. Theory Related Fields 154 703–751.
• [11] Berthet, Q. and Rigollet, P. (2013). Optimal detection of sparse principal components in high dimension. Ann. Statist. 41 1780–1815.
• [12] Capitaine, M., Donati-Martin, C. and Féral, D. (2009). The largest eigenvalues of finite rank deformation of large Wigner matrices: Convergence and nonuniversality of the fluctuations. Ann. Probab. 37 1–47.
• [13] Chen, W.-K. (2019). Phase transition in the spiked random tensor with Rademacher prior. Ann. Statist. 47 2734–2756.
• [14] Ethier, S. N. and Kurtz, T. G. (1986). Markov Processes: Characterization and Convergence. Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, New York..
• [15] Gamarnik, D. and Zadik, I. (2017). High dimensional regression with binary coefficients. Estimating squared error and a phase transtition. In Conference on Learning Theory 948–953.
• [16] Ge, R. and Ma, T. (2017). On the optimization landscape of tensor decompositions. In Advances in Neural Information Processing Systems 30 (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan and R. Garnett, eds.) 3653–3663. Curran Associates, Red Hook.
• [17] Gheissari, R. and Jagannath, A. (2019). On the spectral gap of spherical spin glass dynamics. Ann. Inst. Henri Poincaré Probab. Stat. 55 756–776.
• [18] Gillin, P. and Sherrington, D. (2000). $p>2$ spin glasses with first-order ferromagnetic transitions. J. Phys. A 33 3081–3091.
• [19] Guillemin, V. and Pollack, A. (2010). Differential Topology. AMS Chelsea Publishing, Providence, RI.
• [20] Hillar, C. J. and Lim, L.-H. (2013). Most tensor problems are NP-hard. J. ACM 60 Art. 45, 39.
• [21] Hopkins, S. B., Shi, J., Schramm, T. and Steurer, D. (2016). Fast spectral algorithms from sum-of-squares proofs: Tensor decomposition and planted sparse vectors. In STOC’16—Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing 178–191. ACM, New York.
• [22] Hopkins, S. B., Shi, J. and Steurer, D. (2015). Tensor principal component analysis via sum-of-square proofs. In Conference on Learning Theory 956–1006.
• [23] Hsu, E. P. (2002). Stochastic Analysis on Manifolds. Graduate Studies in Mathematics 38. Amer. Math. Soc., Providence, RI.
• [24] Johnstone, I. (2000). On the Distribution of the Largest Principal Component. Department of Statistics, Stanford Univ., Stanford.
• [25] Kim, C., Bandeira, A. S. and Goemans, M. X. (2017). Community detection in hypergraphs, spiked tensor models, and sum-of-squares. In Sampling Theory and Applications (SampTA), 2017 International Conference on 124–128. IEEE, Los Alamitos.
• [26] Ledoux, M. (2001). The Concentration of Measure Phenomenon. Mathematical Surveys and Monographs 89. Amer. Math. Soc., Providence, RI.
• [27] Lesieur, T., Krzakala, F. and Zdeborová, L. (2017). Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications. J. Stat. Mech. Theory Exp. 7 073403, 86.
• [28] Maïda, M. (2007). Large deviations for the largest eigenvalue of rank one deformations of Gaussian ensembles. Electron. J. Probab. 12 1131–1150.
• [29] Montanari, A., Reichman, D. and Zeitouni, O. (2017). On the limitation of spectral methods: From the Gaussian hidden clique problem to rank one perturbations of Gaussian tensors. IEEE Trans. Inf. Theory 63 1572–1579.
• [30] Péché, S. (2006). The largest eigenvalue of small rank perturbations of Hermitian random matrices. Probab. Theory Related Fields 134 127–173.
• [31] Perry, A., Wein, A. S. and Bandeira, A. S. (2016). Statistical limits of spiked tensor models. Preprint. Available at arXiv:1612.07728.
• [32] Perry, A., Wein, A. S., Bandeira, A. S. and Moitra, A. (2018). Optimality and sub-optimality of PCA I: Spiked random matrix models. Ann. Statist. 46 2416–2451.
• [33] Richard, E. and Montanari, A. (2014). A statistical model for tensor PCA. In Advances in Neural Information Processing Systems 2897–2905.
• [34] Ros, V., Ben Arous, G., Biroli, G. and Cammarota, C. (2018). Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima and phase transitions. Preprint. Available at https://arxiv.org/abs/1804.02686.
• [35] Schoenberg, I. J. (1942). Positive definite functions on spheres. Duke Math. J. 9 96–108.
• [36] Stroock, D. W. and Varadhan, S. R. S. (2006). Multidimensional Diffusion Processes. Classics in Mathematics. Springer, Berlin.
• [37] Subag, E. (2017). The complexity of spherical $p$-spin models—a second moment approach. Ann. Probab. 45 3385–3450.
• [38] Vershynin, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics 47. Cambridge Univ. Press, Cambridge.
• [39] Zdeborová, L. and Krzakala, F. (2016). Statistical physics of inference: Thresholds and algorithms. Adv. Phys. 65 453–552.