Electronic Journal of Statistics

Computational implications of reducing data to sufficient statistics

Andrea Montanari

Full-text: Open access

Abstract

Given a large dataset and an estimation task, it is common to pre-process the data by reducing them to a set of sufficient statistics. This step is often regarded as straightforward and advantageous (in that it simplifies statistical analysis). I show that –on the contrary– reducing data to sufficient statistics can change a computationally tractable estimation problem into an intractable one. I discuss connections with recent work in theoretical computer science, and implications for some techniques to estimate graphical models.

Article information

Source
Electron. J. Statist., Volume 9, Number 2 (2015), 2370-2390.

Dates
Received: October 2014
First available in Project Euclid: 23 October 2015

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1445605703

Digital Object Identifier
doi:10.1214/15-EJS1059

Mathematical Reviews number (MathSciNet)
MR3417186

Zentralblatt MATH identifier
1336.62037

Citation

Montanari, Andrea. Computational implications of reducing data to sufficient statistics. Electron. J. Statist. 9 (2015), no. 2, 2370--2390. doi:10.1214/15-EJS1059. https://projecteuclid.org/euclid.ejs/1445605703


Export citation

References

  • [1] Sanjeev Arora and Boaz Barak, Computational complexity: a modern approach, Cambridge University Press, 2009.
  • [2] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski, A learning algorithm for boltzmann machines, Cognitive science 9 (1985), no. 1, 147–169.
  • [3] Pieter Abbeel, Daphne Koller, and Andrew Y Ng, Learning factor graphs in polynomial time and sample complexity, The Journal of Machine Learning Research 7 (2006), 1743–1788.
  • [4] Animashree Anandkumar, Vincent YF Tan, Furong Huang, and Alan S Willsky, High-dimensional structure estimation in Ising models: Local separation criterion, The Annals of Statistics 40 (2012), no. 3, 1346–1375.
  • [5] Onureena Banerjee, Laurent El Ghaoui, and Alexandre d’Aspremont, Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data, The Journal of Machine Learning Research 9 (2008), 485–516.
  • [6] Guy Bresler, David Gamarnik, and Devavrat Shah, Hardness of parameter estimation in graphical models, unpublished, 2014.
  • [7] José Bento and Andrea Montanari, Which graphical models are difficult to learn?, Neural Information Processing Systems (Vancouver), December 2009.
  • [8] Guy Bresler, Elchanan Mossel, and Allan Sly, Reconstruction of markov random fields from samples: Some observations and algorithms, Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, Springer, 2008, pp. 343–356.
  • [9] Andrej Bogdanov, Elchanan Mossel, and Salil Vadhan, The complexity of distinguishing markov random fields, Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, Springer, 2008, pp. 331–342.
  • [10] Amir Beck and Marc Teboulle, A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM Journal on Imaging Sciences 2 (2009), no. 1, 183–202.
  • [11] Jin-Yi Cai, Xi Chen, Heng Guo, and Pinyan Lu, Inapproximability after uniqueness phase transition in two-spin systems, Combinatorial Optimization and Applications, Springer, 2012, pp. 336–347.
  • [12] Imre Csiszár and Zsolt Talata, Consistent estimation of the basic neighborhood of Markov random fields, The Annals of Statistics (2006), 123–145.
  • [13] Bradley Efron, The geometry of exponential families, The Annals of Statistics 6 (1978), no. 2, 362–376.
  • [14] Leslie Ann Goldberg, Mark Jerrum, and Mike Paterson, The computational complexity of two-state spin systems, Random Structures & Algorithms 23 (2003), no. 2, 133–154.
  • [15] Martin Grötschel, László Lovász, and Alexander Schrijver, The ellipsoid method and its consequences in combinatorial optimization, Combinatorica 1 (1981), no. 2, 169–197.
  • [16] Geoffrey R Grimmett, The random-cluster model, vol. 333, Springer Science & Business Media, 2006.
  • [17] Andreas Galanis, Daniel Stefankovic, and Eric Vigoda, Inapproximability of the partition function for the antiferromagnetic ising and hard-core models, arXiv :1203.2226 (2012).
  • [18] John J Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences 79 (1982), no. 8, 2554–2558.
  • [19] GE Hinton and TJ Sejnowski, Analysing cooperative computation, Proceedings of the Fifth Annual Conference of the Cognitive Science Society, 1983.
  • [20] Ernst Ising, Beitrag zur theorie des ferromagnetismus, Zeitschrift für Physik A Hadrons and Nuclei 31 (1925), no. 1, 253–258.
  • [21] EL Lehmann and George Casella, Theory of point estimation, 2 ed., Springer, 1998.
  • [22] László Lovász, An algorithmic theory of numbers, graphs and convexity, vol. 50, SIAM, 1987.
  • [23] Nicolai Meinshausen and Peter Bühlmann, High-dimensional graphs and variable selection with the lasso, Ann. Statist. 34 (2006), 1436–1462.
  • [24] Marc Mézard and Andrea Montanari, Information, Physics and Computation, Oxford, 2009.
  • [25] Tim Roughgarden and Michael Kearns, Marginals-to-models reducibility, Advances in Neural Information Processing Systems, 2013, pp. 1043–1051.
  • [26] Avik Ray, Sujay Sanghavi, and Sanjay Shakkottai, Greedy learning of graphical models with small girth, Communication, Control, and Computing (Allerton), 2012 50th Annual Allerton Conference on, IEEE, 2012, pp. 2024–2031.
  • [27] Pradeep Ravikumar, Martin J Wainwright, John D Lafferty, et al., High-dimensional Ising model selection using $\ell_1$-regularized logistic regression, The Annals of Statistics 38 (2010), no. 3, 1287–1319.
  • [28] Allan Sly, Computational transition at the uniqueness threshold, Foundations of Computer Science (FOCS), 2010 51st Annual IEEE Symposium on, IEEE, 2010, pp. 287–296.
  • [29] Allan Sly and Nike Sun, The computational hardness of counting in two-spin models on d-regular graphs, Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on, IEEE, 2012, pp. 361–369.
  • [30] Alistair Sinclair, Piyush Srivastava, and Marc Thurley, Approximation algorithms for two-state anti-ferromagnetic spin systems on bounded degree graphs, Journal of Statistical Physics 155 (2014), no. 4, 666–686.
  • [31] Mohit Singh and Nisheeth K Vishnoi, Entropy, optimization and counting, arXiv preprint arXiv :1304.8108 (2013).