The Annals of Applied Statistics

Power-law distributions in binned empirical data

Yogesh Virkar and Aaron Clauset

Full-text: Access has been disabled (more information)

Abstract

Many man-made and natural phenomena, including the intensity of earthquakes, population of cities and size of international wars, are believed to follow power-law distributions. The accurate identification of power-law patterns has significant consequences for correctly understanding and modeling complex systems. However, statistical evidence for or against the power-law hypothesis is complicated by large fluctuations in the empirical distribution’s tail, and these are worsened when information is lost from binning the data. We adapt the statistically principled framework for testing the power-law hypothesis, developed by Clauset, Shalizi and Newman, to the case of binned data. This approach includes maximum-likelihood fitting, a hypothesis test based on the Kolmogorov–Smirnov goodness-of-fit statistic and likelihood ratio tests for comparing against alternative explanations. We evaluate the effectiveness of these methods on synthetic binned data with known structure, quantify the loss of statistical power due to binning, and apply the methods to twelve real-world binned data sets with heavy-tailed patterns.

Article information

Source
Ann. Appl. Stat. Volume 8, Number 1 (2014), 89-119.

Dates
First available in Project Euclid: 8 April 2014

Permanent link to this document
http://projecteuclid.org/euclid.aoas/1396966280

Digital Object Identifier
doi:10.1214/13-AOAS710

Mathematical Reviews number (MathSciNet)
MR3191984

Zentralblatt MATH identifier
06302229

Keywords
Power-law distribution heavy-tailed distributions model selection binned data

Citation

Virkar, Yogesh; Clauset, Aaron. Power-law distributions in binned empirical data. Ann. Appl. Stat. 8 (2014), no. 1, 89--119. doi:10.1214/13-AOAS710. http://projecteuclid.org/euclid.aoas/1396966280.


Export citation

References

  • Aban, I. B. and Meerschaert, M. M. (2004). Generalized least-squares estimators for the thickness of heavy tails. J. Statist. Plann. Inference 119 341–352.
    Mathematical Reviews (MathSciNet): MR2019645
    Digital Object Identifier: doi:10.1016/S0378-3758(02)00419-6
  • Arnold, B. C. (1983). Pareto Distributions. Statistical Distributions in Scientific Work 5. International Co-operative Publishing House, Burtonsville, MD.
    Mathematical Reviews (MathSciNet): MR751409
  • Asal, V. and Rethemeyer, R. K. (2008). The nature of the beast: Organizational structures and the lethality of terrorist attacks. The Journal of Politics 70 437–449.
  • Barndorff-Nielsen, O. E. and Cox, D. R. (1995). Inference and Asymptotics. Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR1317097
  • Beirlant, J. and Teugels., J. L. (1989). Asymptotic normality of Hill’s estimator. Extreme Value Theory 51 148–155.
    Mathematical Reviews (MathSciNet): MR992055
    Digital Object Identifier: doi:10.1007/978-1-4612-3634-4_13
  • Breiman, L., Stone, C. J. and Kooperberg, C. (1990). Robust confidence bounds for extreme upper quantiles. J. Stat. Comput. Simul. 37 127–149.
    Mathematical Reviews (MathSciNet): MR1082452
    Digital Object Identifier: doi:10.1080/00949659008811300
  • Cadez, I. V., Smyth, P., McLachlan, G. J. and McLaren, C. E. (2002). Maximum likelihood estimation of mixture of densities for binned and truncated multivariate data. Machine Learning 47 7–34.
  • Chapuis, A. and Tetzlaff, T. (2012). The variability of tidewater-glacier calving: Origin of event-size and interval distributions. Available at arXiv:1205.1640.
    arXiv: arXiv:1205.1640
  • Clauset, A., Shalizi, C. R. and Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Rev. 51 661–703.
    Mathematical Reviews (MathSciNet): MR2563829
    Digital Object Identifier: doi:10.1137/070710111
  • Clauset, A. and Woodard, R. (2013). Estimating the historical and future probabilities of large terrorist events. Ann. Appl. Stat. 7 1838–1865.
    Mathematical Reviews (MathSciNet): MR3161698
    Digital Object Identifier: doi:10.1214/12-AOAS614
    Project Euclid: euclid.aoas/1387823295
  • Clauset, A., Young, M. and Gleditsch, K. S. (2007). On the frequency of severe terrorist events. Journal of Conflict Resolution 51 58–87.
  • Cramér, H. (1946). A contribution to the theory of statistical estimation. Skand. Aktuarietidskr. 29 85–94.
    Mathematical Reviews (MathSciNet): MR17505
  • Danielsson, J., de Haan, L., Peng, L. and de Vries, C. G. (2001). Using a bootstrap method to choose the sample fraction in tail index estimation. J. Multivariate Anal. 76 226–248.
    Mathematical Reviews (MathSciNet): MR1821820
    Digital Object Identifier: doi:10.1006/jmva.2000.1903
  • Dekkers, A. L. M. and de Haan, L. (1993). Optimal choice of sample fraction in extreme-value estimation. J. Multivariate Anal. 47 173–195.
    Mathematical Reviews (MathSciNet): MR1247373
    Digital Object Identifier: doi:10.1006/jmva.1993.1078
  • Drees, H. and Kaufmann, E. (1998). Selecting the optimal sample fraction in univariate extreme value estimation. Stochastic Process. Appl. 75 149–172.
    Mathematical Reviews (MathSciNet): MR1632189
    Digital Object Identifier: doi:10.1016/S0304-4149(98)00017-9
  • Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability 57. Chapman & Hall, New York.
    Mathematical Reviews (MathSciNet): MR1270903
  • Gabaix, X. (2009). Power laws in economics and finance. Annual Review of Economics 1 255–293.
  • Goh, K.-I., Cusick, M. E., Valle, D., Childs, B., Vidal, M. and Barabási, A.-L. (2007). The human disease network. Proc. Natl. Acad. Sci. USA 104 8685–8690.
  • Goldstein, M. L., Morris, S. A. and Yen, G. G. (2004). Problems with fitting to the power-law distribution. Eur. Phys. J. B 41 255–258.
  • Grünwald, P. D. (2007). The Minimum Length Description Principle. MIT Press, Cambridge, MA.
  • Hall, P. (1982). On some simple estimates of an exponent of regular variation. J. R. Stat. Soc. Ser. B Stat. Methodol. 44 37–42.
    Mathematical Reviews (MathSciNet): MR655370
  • Handcock, M. S. and Jones, J. H. (2004). Likelihood-based inference for stochastic models of sexual network evolution. Theoretical Population Biology 65 413–422.
  • Heritage Provider Network (2012). Health heritage prize data files, HHP_release3. Available at http://bit.ly/wG8Psl.
  • Hill, B. M. (1975). A simple general approach to inference about the tail of a distribution. Ann. Statist. 3 1163–1174.
    Mathematical Reviews (MathSciNet): MR378204
    Digital Object Identifier: doi:10.1214/aos/1176343247
    Project Euclid: euclid.aos/1176343247
  • Horn, S. D. (1977). Goodness-of-fit tests for discrete data: A review and an application to a health development scale. Biometrics 33 237–247.
  • Ijiri, Y. and Simon, H. A. (1977). Skew Distributions and the Sizes of Business Firms. North-Holland, Amsterdam.
  • Jarvinen, B., Neumann, C. and Davis, M. A. S. (2012). NHC data archive. National Hurricane Center. Available at http://1.usa.gov/cCcwTg.
  • Kass, R. E. and Raftery, A. E. (1994). Bayes factors. J. Amer. Statist. Assoc. 90 773–795.
  • Kratz, M. and Resnick, S. I. (1996). The $\mathrm{QQ}$-estimator and heavy tails. Comm. Statist. Stochastic Models 12 699–724.
    Mathematical Reviews (MathSciNet): MR1410853
    Digital Object Identifier: doi:10.1080/15326349608807407
  • McLachlan, G. J. and Jones, P. N. (1988). Fitting mixture models to grouped and truncated data via the EM algorithm. Biometrics 44 571–578.
  • Mitzenmacher, M. (2004). A brief history of generative models for power law and lognormal distributions. Internet Math. 1 226–251.
    Mathematical Reviews (MathSciNet): MR2077227
    Digital Object Identifier: doi:10.1080/15427951.2004.10129088
    Project Euclid: euclid.im/1089229510
  • Mitzenmacher, M. (2006). The future of power law research. Internet Math. 2 525–534.
  • Newman, M. E. J. (2005). Power laws, Pareto distributions and Zipf’s law. Contemporary Physics 46 323–351.
  • Noether, G. E. (1963). Note on the Kolmogorov statistic in the discrete case. Metrika 7 115–116.
    Mathematical Reviews (MathSciNet): MR158462
    Digital Object Identifier: doi:10.1007/BF02613966
  • Orphanet Report Series, Rare Diseases collection (2011). Prevalence of rare diseases: Bibliographic data. Available at http://bit.ly/MezSZ6.
  • Persing, J. and Montgomery, M. T. (2003). Hurricane superintensity. J. Atmospheric Sci. 60 2349–2371.
  • Press, W. H., Teukolsky, S. A., Vetterling, W. T. and Flannery, B. P. (1992). Numerical Recipes in C: The Art of Scientific Computing, 2nd ed. Cambridge Univ. Press, Cambridge.
    Mathematical Reviews (MathSciNet): MR1201159
  • Rao, C. R. (1947). Minimum variance and the estimation of several parameters. Proc. Cambridge Philos. Soc. 43 280–283.
    Mathematical Reviews (MathSciNet): MR19904
    Digital Object Identifier: doi:10.1017/S0305004100023471
  • Rao, C. R. (1957). Maximum likelihood estimation for the multinomial distribution. Sankhyā 18 139–148.
    Mathematical Reviews (MathSciNet): MR105183
  • Reed, W. J. and Hughes, B. D. (2002). From gene families and genera to income and internet file sizes: Why power laws are so common in nature. Phys. Rev. E (3) 66 067103.
  • Reiss, R.-D. and Thomas, M. (2007). Statistical Analysis of Extreme Values with Applications to Insurance, Finance, Hydrology and Other Fields, 3rd ed. Birkhäuser, Basel.
    Mathematical Reviews (MathSciNet): MR2334035
  • Richardson, L. F. (1960). Statistics of Deadly Quarrels. The Boxwood Press, Pittsburgh.
    Mathematical Reviews (MathSciNet): MR118552
  • Schultze, J. and Steinebach, J. (1996). On least squares estimates of an exponential tail coefficient. Statist. Decisions 14 353–372.
    Mathematical Reviews (MathSciNet): MR1437826
  • Shinokazi, K., Yoda, K., Hozumi, K. and Kira, T. (1964). A quantitative analysis of plant form—The pipe model theory II: Further evidence of the theory and its application in forest ecology. Japanese Journal of Ecology 14 133–139.
  • Sornette, D. (2006). Critical Phenomena in Natural Sciences: Chaos, Fractals, Selforganization and Disorder: Concepts and Tools, 2nd ed. Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR2220576
  • Stoev, S. A., Michailidis, G. and Taqqu, M. S. (2011). Estimating heavy-tail exponents through max self-similarity. IEEE Trans. Inform. Theory 57 1615–1636.
    Mathematical Reviews (MathSciNet): MR2815838
    Digital Object Identifier: doi:10.1109/TIT.2010.2103751
  • Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B Stat. Methodol. 36 111–147.
    Mathematical Reviews (MathSciNet): MR356377
  • Storm Prediction Center (2011). Severe weather database files (1950–2011). Available at http://1.usa.gov/Lj7cC9.
  • Stumpf, M. P. H. and Porter, M. A. (2012). Critical truths about power laws. Science 335 665–666.
    Mathematical Reviews (MathSciNet): MR2932329
    Digital Object Identifier: doi:10.1126/science.1216142
  • Tate, M. W. and Hye, L. A. (1973). Inaccuracy of the $\chi^{2}$ test of goodness of fit when expected frequencies are small. J. Amer. Statist. Assoc. 68 836–841.
  • Virkar, Y. and Clauset, A. (2014). Supplement to “Power-law distributions in binned empirical data.” DOI:10.1214/13-AOAS710SUPP.
    Mathematical Reviews (MathSciNet): MR3191984
    Digital Object Identifier: doi:10.1214/13-AOAS710
    Project Euclid: euclid.aoas/1396966280
  • Vuong, Q. H. (1989). Likelihood ratio tests for model selection and nonnested hypotheses. Econometrica 57 307–333.
    Mathematical Reviews (MathSciNet): MR996939
    Digital Object Identifier: doi:10.2307/1912557
  • Wasserman, L. (2004). All of Statistics: A Concise Course in Statistical Inference. Springer, New York.
    Mathematical Reviews (MathSciNet): MR2055670
  • West, G. B., Enquist, B. J. and Brown, J. H. (2009). A general quantitative theory of forest structure and dynamics. Proc. Natl. Acad. Sci. USA 106 7040–7045.
  • World Glacier Monitoring Service and National Snow and Ice Data Center (2012). World glacier inventory. Available at http://bit.ly/MhLdt6.
  • Yamamoto, K. and Kobayashi, S. (1993). Analysis of crown structure based on the pipe model theory. Journal of the Japanese Forestry Society 75 445–448.

Supplemental materials

  • Supplementary material: Supplement to “Power-law distributions in binned empirical data”. In this supplemental file, we derive a closed-form expression for the binned MLE in Section 1.1, quantify the amount of information loss on using a coarser binning scheme in Section 1.2 and include the likelihood ratio test for the binned case in Section 2.
    Digital Object Identifier: doi:10.1214/13-AOAS710SUPP
    Supplemental files available for subscribers.

The Institute of Mathematical Statistics

The Annals of Applied Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more