The Annals of Statistics

Modeling the variability of rankings

Peter Hall and Hugh Miller

Full-text: Open access


For better or for worse, rankings of institutions, such as universities, schools and hospitals, play an important role today in conveying information about relative performance. They inform policy decisions and budgets, and are often reported in the media. While overall rankings can vary markedly over relatively short time periods, it is not unusual to find that the ranks of a small number of “highly performing” institutions remain fixed, even when the data on which the rankings are based are extensively revised, and even when a large number of new institutions are added to the competition. In the present paper, we endeavor to model this phenomenon. In particular, we interpret as a random variable the value of the attribute on which the ranking should ideally be based. More precisely, if p items are to be ranked then the true, but unobserved, attributes are taken to be values of p independent and identically distributed variates. However, each attribute value is observed only with noise, and via a sample of size roughly equal to n, say. These noisy approximations to the true attributes are the quantities that are actually ranked. We show that, if the distribution of the true attributes is light-tailed (e.g., normal or exponential) then the number of institutions whose ranking is correct, even after recalculation using new data and even after many new institutions are added, is essentially fixed. Formally, p is taken to be of order nC for any fixed C > 0, and the number of institutions whose ranking is reliable depends very little on p. On the other hand, cases where the number of reliable rankings increases significantly when new institutions are added are those for which the distribution of the true attributes is relatively heavy-tailed, for example, with tails that decay like xα for some α > 0. These properties and others are explored analytically, under general conditions. A numerical study links the results to outcomes for real-data problems.

Article information

Ann. Statist., Volume 38, Number 5 (2010), 2652-2677.

First available in Project Euclid: 11 July 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G32: Statistics of extreme values; tail inference
Secondary: 62E20: Asymptotic distribution theory

Bootstrap exponential distribution exponential tails extreme values order statistics Pareto distribution performance rankings regularly varying tails


Hall, Peter; Miller, Hugh. Modeling the variability of rankings. Ann. Statist. 38 (2010), no. 5, 2652--2677. doi:10.1214/10-AOS794.

Export citation


  • Alon, U., Barkai, N., Notterman, D. A., Gish, K., Ybarra, S., Mack, D. and Levine, A. J. (1999). Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc. Natl. Acad. Sci. 96 6745–6750.
  • Amosova, N. N. (1972). Limit theorems for the probabilities of moderate deviations. Vestnik Leningrad. Univ. No. 13 Mat. Meh. Astronom. Vyp. 5–14, 148.
  • Barker, L. E., Smith, P. J., Gerzoff, R. B., Luman, E. T., McCauley, M. M. and Strine, T. W. (2005). Ranking states’ immunization coverage: An example from the National Immunization Survey. Stat. Med. 24 605–613.
  • Brijs, T., Van Den Bossche, F., Wets, G. and Karlis, D. (2006). A model for identifying and ranking dangerous accident locations: A case study in Flanders. Statist. Neerlandica 60 457–476.
  • Brijs, T., Karlis, D., Van Den Bossche, F. and Wets, G. (2007). A Bayesian model for ranking hazardous road sites. J. Roy. Statist. Soc. Ser. A 170 1001–1017.
  • Cesário, L. C. and Barreto, M. C. M. (2003). Study of the performance of bootstrap confidence intervals for the mean of a normal distribution using perfectly ranked set sampling. Rev. Mat. Estatíst. 21 7–20.
  • Chen, H., Stansy, E. A. and Wolfe, D. A. (2006). An empirical assessment of ranking accuracy in ranked set sampling. Comput. Statist. Data Anal. 51 1411–1419.
  • Corain, L. and Salmaso, L. (2007). A non-parametric method for defining a global preference ranking of industrial products. J. Appl. Statist. 34 203–216.
  • Goldstein, H. and Spiegelhalter, D. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. J. Roy. Statist. Soc. Ser. A 159 385–443.
  • Hall, P. and Miller, H. (2009). Using the bootstrap to quantify the authority of an empirical ranking. Ann. Statist. 37 3929–3959.
  • Hill, B. M. (1975). A simple general approach to inference about the tail of a distribution. Ann. Statist. 3 1163–1174.
  • Hui, T. P., Modarres, R. and Zheng, G. (2005). Bootstrap confidence interval estimation of mean via ranked set sampling linear regression. J. Stat. Comput. Simul. 75 543–553.
  • Joe, H. (2000). Inequalities for random utility models, with applications to ranking and subset choice data. Methodol. Comput. Appl. Probab. 2 359–372.
  • Joe, H. (2001). Multivariate extreme value distributions and coverage of ranking probabilities. J. Math. Psych. 45 180–188.
  • Langford, I. H. and Leyland, A. H. (1996). Discussion of “League tables and their limitations: Statistical issues in comparisons of institutional performance” by Goldstein and Spiegelhalter. J. Roy. Statist. Soc. Ser. A 159 427–428.
  • McHale, I. and Scarf, P. (2005). Ranking football players. Significance 2 54–57.
  • Mease, D. (2003). A penalized maximum likelihood approach for the ranking of college football teams independent of victory margins. Amer. Statist. 57 241–248.
  • Mukherjee, S. N., Sykacek, P., Roberts, S. J. and Gurr, S. J. (2003). Gene ranking using bootstrapped p-values. Sigkdd Explorations 5 14–18.
  • Murphy, T. B. and Martin, D. (2003). Mixtures of distance-based models for ranking data. Comput. Statist. Data Anal. 41 645–655.
  • Nordberg, L. (2006). On the reliability of performance rankings. In Festschrift for Tarmo Pukkila on His 60th Birthday (E. P. Liski, J. Isotalo, J. Niemelä and G. P. H. Styan, eds.) 205–216. Univ. Tampere, Tampere, Finland.
  • Opgen-Rhein, R. and Strimmer, K. (2007). Accurate ranking of differentially expressed genes by a distribution-free shrinkage approach. Stat. Appl. Genet. Mol. Biol. 6 Art. 9, 20pp. (electronic).
  • Quevedo, J. R., Bahamonde, A. and Luaces, O. (2007). A simple and efficient method for variable ranking according to their usefulness for learning. Comput. Statist. Data Anal. 52 578–595.
  • Rényi, A. (1953). On the theory of order statistics. Acta Math. Acad. Sci. Hungar. 4 191–232.
  • Rubin, H. and Sethuraman, J. (1965). Probabilities of moderate deviations. Sankhyā Ser. A 27 325–346.
  • Taconeli, C. A. and Barreto, M. C. M. (2005). Evaluation of a bootstrap confidence interval approach in perfectly ranked set sampling. Rev. Mat. Estatíst. 23 33–53.
  • Xie, M., Singh, K. and Zhang, C. H. (2009). Confidence intervals for population ranks in the presence of ties and near ties. J. Amer. Statist. Assoc. 104 775–787.