Open Access
Translator Disclaimer
October 2010 Modeling the variability of rankings
Peter Hall, Hugh Miller
Ann. Statist. 38(5): 2652-2677 (October 2010). DOI: 10.1214/10-AOS794

Abstract

For better or for worse, rankings of institutions, such as universities, schools and hospitals, play an important role today in conveying information about relative performance. They inform policy decisions and budgets, and are often reported in the media. While overall rankings can vary markedly over relatively short time periods, it is not unusual to find that the ranks of a small number of “highly performing” institutions remain fixed, even when the data on which the rankings are based are extensively revised, and even when a large number of new institutions are added to the competition. In the present paper, we endeavor to model this phenomenon. In particular, we interpret as a random variable the value of the attribute on which the ranking should ideally be based. More precisely, if p items are to be ranked then the true, but unobserved, attributes are taken to be values of p independent and identically distributed variates. However, each attribute value is observed only with noise, and via a sample of size roughly equal to n, say. These noisy approximations to the true attributes are the quantities that are actually ranked. We show that, if the distribution of the true attributes is light-tailed (e.g., normal or exponential) then the number of institutions whose ranking is correct, even after recalculation using new data and even after many new institutions are added, is essentially fixed. Formally, p is taken to be of order nC for any fixed C > 0, and the number of institutions whose ranking is reliable depends very little on p. On the other hand, cases where the number of reliable rankings increases significantly when new institutions are added are those for which the distribution of the true attributes is relatively heavy-tailed, for example, with tails that decay like xα for some α > 0. These properties and others are explored analytically, under general conditions. A numerical study links the results to outcomes for real-data problems.

Citation

Download Citation

Peter Hall. Hugh Miller. "Modeling the variability of rankings." Ann. Statist. 38 (5) 2652 - 2677, October 2010. https://doi.org/10.1214/10-AOS794

Information

Published: October 2010
First available in Project Euclid: 11 July 2010

zbMATH: 1200.62149
MathSciNet: MR2722452
Digital Object Identifier: 10.1214/10-AOS794

Subjects:
Primary: 62G32
Secondary: 62E20

Keywords: bootstrap , exponential distribution , exponential tails , Extreme values , order statistics , Pareto distribution , performance rankings , regularly varying tails

Rights: Copyright © 2010 Institute of Mathematical Statistics

JOURNAL ARTICLE
26 PAGES


SHARE
Vol.38 • No. 5 • October 2010
Back to Top