Open Access
Translator Disclaimer
December 2019 Active ranking from pairwise comparisons and when parametric assumptions do not help
Reinhard Heckel, Nihar B. Shah, Kannan Ramchandran, Martin J. Wainwright
Ann. Statist. 47(6): 3099-3126 (December 2019). DOI: 10.1214/18-AOS1772


We consider sequential or active ranking of a set of $n$ items based on noisy pairwise comparisons. Items are ranked according to the probability that a given item beats a randomly chosen item, and ranking refers to partitioning the items into sets of prespecified sizes according to their scores. This notion of ranking includes as special cases the identification of the top-$k$ items and the total ordering of the items. We first analyze a sequential ranking algorithm that counts the number of comparisons won, and uses these counts to decide whether to stop, or to compare another pair of items, chosen based on confidence intervals specified by the data collected up to that point. We prove that this algorithm succeeds in recovering the ranking using a number of comparisons that is optimal up to logarithmic factors. This guarantee does depend on whether or not the underlying pairwise probability matrix, satisfies a particular structural property, unlike a significant body of past work on pairwise ranking based on parametric models such as the Thurstone or Bradley–Terry–Luce models. It has been a long-standing open question as to whether or not imposing these parametric assumptions allows for improved ranking algorithms. For stochastic comparison models, in which the pairwise probabilities are bounded away from zero, our second contribution is to resolve this issue by proving a lower bound for parametric models. This shows, perhaps surprisingly, that these popular parametric modeling choices offer at most logarithmic gains for stochastic comparisons.


Download Citation

Reinhard Heckel. Nihar B. Shah. Kannan Ramchandran. Martin J. Wainwright. "Active ranking from pairwise comparisons and when parametric assumptions do not help." Ann. Statist. 47 (6) 3099 - 3126, December 2019.


Received: 1 September 2016; Revised: 1 March 2018; Published: December 2019
First available in Project Euclid: 31 October 2019

Digital Object Identifier: 10.1214/18-AOS1772

Primary: 68T99

Keywords: Active learning , Bradley–Terry–Luce model , Multi-armed bandits , online learning , pairwise comparisons , ranking

Rights: Copyright © 2019 Institute of Mathematical Statistics


Vol.47 • No. 6 • December 2019
Back to Top