Open Access
June, 1963 Pairwise Comparison and Ranking: Optimum Properties of the Row Sum Procedure
Peter J. Huber
Ann. Math. Statist. 34(2): 511-520 (June, 1963). DOI: 10.1214/aoms/1177704162


The present paper is concerned with those pairwise comparison experiments for which ranking in descending order of the "row sums" is the appropriate ranking procedure. More precisely, assume that $n$ items are compared pairwise, omitting no pairs, and that the result of the comparison between item $i$ and $j$ is expressed as a real number $x_{ij}$ satisfying \begin{equation*}\tag{1}x_{ij} + x_{ji} = 0\quad\text{for all} i, j.\end{equation*} $x_{ij}$ might be some measured difference between $i$ and $j$, or it might take the values $1, 0, -1$ according as item $i$ is judged to be superior, equal or inferior to item $j$, or it might be a statistic summarizing the results of several comparisons between the two items, etc. It will be shown that ranking in descending order of the scores \begin{equation*}\tag{2}s_i = \sum^n_{j=1} x_{ij}\end{equation*} (row sum procedure) uniformly minimizes the risk among all permutation invariant procedures, and for all "reasonable" loss functions, provided the $x_{ij} (i < j)$ are independent random variables distributed according to an exponential distribution of the type \begin{equation*}\tag{3}P(x_{ij} \leqq t) = c(\vartheta_i - \vartheta_j) \int^t_{-\infty} e^{(\vartheta_i-\vartheta_j)\tau} \mu(d\tau),\end{equation*} where $\mu$ is a symmetric probability measure on the real line. Then $s = (s_1, \cdots, s_n)$ is a sufficient statistic for the joint distribution of the $x_{ij}$. Under suitable regularity conditions, the converse also holds: if the distributions of the $x_{ij}$ are not of this form, then $s$ is not sufficient, and the row sum procedure is not optimal. Thus, the model just described seems to constitute the natural domain of the row sum ranking procedure. Fortunately, this domain contains the important case where the $x_{ij} (i < j)$ are independent normal variables with mean $\vartheta_i - \vartheta_j$ and equal variance $\sigma^2$. Another particular case--the case of tournaments without draws--where the $x_{ij}$ can take only two values, has been treated in the joint paper [1]. Incidentally, the present paper grew out of an attempt to cover the case of tournaments with draws (see example (iv) in Section 4 below). Although one cannot expect strict optimality properties, one may still ask whether the row sum procedure has "nice" (e.g., minimax) properties even outside the above model. A modest result in this direction is the following. Take a particular joint distribution of the $x_{ij}$ belonging to the above model, and consider the class of all joint distributions of the $x_{ij}$ that lead (up to permutations) to the same joint distribution of the $s_i$ as this particular one. Then this latter will be least favorable for the ranking problem, and the row sum procedure will have minimax properties in each such class (modified minimax principle of Wesler [2]). For instance, if the $x_{ij} (i < j)$ are independent normal variables with mean $\vartheta_{ij}$ and equal variance, then the joint distribution of the $s_i$ depends only on the $\vartheta_i = (1/n) \sum^n_{j=1} \vartheta_{ij}$, and the linear model $\vartheta'_{ij} = \vartheta_i - \vartheta_j$ is least favorable, if the $\vartheta_i$ are kept fixed (up to permutations). There is an immediate generalization of the results to the case of comparison in $k$-tuples instead of in pairs, as indicated in the last section of this paper.


Download Citation

Peter J. Huber. "Pairwise Comparison and Ranking: Optimum Properties of the Row Sum Procedure." Ann. Math. Statist. 34 (2) 511 - 520, June, 1963.


Published: June, 1963
First available in Project Euclid: 27 April 2007

zbMATH: 0116.36702
MathSciNet: MR148185
Digital Object Identifier: 10.1214/aoms/1177704162

Rights: Copyright © 1963 Institute of Mathematical Statistics

Vol.34 • No. 2 • June, 1963
Back to Top