Pairwise Comparison and Ranking: Optimum Properties of the Row Sum Procedure

Abstract

The present paper is concerned with those pairwise comparison experiments for which ranking in descending order of the "row sums" is the appropriate ranking procedure. More precisely, assume that $n$ items are compared pairwise, omitting no pairs, and that the result of the comparison between item $i$ and $j$ is expressed as a real number $x_{ij}$ satisfying \begin{equation*}\tag{1}x_{ij} + x_{ji} = 0\quad\text{for all} i, j.\end{equation*} $x_{ij}$ might be some measured difference between $i$ and $j$, or it might take the values $1, 0, -1$ according as item $i$ is judged to be superior, equal or inferior to item $j$, or it might be a statistic summarizing the results of several comparisons between the two items, etc. It will be shown that ranking in descending order of the scores \begin{equation*}\tag{2}s_i = \sum^n_{j=1} x_{ij}\end{equation*} (row sum procedure) uniformly minimizes the risk among all permutation invariant procedures, and for all "reasonable" loss functions, provided the $x_{ij} (i < j)$ are independent random variables distributed according to an exponential distribution of the type \begin{equation*}\tag{3}P(x_{ij} \leqq t) = c(\vartheta_i - \vartheta_j) \int^t_{-\infty} e^{(\vartheta_i-\vartheta_j)\tau} \mu(d\tau),\end{equation*} where $\mu$ is a symmetric probability measure on the real line. Then $s = (s_1, \cdots, s_n)$ is a sufficient statistic for the joint distribution of the $x_{ij}$. Under suitable regularity conditions, the converse also holds: if the distributions of the $x_{ij}$ are not of this form, then $s$ is not sufficient, and the row sum procedure is not optimal. Thus, the model just described seems to constitute the natural domain of the row sum ranking procedure. Fortunately, this domain contains the important case where the $x_{ij} (i < j)$ are independent normal variables with mean $\vartheta_i - \vartheta_j$ and equal variance $\sigma^2$. Another particular case--the case of tournaments without draws--where the $x_{ij}$ can take only two values, has been treated in the joint paper [1]. Incidentally, the present paper grew out of an attempt to cover the case of tournaments with draws (see example (iv) in Section 4 below). Although one cannot expect strict optimality properties, one may still ask whether the row sum procedure has "nice" (e.g., minimax) properties even outside the above model. A modest result in this direction is the following. Take a particular joint distribution of the $x_{ij}$ belonging to the above model, and consider the class of all joint distributions of the $x_{ij}$ that lead (up to permutations) to the same joint distribution of the $s_i$ as this particular one. Then this latter will be least favorable for the ranking problem, and the row sum procedure will have minimax properties in each such class (modified minimax principle of Wesler [2]). For instance, if the $x_{ij} (i < j)$ are independent normal variables with mean $\vartheta_{ij}$ and equal variance, then the joint distribution of the $s_i$ depends only on the $\vartheta_i = (1/n) \sum^n_{j=1} \vartheta_{ij}$, and the linear model $\vartheta'_{ij} = \vartheta_i - \vartheta_j$ is least favorable, if the $\vartheta_i$ are kept fixed (up to permutations). There is an immediate generalization of the results to the case of comparison in $k$-tuples instead of in pairs, as indicated in the last section of this paper.