Rank inversions in scoring multipart examinations

Abstract

Let $(X_i, Y_i), i = 1, \dots, n$, be independent random vectors with a standard bivariate normal distribution and let $s_X$ and $s_Y$ be the sample standard deviations. For arbitrary $p, 0 < p < 1$, define $T_i = pX_i + (1 - p) Y_i$ and $Z_i = pX_i / s_X + (1 - p) Y_i / s_Y, i = 1, \dots, n$. The couple of pairs $(T_i, Z_i)$ and $(T_j, Z_j)$ is said to be discordant if either $T_i < T_j$ and $Z_i > Z_j$ or $T_i > T_j$ and $Z_i < Z_j$. It is shown that the expected number of discordant couples of pairs is asymptotically equal to $n^{3/2}$ times an explicit constant depending on p and the correlation coefficient of $X_i$ and $Y_i$. By an application of the Durbin-Stuart inequality, this implies an asymptotic lower bound on the expected value of the sum of $(\rank(Z_i) - \rank(T_i))^+$. The problem arose in a court challenge to a standard procedure for the scoring of multipart written civil service examinations. Here the sum of the positive rank differences represents a measure of the unfairness of the method of scoring.