The Scoring of Multiple Choice Questionnaires

Abstract

In many types of questionnaires the element of guessing is important. An approach to correcting for guessing is proposed. Here one regards the score assigned to a subject on a question as an estimate of the unknown value of the subject's knowledge on the question. This value is one when the subject knows the answer and is zero otherwise. To derive scores with minimum mean squared error, it is necessary to consider the responses of the whole population of subjects. Thus the score for a correct answer to a question depends on the proportion $p$ of correct answers in the population. In the simplest model, we assume that a subject who knows the answer, responds correctly and that others select a response at random among $r$ choices. Then an incorrect response is scored zero and a correct one is assigned a score of $\lambda/p$ where $\lambda$ is the proportion of the population who knew the answer and can be estimated using the relation $p = \lambda + (1 - \lambda)/r$. The general approach is also applied to a variety of more complicated models, each of which are examples of a specified general formulation. These models include the "pairs of questions" model, the "partial knowledge" model and the "scaled questions" model. While the method applies neatly to single questions, there are fundamental difficulties in extending it to obtain an overall or composite score for a subject on an examination consisting of many questions. This problem is discussed briefly and the simple minded procedure of totaling the scores for the individual questions is partially evaluated.