Annals of Statistics

A General Method for Comparing Probability Assessors

Mark J. Schervish

Full-text: Open access


A probability assessor or forecaster is a person who assigns subjective probabilities to events which will eventually occur or not occur. There are two purposes for which one might wish to compare two forecasters. The first is to see who has given better forecasts in the past. The second is to decide who will give better forecasts in the future. A method of comparison suitable for the first purpose may not be suitable for the second and vice versa. A criterion called calibration has been suggested for comparing the forecasts of different forecasters. Calibration, in a frequency sense, is a function of long run (future) properties of forecasts and hence is not suitable for making comparisons in the present. A method for comparing forecasters based on past performance is the use of scoring rules. In this paper a general method for comparing forecasters after a finite number of trials is introduced. The general method is proven to include calculating all proper scoring rules as special cases. It also includes comparison of forecasters in all simple two-decision problems as special cases. The relationship between the general method and calibration is also explored. The general method is also translated into a method for deciding who will give better forecasts in the future. An example is given using weather forecasts.

Article information

Ann. Statist., Volume 17, Number 4 (1989), 1856-1879.

First available in Project Euclid: 12 April 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier


Primary: 62B15: Theory of statistical experiments
Secondary: 62C10: Bayesian problems; characterization of Bayes procedures

Calibration dominance forecasters loss functions refinement scoring rules sufficiency


Schervish, Mark J. A General Method for Comparing Probability Assessors. Ann. Statist. 17 (1989), no. 4, 1856--1879. doi:10.1214/aos/1176347398.

Export citation