Translator Disclaimer
February 2023 Local scale invariance and robustness of proper scoring rules
David Bolin, Jonas Wallin
Author Affiliations +
Statist. Sci. 38(1): 140-159 (February 2023). DOI: 10.1214/22-STS864

Abstract

Averages of proper scoring rules are often used to rank probabilistic forecasts. In many cases, the individual terms in these averages are based on observations and forecasts from different distributions. We show that some of the most popular proper scoring rules, such as the continuous ranked probability score (CRPS), give more importance to observations with large uncertainty, which can lead to unintuitive rankings. To describe this issue, we define the concept of local scale invariance for scoring rules. A new class of generalized proper kernel scoring rules is derived and as a member of this class we propose the scaled CRPS (SCRPS). This new proper scoring rule is locally scale invariant and, therefore, works in the case of varying uncertainty. Like the CRPS, it is computationally available for output from ensemble forecasts, and does not require the ability to evaluate densities of forecasts.

We further define robustness of scoring rules, show why this also can be an important concept for average scores unless one is specifically interested in extremes, and derive new proper scoring rules that are robust against outliers. The theoretical findings are illustrated in three different applications from spatial statistics, stochastic volatility models and regression for count data.

Acknowledgments

The authors would like to acknowledge the Editors and the reviewers, as well as Håvard Rue, Finn Lindgren and Tilmann Gneiting for helpful comments and suggestions that greatly improved the manuscript.

Citation

Download Citation

David Bolin. Jonas Wallin. "Local scale invariance and robustness of proper scoring rules." Statist. Sci. 38 (1) 140 - 159, February 2023. https://doi.org/10.1214/22-STS864

Information

Published: February 2023
First available in Project Euclid: 28 October 2022

Digital Object Identifier: 10.1214/22-STS864

Keywords: forecast ranking , Model selection , probabilistic forecasting , spatial statistics

Rights: Copyright © 2023 Institute of Mathematical Statistics

JOURNAL ARTICLE
20 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.38 • No. 1 • February 2023
Back to Top