Open Access
Translator Disclaimer
2019 Why scoring functions cannot assess tail properties
Jonas R. Brehmer, Kirstin Strokorb
Electron. J. Statist. 13(2): 4015-4034 (2019). DOI: 10.1214/19-EJS1622


Motivated by the growing interest in sound forecast evaluation techniques with an emphasis on distribution tails rather than average behaviour, we investigate a fundamental question arising in this context: Can statistical features of distribution tails be elicitable, i.e. be the unique minimizer of an expected score? We demonstrate that expected scores are not suitable to distinguish genuine tail properties in a very strong sense. Specifically, we introduce the class of max-functionals, which contains key characteristics from extreme value theory, for instance the extreme value index. We show that its members fail to be elicitable and that their elicitation complexity is in fact infinite under mild regularity assumptions. Further we prove that, even if the information of a max-functional is reported via the entire distribution function, a proper scoring rule cannot separate max-functional values. These findings highlight the caution needed in forecast evaluation and statistical inference if relevant information is encoded by such functionals.


Download Citation

Jonas R. Brehmer. Kirstin Strokorb. "Why scoring functions cannot assess tail properties." Electron. J. Statist. 13 (2) 4015 - 4034, 2019.


Received: 1 May 2019; Published: 2019
First available in Project Euclid: 5 October 2019

zbMATH: 07116195
MathSciNet: MR4015787
Digital Object Identifier: 10.1214/19-EJS1622

Primary: 62C05 , 62G32
Secondary: 91B06

Keywords: consistency , elicitability , elicitation complexity , extreme value index , max-functional , proper scoring rule , scoring functions , tail equivalence


Vol.13 • No. 2 • 2019
Back to Top