The relative performance of competing point forecasts is usually measured in terms of loss or scoring functions. It is widely accepted that these scoring function should be strictly consistent in the sense that the expected score is minimized by the correctly specified forecast for a certain statistical functional such as the mean, median, or a certain risk measure. Thus, strict consistency opens the way to meaningful forecast comparison, but is also important in regression and M-estimation. Usually strictly consistent scoring functions for an elicitable functional are not unique. To give guidance on the choice of a scoring function, this paper introduces two additional quality criteria. Order-sensitivity opens the possibility to compare two deliberately misspecified forecasts given that the forecasts are ordered in a certain sense. On the other hand, equivariant scoring functions obey similar equivariance properties as the functional at hand – such as translation invariance or positive homogeneity. In our study, we consider scoring functions for popular functionals, putting special emphasis on vector-valued functionals, e.g. the pair (mean, variance) or (Value at Risk, Expected Shortfall).
Electron. J. Statist.
13(1):
1166-1211
(2019).
DOI: 10.1214/19-EJS1552
Abernethy, J. D. and Frongillo, R. (2012). A Characterization of Scoring Rules for Linear Properties. In, Proceedings of the 25th Annual Conference on Learning Theory. Proceedings of Machine Learning Research 23 27.1–27.13. PMLR, Edinburgh, Scotland.Abernethy, J. D. and Frongillo, R. (2012). A Characterization of Scoring Rules for Linear Properties. In, Proceedings of the 25th Annual Conference on Learning Theory. Proceedings of Machine Learning Research 23 27.1–27.13. PMLR, Edinburgh, Scotland.
Aliprantis, C. D. and Border, K. C. (2006)., Infinite Dimensional Analysis: A Hitchhiker’s Guide, 3rd ed. Springer, Berlin Heidelberg New York. 1156.46001Aliprantis, C. D. and Border, K. C. (2006)., Infinite Dimensional Analysis: A Hitchhiker’s Guide, 3rd ed. Springer, Berlin Heidelberg New York. 1156.46001
Banerjee, A., Guo, X. and Wang, H. (2005). On the Optimality of Conditional Expectation as a Bregman Predictor., IEEE Transactions on Information Theory 51 2664–2669. 1284.94025 10.1109/TIT.2005.850145Banerjee, A., Guo, X. and Wang, H. (2005). On the Optimality of Conditional Expectation as a Bregman Predictor., IEEE Transactions on Information Theory 51 2664–2669. 1284.94025 10.1109/TIT.2005.850145
Bellini, F. and Bignozzi, V. (2015). On elicitable risk measures., Quantitative Finance 15 725–733. 1395.91506 10.1080/14697688.2014.946955Bellini, F. and Bignozzi, V. (2015). On elicitable risk measures., Quantitative Finance 15 725–733. 1395.91506 10.1080/14697688.2014.946955
Ehm, W., Gneiting, T., Jordan, A. and Krüger, F. (2016). Of quantiles and expectiles: Consistent scoring functions, Choquet representations and forecast rankings., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78 505–562. 07064964 10.1111/rssb.12154Ehm, W., Gneiting, T., Jordan, A. and Krüger, F. (2016). Of quantiles and expectiles: Consistent scoring functions, Choquet representations and forecast rankings., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78 505–562. 07064964 10.1111/rssb.12154
Embrechts, P. and Hofert, M. (2014). Statistics and Quantitative Risk Management for Banking and Insurance., Annual Review of Statistics and Its Application 1 493–514.Embrechts, P. and Hofert, M. (2014). Statistics and Quantitative Risk Management for Banking and Insurance., Annual Review of Statistics and Its Application 1 493–514.
Fissler, T. and Ziegel, J. F. (2016). Higher order elicitability and Osband’s principle., Annals of Statistics 44 1680–1707. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712Fissler, T. and Ziegel, J. F. (2016). Higher order elicitability and Osband’s principle., Annals of Statistics 44 1680–1707. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712
Fissler, T. and Ziegel, J. F. (2019). Erratum: Higher Order Elicitability and Osband’s Principle., arXiv 1901.08826v1. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712Fissler, T. and Ziegel, J. F. (2019). Erratum: Higher Order Elicitability and Osband’s Principle., arXiv 1901.08826v1. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712
Friedman, D. (1983). Effective Scoring Rules for Probabilistic Forecasts., Management Science 29 447–454. 0515.62011 10.1287/mnsc.29.4.447Friedman, D. (1983). Effective Scoring Rules for Probabilistic Forecasts., Management Science 29 447–454. 0515.62011 10.1287/mnsc.29.4.447
Frongillo, R. and Kash, I. (2015a). Vector-Valued Property Elicitation. In, Proceedings of The 28th Conference on Learning Theory. Proceedings of Machine Learning Research 40 710–727. PMLR, Paris, France.Frongillo, R. and Kash, I. (2015a). Vector-Valued Property Elicitation. In, Proceedings of The 28th Conference on Learning Theory. Proceedings of Machine Learning Research 40 710–727. PMLR, Paris, France.
Frongillo, R. and Kash, I. (2015b). On Elicitation Complexity. In, Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.) 3258–3266. Curran Associates, Inc.Frongillo, R. and Kash, I. (2015b). On Elicitation Complexity. In, Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.) 3258–3266. Curran Associates, Inc.
Gneiting, T. (2011). Making and Evaluating Point Forecasts., Journal of the American Statistical Association 106 746–762. 1232.62028 10.1198/jasa.2011.r10138Gneiting, T. (2011). Making and Evaluating Point Forecasts., Journal of the American Statistical Association 106 746–762. 1232.62028 10.1198/jasa.2011.r10138
Heinrich, C. (2014). The mode functional is not elicitable., Biometrika 101 245–251. 1400.62026 10.1093/biomet/ast048Heinrich, C. (2014). The mode functional is not elicitable., Biometrika 101 245–251. 1400.62026 10.1093/biomet/ast048
Huber, P. J. (1964). Robust Estimation of a Location Parameter., Annals of Mathematical Statistics 73–101. 0136.39805 10.1214/aoms/1177703732 euclid.aoms/1177703732Huber, P. J. (1964). Robust Estimation of a Location Parameter., Annals of Mathematical Statistics 73–101. 0136.39805 10.1214/aoms/1177703732 euclid.aoms/1177703732
Huber, P. J. and Ronchetti, E. M. (2009)., Robust Statistics, 2nd ed. John Wiley & Sons, Inc., Hoboken, New Jersey. 1276.62022Huber, P. J. and Ronchetti, E. M. (2009)., Robust Statistics, 2nd ed. John Wiley & Sons, Inc., Hoboken, New Jersey. 1276.62022
Koenker, R. (2005)., Quantile Regression. Cambridge University Press, Cambridge. 1111.62037Koenker, R. (2005)., Quantile Regression. Cambridge University Press, Cambridge. 1111.62037
Lambert, N., Pennock, D. M. and Shoham, Y. (2008). Eliciting properties of probability distributions. In, Proceedings of the 9th ACM Conference on Electronic Commerce 129–138. ACM, Chicago, Il, USA.Lambert, N., Pennock, D. M. and Shoham, Y. (2008). Eliciting properties of probability distributions. In, Proceedings of the 9th ACM Conference on Electronic Commerce 129–138. ACM, Chicago, Il, USA.
Murphy, A. H. and Daan, H. (1985). Forecast Evaluation. In, Probability, Statistics and Decision Making in the Atmospheric Sciences (A. H. Murphy and R. W. Katz, eds.) 379–437. Westview Press, Boulder, Colorado.Murphy, A. H. and Daan, H. (1985). Forecast Evaluation. In, Probability, Statistics and Decision Making in the Atmospheric Sciences (A. H. Murphy and R. W. Katz, eds.) 379–437. Westview Press, Boulder, Colorado.
Nau, R. F. (1985). Should Scoring Rules Be ‘Effective’?, Management Science 31 527–535. 0608.60041 10.1287/mnsc.31.5.527Nau, R. F. (1985). Should Scoring Rules Be ‘Effective’?, Management Science 31 527–535. 0608.60041 10.1287/mnsc.31.5.527
Newey, W. K. and Powell, J. L. (1987). Asymmetric Least Squares Estimation and Testing., Econometrica 55 819–847. 0625.62047 10.2307/1911031Newey, W. K. and Powell, J. L. (1987). Asymmetric Least Squares Estimation and Testing., Econometrica 55 819–847. 0625.62047 10.2307/1911031
Nolde, N. and Ziegel, J. F. (2017). Elicitability and backtesting: Perspectives for banking regulation., Annals of Applied Statistics. To appear. 1383.62247 10.1214/17-AOAS1041 euclid.aoas/1514430265Nolde, N. and Ziegel, J. F. (2017). Elicitability and backtesting: Perspectives for banking regulation., Annals of Applied Statistics. To appear. 1383.62247 10.1214/17-AOAS1041 euclid.aoas/1514430265
Patton, A. J. (2011). Data-based ranking of realised volatility estimators., Journal of Econometrics 161 284–303. 06610694 10.1016/j.jeconom.2010.12.010Patton, A. J. (2011). Data-based ranking of realised volatility estimators., Journal of Econometrics 161 284–303. 06610694 10.1016/j.jeconom.2010.12.010
Savage, L. J. (1971). Elicitation of Personal Probabilities and Expectations., Journal of the American Statistical Association 66 783–801. 0253.92008 10.1080/01621459.1971.10482346Savage, L. J. (1971). Elicitation of Personal Probabilities and Expectations., Journal of the American Statistical Association 66 783–801. 0253.92008 10.1080/01621459.1971.10482346
Steinwart, I. (2007). How to Compare Different Loss Functions and Their Risks., Constructive Approximation 26 225–287. 1127.68089 10.1007/s00365-006-0662-3Steinwart, I. (2007). How to Compare Different Loss Functions and Their Risks., Constructive Approximation 26 225–287. 1127.68089 10.1007/s00365-006-0662-3
Steinwart, I., Pasin, C., Williamson, R. and Zhang, S. (2014). Elicitation and Identification of Properties. In, Proceedings of The 27th Conference on Learning Theory. Proceedings of Machine Learning Research 35 482–526. PMLR, Barcelona, Spain.Steinwart, I., Pasin, C., Williamson, R. and Zhang, S. (2014). Elicitation and Identification of Properties. In, Proceedings of The 27th Conference on Learning Theory. Proceedings of Machine Learning Research 35 482–526. PMLR, Barcelona, Spain.
Thomson, W. (1979). Eliciting production possibilities from a well-informed manager., Journal of Economic Theory 20 360–380. 0414.90055 10.1016/0022-0531(79)90042-5Thomson, W. (1979). Eliciting production possibilities from a well-informed manager., Journal of Economic Theory 20 360–380. 0414.90055 10.1016/0022-0531(79)90042-5
van der Vaart, A. W. (1998)., Asymptotic Statistics. Cambridge University Press, Cambridge. 0910.62001van der Vaart, A. W. (1998)., Asymptotic Statistics. Cambridge University Press, Cambridge. 0910.62001
Weber, S. (2006). Distribution-Invariant Risk Measures, Information, and Dynamic Consistency., Mathematical Finance 16 419–441. 1145.91037 10.1111/j.1467-9965.2006.00277.xWeber, S. (2006). Distribution-Invariant Risk Measures, Information, and Dynamic Consistency., Mathematical Finance 16 419–441. 1145.91037 10.1111/j.1467-9965.2006.00277.x