Order-sensitivity and equivariance of scoring functions

Tobias Fissler; Johanna F. Ziegel

doi:10.1214/19-EJS1552

2019 Order-sensitivity and equivariance of scoring functions

Tobias Fissler, Johanna F. Ziegel

Electron. J. Statist. 13(1): 1166-1211 (2019). DOI: 10.1214/19-EJS1552

Abstract

The relative performance of competing point forecasts is usually measured in terms of loss or scoring functions. It is widely accepted that these scoring function should be strictly consistent in the sense that the expected score is minimized by the correctly specified forecast for a certain statistical functional such as the mean, median, or a certain risk measure. Thus, strict consistency opens the way to meaningful forecast comparison, but is also important in regression and M-estimation. Usually strictly consistent scoring functions for an elicitable functional are not unique. To give guidance on the choice of a scoring function, this paper introduces two additional quality criteria. Order-sensitivity opens the possibility to compare two deliberately misspecified forecasts given that the forecasts are ordered in a certain sense. On the other hand, equivariant scoring functions obey similar equivariance properties as the functional at hand – such as translation invariance or positive homogeneity. In our study, we consider scoring functions for popular functionals, putting special emphasis on vector-valued functionals, e.g. the pair (mean, variance) or (Value at Risk, Expected Shortfall).

References

1.

Abernethy, J. D. and Frongillo, R. (2012). A Characterization of Scoring Rules for Linear Properties. In, Proceedings of the 25th Annual Conference on Learning Theory. Proceedings of Machine Learning Research 23 27.1–27.13. PMLR, Edinburgh, Scotland.Abernethy, J. D. and Frongillo, R. (2012). A Characterization of Scoring Rules for Linear Properties. In, Proceedings of the 25th Annual Conference on Learning Theory. Proceedings of Machine Learning Research 23 27.1–27.13. PMLR, Edinburgh, Scotland.

2.

Acerbi, C. and Szekely, B. (2014). Backtesting Expected Shortfall., Risk Magazine 27 76–81.Acerbi, C. and Szekely, B. (2014). Backtesting Expected Shortfall., Risk Magazine 27 76–81.

3.

Aliprantis, C. D. and Border, K. C. (2006)., Infinite Dimensional Analysis: A Hitchhiker’s Guide, 3rd ed. Springer, Berlin Heidelberg New York. 1156.46001Aliprantis, C. D. and Border, K. C. (2006)., Infinite Dimensional Analysis: A Hitchhiker’s Guide, 3rd ed. Springer, Berlin Heidelberg New York. 1156.46001

4.

Banerjee, A., Guo, X. and Wang, H. (2005). On the Optimality of Conditional Expectation as a Bregman Predictor., IEEE Transactions on Information Theory 51 2664–2669. 1284.94025 10.1109/TIT.2005.850145Banerjee, A., Guo, X. and Wang, H. (2005). On the Optimality of Conditional Expectation as a Bregman Predictor., IEEE Transactions on Information Theory 51 2664–2669. 1284.94025 10.1109/TIT.2005.850145

5.

Bellini, F. and Bignozzi, V. (2015). On elicitable risk measures., Quantitative Finance 15 725–733. 1395.91506 10.1080/14697688.2014.946955Bellini, F. and Bignozzi, V. (2015). On elicitable risk measures., Quantitative Finance 15 725–733. 1395.91506 10.1080/14697688.2014.946955

6.

Brehmer, J. R. (2017). Elicitability and its Application in Risk Management, Master’s thesis, University of, Mannheim.Brehmer, J. R. (2017). Elicitability and its Application in Risk Management, Master’s thesis, University of, Mannheim.

7.

Diebold, F. X. and Mariano, R. S. (1995). Comparing predictive accuracy., Journal of Business and Economic Statistics 13 253–263.Diebold, F. X. and Mariano, R. S. (1995). Comparing predictive accuracy., Journal of Business and Economic Statistics 13 253–263.

8.

Ehm, W., Gneiting, T., Jordan, A. and Krüger, F. (2016). Of quantiles and expectiles: Consistent scoring functions, Choquet representations and forecast rankings., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78 505–562. 07064964 10.1111/rssb.12154Ehm, W., Gneiting, T., Jordan, A. and Krüger, F. (2016). Of quantiles and expectiles: Consistent scoring functions, Choquet representations and forecast rankings., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78 505–562. 07064964 10.1111/rssb.12154

9.

Embrechts, P. and Hofert, M. (2014). Statistics and Quantitative Risk Management for Banking and Insurance., Annual Review of Statistics and Its Application 1 493–514.Embrechts, P. and Hofert, M. (2014). Statistics and Quantitative Risk Management for Banking and Insurance., Annual Review of Statistics and Its Application 1 493–514.

10.

Embrechts, P., Puccetti, G., Rüschendorf, L., Wang, R. and Beleraj, A. (2014). An Academic Response to Basel 3.5., Risks 2 25–48.Embrechts, P., Puccetti, G., Rüschendorf, L., Wang, R. and Beleraj, A. (2014). An Academic Response to Basel 3.5., Risks 2 25–48.

11.

Fissler, T. (2017). On Higher Order Elicitability and Some Limit Theorems on the Poisson and Wiener Space, PhD thesis, University of, Bern.Fissler, T. (2017). On Higher Order Elicitability and Some Limit Theorems on the Poisson and Wiener Space, PhD thesis, University of, Bern.

12.

Fissler, T. and Ziegel, J. F. (2016). Higher order elicitability and Osband’s principle., Annals of Statistics 44 1680–1707. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712Fissler, T. and Ziegel, J. F. (2016). Higher order elicitability and Osband’s principle., Annals of Statistics 44 1680–1707. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712

13.

Fissler, T. and Ziegel, J. F. (2019). Erratum: Higher Order Elicitability and Osband’s Principle., arXiv 1901.08826v1. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712Fissler, T. and Ziegel, J. F. (2019). Erratum: Higher Order Elicitability and Osband’s Principle., arXiv 1901.08826v1. 1355.62006 10.1214/16-AOS1439 euclid.aos/1467894712

14.

Friedman, D. (1983). Effective Scoring Rules for Probabilistic Forecasts., Management Science 29 447–454. 0515.62011 10.1287/mnsc.29.4.447Friedman, D. (1983). Effective Scoring Rules for Probabilistic Forecasts., Management Science 29 447–454. 0515.62011 10.1287/mnsc.29.4.447

15.

Frongillo, R. and Kash, I. (2015a). Vector-Valued Property Elicitation. In, Proceedings of The 28th Conference on Learning Theory. Proceedings of Machine Learning Research 40 710–727. PMLR, Paris, France.Frongillo, R. and Kash, I. (2015a). Vector-Valued Property Elicitation. In, Proceedings of The 28th Conference on Learning Theory. Proceedings of Machine Learning Research 40 710–727. PMLR, Paris, France.

16.

Frongillo, R. and Kash, I. (2015b). On Elicitation Complexity. In, Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.) 3258–3266. Curran Associates, Inc.Frongillo, R. and Kash, I. (2015b). On Elicitation Complexity. In, Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.) 3258–3266. Curran Associates, Inc.

17.

Gneiting, T. (2011). Making and Evaluating Point Forecasts., Journal of the American Statistical Association 106 746–762. 1232.62028 10.1198/jasa.2011.r10138Gneiting, T. (2011). Making and Evaluating Point Forecasts., Journal of the American Statistical Association 106 746–762. 1232.62028 10.1198/jasa.2011.r10138

18.

Heinrich, C. (2014). The mode functional is not elicitable., Biometrika 101 245–251. 1400.62026 10.1093/biomet/ast048Heinrich, C. (2014). The mode functional is not elicitable., Biometrika 101 245–251. 1400.62026 10.1093/biomet/ast048

19.

Huber, P. J. (1964). Robust Estimation of a Location Parameter., Annals of Mathematical Statistics 73–101. 0136.39805 10.1214/aoms/1177703732 euclid.aoms/1177703732Huber, P. J. (1964). Robust Estimation of a Location Parameter., Annals of Mathematical Statistics 73–101. 0136.39805 10.1214/aoms/1177703732 euclid.aoms/1177703732

20.

Huber, P. J. and Ronchetti, E. M. (2009)., Robust Statistics, 2nd ed. John Wiley & Sons, Inc., Hoboken, New Jersey. 1276.62022Huber, P. J. and Ronchetti, E. M. (2009)., Robust Statistics, 2nd ed. John Wiley & Sons, Inc., Hoboken, New Jersey. 1276.62022

21.

Koenker, R. (2005)., Quantile Regression. Cambridge University Press, Cambridge. 1111.62037Koenker, R. (2005)., Quantile Regression. Cambridge University Press, Cambridge. 1111.62037

22.

Königsberger, K. (2004)., Analysis 2, 5th ed. Springer-Verlag, Berlin Heidelberg New York.Königsberger, K. (2004)., Analysis 2, 5th ed. Springer-Verlag, Berlin Heidelberg New York.

23.

Lambert, N. (2013). Elicitation and Evaluation of Statistical Functionals., Preprint.Lambert, N. (2013). Elicitation and Evaluation of Statistical Functionals., Preprint.

24.

Lambert, N., Pennock, D. M. and Shoham, Y. (2008). Eliciting properties of probability distributions. In, Proceedings of the 9th ACM Conference on Electronic Commerce 129–138. ACM, Chicago, Il, USA.Lambert, N., Pennock, D. M. and Shoham, Y. (2008). Eliciting properties of probability distributions. In, Proceedings of the 9th ACM Conference on Electronic Commerce 129–138. ACM, Chicago, Il, USA.

25.

Lehmann, E. L. and Casella, G. (1998)., Theory of Point Estimation, 2nd ed. Springer Science & Business Media, New York.Lehmann, E. L. and Casella, G. (1998)., Theory of Point Estimation, 2nd ed. Springer Science & Business Media, New York.

26.

Murphy, A. H. and Daan, H. (1985). Forecast Evaluation. In, Probability, Statistics and Decision Making in the Atmospheric Sciences (A. H. Murphy and R. W. Katz, eds.) 379–437. Westview Press, Boulder, Colorado.Murphy, A. H. and Daan, H. (1985). Forecast Evaluation. In, Probability, Statistics and Decision Making in the Atmospheric Sciences (A. H. Murphy and R. W. Katz, eds.) 379–437. Westview Press, Boulder, Colorado.

27.

Nau, R. F. (1985). Should Scoring Rules Be ‘Effective’?, Management Science 31 527–535. 0608.60041 10.1287/mnsc.31.5.527Nau, R. F. (1985). Should Scoring Rules Be ‘Effective’?, Management Science 31 527–535. 0608.60041 10.1287/mnsc.31.5.527

28.

Newey, W. K. and Powell, J. L. (1987). Asymmetric Least Squares Estimation and Testing., Econometrica 55 819–847. 0625.62047 10.2307/1911031Newey, W. K. and Powell, J. L. (1987). Asymmetric Least Squares Estimation and Testing., Econometrica 55 819–847. 0625.62047 10.2307/1911031

29.

Nolde, N. and Ziegel, J. F. (2017). Elicitability and backtesting: Perspectives for banking regulation., Annals of Applied Statistics. To appear. 1383.62247 10.1214/17-AOAS1041 euclid.aoas/1514430265Nolde, N. and Ziegel, J. F. (2017). Elicitability and backtesting: Perspectives for banking regulation., Annals of Applied Statistics. To appear. 1383.62247 10.1214/17-AOAS1041 euclid.aoas/1514430265

30.

Osband, K. H. (1985). Providing Incentives for Better Cost Forecasting, PhD thesis, University of California, Berkeley.Osband, K. H. (1985). Providing Incentives for Better Cost Forecasting, PhD thesis, University of California, Berkeley.

31.

Patton, A. J. (2011). Data-based ranking of realised volatility estimators., Journal of Econometrics 161 284–303. 06610694 10.1016/j.jeconom.2010.12.010Patton, A. J. (2011). Data-based ranking of realised volatility estimators., Journal of Econometrics 161 284–303. 06610694 10.1016/j.jeconom.2010.12.010

32.

Patton, A. J. (2017). Comparing Possibly Misspecified Forecasts. Working paper, Duke, University.Patton, A. J. (2017). Comparing Possibly Misspecified Forecasts. Working paper, Duke, University.

33.

Reichelstein, S. and Osband, K. (1984). Incentives in government contracts., Journal of Public Economics 24 257–270.Reichelstein, S. and Osband, K. (1984). Incentives in government contracts., Journal of Public Economics 24 257–270.

34.

Saerens, M. (2000). Building cost functions minimizing to some summary statistics., IEEE Transactions on Neural Networks 11 1263–1271.Saerens, M. (2000). Building cost functions minimizing to some summary statistics., IEEE Transactions on Neural Networks 11 1263–1271.

35.

Savage, L. J. (1971). Elicitation of Personal Probabilities and Expectations., Journal of the American Statistical Association 66 783–801. 0253.92008 10.1080/01621459.1971.10482346Savage, L. J. (1971). Elicitation of Personal Probabilities and Expectations., Journal of the American Statistical Association 66 783–801. 0253.92008 10.1080/01621459.1971.10482346

36.

Scott, J. and Marshall, G. (2009)., A Dictionary of Sociology, 3 rev. ed. Oxford University Press.Scott, J. and Marshall, G. (2009)., A Dictionary of Sociology, 3 rev. ed. Oxford University Press.

37.

Steinwart, I. (2007). How to Compare Different Loss Functions and Their Risks., Constructive Approximation 26 225–287. 1127.68089 10.1007/s00365-006-0662-3Steinwart, I. (2007). How to Compare Different Loss Functions and Their Risks., Constructive Approximation 26 225–287. 1127.68089 10.1007/s00365-006-0662-3

38.

Steinwart, I., Pasin, C., Williamson, R. and Zhang, S. (2014). Elicitation and Identification of Properties. In, Proceedings of The 27th Conference on Learning Theory. Proceedings of Machine Learning Research 35 482–526. PMLR, Barcelona, Spain.Steinwart, I., Pasin, C., Williamson, R. and Zhang, S. (2014). Elicitation and Identification of Properties. In, Proceedings of The 27th Conference on Learning Theory. Proceedings of Machine Learning Research 35 482–526. PMLR, Barcelona, Spain.

39.

Thomson, W. (1979). Eliciting production possibilities from a well-informed manager., Journal of Economic Theory 20 360–380. 0414.90055 10.1016/0022-0531(79)90042-5Thomson, W. (1979). Eliciting production possibilities from a well-informed manager., Journal of Economic Theory 20 360–380. 0414.90055 10.1016/0022-0531(79)90042-5

40.

van der Vaart, A. W. (1998)., Asymptotic Statistics. Cambridge University Press, Cambridge. 0910.62001van der Vaart, A. W. (1998)., Asymptotic Statistics. Cambridge University Press, Cambridge. 0910.62001

41.

Weber, S. (2006). Distribution-Invariant Risk Measures, Information, and Dynamic Consistency., Mathematical Finance 16 419–441. 1145.91037 10.1111/j.1467-9965.2006.00277.xWeber, S. (2006). Distribution-Invariant Risk Measures, Information, and Dynamic Consistency., Mathematical Finance 16 419–441. 1145.91037 10.1111/j.1467-9965.2006.00277.x

Creative Commons Attribution 4.0 International License.

Citation Download Citation

Tobias Fissler and Johanna F. Ziegel "Order-sensitivity and equivariance of scoring functions," Electronic Journal of Statistics 13(1), 1166-1211, (2019). https://doi.org/10.1214/19-EJS1552

Received: 1 November 2017; Published: 2019

Access the abstract

JOURNAL ARTICLE
46 PAGES

DOWNLOAD PDF + SAVE TO MY LIBRARY