Annals of Statistics

Higher order elicitability and Osband’s principle

Tobias Fissler and Johanna F. Ziegel

Full-text: Open access


A statistical functional, such as the mean or the median, is called elicitable if there is a scoring function or loss function such that the correct forecast of the functional is the unique minimizer of the expected score. Such scoring functions are called strictly consistent for the functional. The elicitability of a functional opens the possibility to compare competing forecasts and to rank them in terms of their realized scores. In this paper, we explore the notion of elicitability for multi-dimensional functionals and give both necessary and sufficient conditions for strictly consistent scoring functions. We cover the case of functionals with elicitable components, but we also show that one-dimensional functionals that are not elicitable can be a component of a higher order elicitable functional. In the case of the variance, this is a known result. However, an important result of this paper is that spectral risk measures with a spectral measure with finite support are jointly elicitable if one adds the “correct” quantiles. A direct consequence of applied interest is that the pair (Value at Risk, Expected Shortfall) is jointly elicitable under mild conditions that are usually fulfilled in risk management applications.

Article information

Ann. Statist., Volume 44, Number 4 (2016), 1680-1707.

Received: April 2015
Revised: January 2016
First available in Project Euclid: 7 July 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62C99: None of the above, but in this section
Secondary: 91B06: Decision theory [See also 62Cxx, 90B50, 91A35]

Consistency decision theory elicitability Expected Shortfall point forecasts propriety scoring functions scoring rules spectral risk measures Value at Risk


Fissler, Tobias; Ziegel, Johanna F. Higher order elicitability and Osband’s principle. Ann. Statist. 44 (2016), no. 4, 1680--1707. doi:10.1214/16-AOS1439.

Export citation


  • Abernethy, J. D. and Frongillo, R. M. (2012). A characterization of scoring rules for linear properties. In Proceedings of the 25th Annual Conference on Learning Theory. Journal of Machine Learning Research: Workshop and Conference Proceedings, Vol. 23 (S. Mannor, N. Srebro andR. C. Williamson, eds.) 27.1–27.13.
  • Acerbi, C. (2002). Spectral measures of risk: A coherent representation of subjective risk aversion. J. Bank. Financ. 26 1505–1518.
  • Acerbi, C. and Székely, B. (2014). Backtesting expected shortfall. Risk Mag. 27 76–81.
  • Artzner, P., Delbaen, F., Eber, J.-M. and Heath, D. (1999). Coherent measures of risk. Math. Finance 9 203–228.
  • Banerjee, A., Guo, X. and Wang, H. (2005). On the optimality of conditional expectation as a Bregman predictor. IEEE Trans. Inform. Theory 51 2664–2669.
  • Bellini, F. and Bignozzi, V. (2015). On elicitable risk measures. Quant. Finance 15 725–733.
  • Cont, R., Deguest, R. and Scandolo, G. (2010). Robustness and sensitivity analysis of risk measurement procedures. Quant. Finance 10 593–606.
  • Daníelsson, J., Embrechts, P., Goodhart, C., Keating, C., Muennich, F., Renault, O. and Shin, H. S. (2001). An academic response to Basel II. Special Paper No. 130.
  • Davis, M. H. A. (2016). Verification of internal risk measure estimates. Stat. Risk Model. To appear.
  • Dawid, A. P. and Sebastiani, P. (1999). Coherent dispersion criteria for optimal experimental design. Ann. Statist. 27 65–81.
  • Delbaen, F. (2012). Monetary Utility Functions. Osaka Univ. Press, Osaka.
  • Delbaen, F., Bellini, F., Bignozzi, V. and Ziegel, J. F. (2016). Risk measures with the CxLS property. Finance Stoch. 20 433–453.
  • Embrechts, P. and Hofert, M. (2014). Statistics and quantitative risk management for banking and insurance. Ann. Rev. Stat. Appl. 1.
  • Emmer, S., Kratz, M. and Tasche, D. (2015). What is the best risk measure in practice? A comparison of standard measures. J. Risk 18 31–60.
  • Engelberg, J., Manski, C. F. and Williams, J. (2009). Comparing the point predictions and subjective probability distributions of professional forecasters. J. Bus. Econom. Statist. 27 30–41.
  • Fissler, T. and Ziegel, J. F. (2016). Supplement to “Higher order elicitability and Osband’s principle.” DOI:10.1214/16-AOS1439SUPP.
  • Frongillo, R. and Kash, I. (2015). Vector-valued property elicitation. J. Mach. Learn. Res. Workshop Conf. Proc. 40 1–18.
  • Gneiting, T. (2011). Making and evaluating point forecasts. J. Amer. Statist. Assoc. 106 746–762.
  • Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102 359–378.
  • Grauert, H. and Fischer, W. (1978). Differential- und Integralrechnung. II: Differentialrechnung in Mehreren Veränderlichen Differentialgleichungen, corrected ed. Heidelberger Taschenbücher [Heidelberg Paperbacks] 36. Springer, Berlin.
  • Heinrich, C. (2014). The mode functional is not elicitable. Biometrika 101 245–251.
  • Huber, P. J. (1964). Robust estimation of a location parameter. Ann. Math. Statist. 35 73–101.
  • Jouini, E., Schachermayer, W. and Touzi, N. (2006). Law invariant risk measures have the Fatou property. In Advances in Mathematical Economics. Vol. 9. Adv. Math. Econ. 9 49–71. Springer, Tokyo.
  • Koenker, R. (2005). Quantile Regression. Econometric Society Monographs 38. Cambridge Univ. Press, Cambridge.
  • Kou, S. and Peng, X. (2014). On the measurement of economic tail risk. Preprint. Available at
  • Kou, S., Peng, X. and Heyde, C. C. (2013). External risk measures and Basel accords. Math. Oper. Res. 38 393–417.
  • Krätschmer, V., Schied, A. and Zähle, H. (2012). Qualitative and infinitesimal robustness of tail-dependent statistical functionals. J. Multivariate Anal. 103 35–47.
  • Krätschmer, V., Schied, A. and Zähle, H. (2015). Quasi-Hadamard differentiability of general risk functionals and its applications. Stat. Risk Model. 32 25–47.
  • Krätschmer, V., Schied, A. and Zähle, H. (2014). Comparative and qualitative robustness for law-invariant risk measures. Finance Stoch. 18 271–295.
  • Kusuoka, S. (2001). On law invariant coherent risk measures. In Advances in Mathematical Economics, Vol. 3. Adv. Math. Econ. 3 83–95. Springer, Tokyo.
  • Lambert, N. (2013). Elicitation and evaluation of statistical functionals. Preprint. Available at
  • Lambert, N., Pennock, D. M. and Shoham, Y. (2008). Eliciting properties of probability distributions. In Proceedings of the 9th ACM Conference on Electronic Commerce 129–138. ACM, Chicago, IL.
  • Murphy, A. H. and Daan, H. (1985). Forecast evaluation. In Probability, Statistics and Decision Making in the Atmospheric Sciences (A. H. Murphy and R. W. Katz, eds.) 379–437. Westview Press, Boulder, CO.
  • Newey, W. K. and Powell, J. L. (1987). Asymmetric least squares estimation and testing. Econometrica 55 819–847.
  • Osband, K. H. (1985). Providing incentives for better cost forecasting. Ph.D. thesis, Univ. California, Berkeley.
  • Osband, K. and Reichelstein, S. (1985). Information-eliciting compensation schemes. J. Public Econ. 27 107–115.
  • Savage, L. J. (1971). Elicitation of personal probabilities and expectations. J. Amer. Statist. Assoc. 66 783–801.
  • Steinwart, I., Pasin, C., Williamson, R. and Zhang, S. (2014). Elicitation and identification of properties. In Proceedings of the 27th Conference on Learning Theory. Journal of Machine Learning Research: Workshop and Conference Proceedings, Vol. 35 (M. F. Balcan andC. Szepesvari, eds.) 482–526.
  • Wang, R. and Ziegel, J. F. (2015). Elicitable distortion risk measures: A concise proof. Statist. Probab. Lett. 100 172–175.
  • Weber, S. (2006). Distribution-invariant risk measures, information, and dynamic consistency. Math. Finance 16 419–441.
  • Ziegel, J. F. (2014). Coherence and elicitability. Math. Finance. To appear.

Supplemental materials