Electronic Journal of Statistics

Forecast dominance testing via sign randomization

Werner Ehm and Fabian Krüger

Full-text: Open access

Abstract

We propose randomization tests of whether forecast 1 outperforms forecast 2 across a class of scoring functions. This hypothesis is of applied interest: While the prediction context often prescribes a certain class of scoring functions, it is typically hard to motivate a specific choice on statistical or substantive grounds. We investigate the asymptotic behavior of the test statistics under mild conditions, avoiding the need to assume particular dynamic properties of forecasts and realizations. The properties of the one-sided tests depend on a corresponding version of Anderson’s inequality, which we state as a conjecture of independent interest. Numerical experiments and a data example indicate that the tests have good size and power properties in practically relevant situations.

Article information

Source
Electron. J. Statist., Volume 12, Number 2 (2018), 3758-3793.

Dates
Received: November 2017
First available in Project Euclid: 28 November 2018

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1543374044

Digital Object Identifier
doi:10.1214/18-EJS1495

Subjects
Primary: 62G09: Resampling methods 62G10: Hypothesis testing 62M20: Prediction [See also 60G25]; filtering [See also 60G35, 93E10, 93E11]

Keywords
Comparative forecast evaluation hypothesis testing randomization

Rights
Creative Commons Attribution 4.0 International License.

Citation

Ehm, Werner; Krüger, Fabian. Forecast dominance testing via sign randomization. Electron. J. Statist. 12 (2018), no. 2, 3758--3793. doi:10.1214/18-EJS1495. https://projecteuclid.org/euclid.ejs/1543374044


Export citation

References

  • [1] Andersen, T. G., Bollerslev, T., Diebold, F. X., and Labys, P. (2003). Modeling and forecasting realized volatility., Econometrica 71, 579-625.
  • [2] Anderson, T. W. (1955). The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities., Proc. Amer. Math. Soc. 6 170-176.
  • [3] Azmoodeh, E., Sottinen, E., Viitasaari, L., and Yazigi, A. (2014). Necessary and sufficient conditions for Hölder continuity of Gaussian processes., Statist. Probab. Lett. 94 230-235.
  • [4] Bandi, F. M., and Corradi, V. (2014). Nonparametric nonstationary tests., Econometric Theory 30 127-149.
  • [5] Banerjee, A., Guo, X., and Wang, H. (2005). On the optimality of conditional expectation as a Bregman predictor., IEEE Trans. Inform. Theory 51 2664-2669.
  • [6] Bollerslev, T. (1986). Generalized autoregressive conditional heteroscdasticity., J. Econometrics 31 307-327.
  • [7] Burkholder, D. L. (1966). Martingale transforms., Ann. Math. Statist. 37 1494-2004.
  • [8] Chen, L. Y. H., and Röllin, A. (2013). Approximating dependent rare events., Bernoulli 19 1243-1267.
  • [9] Chung, E., and Romano, J. P. (2013). Exact and asymptotically robust permutation tests., Ann. Statist. 41 484-507.
  • [10] Corradi, V., and Swanson, N. R. (2006). The effect of data transformation on common cycles, cointegration and unit root tests. Monte Carlo results and a simple test., J. Econometrics 132 195-229.
  • [11] Das Gupta, S. (1976). A generalization of Anderson’s theorem on unimodal functions., Proc. Amer. Math. Soc. 60 85-91.
  • [12] Dedecker, J., Merlevède, F., and Rio, E. (2013). Strong approximation results for the empirical process of stationary sequences., Ann. Probab. 41 3658-3696.
  • [13] DelSole, T., and Tippett, M. K. (2014). Comparing forecast skill., Mon. Weather Rev. 142 4658-4678.
  • [14] Diebold, F. X., and Mariano, R. S. (1995). Comparing predictive accuracy., J. Bus. Econom. Statist. 13 253-263.
  • [15] Efron, B., and Hastie, T. (2016). Computer age statistical inference: Algorithms, evidence, and data science. Cambridge University Press, Cambridge.
  • [16] Ehm, W., Gneiting, T., Jordan, A., and Krüger, F. (2016). Of quantiles and expectiles: consistent scoring functions, Choquet representations, and forecast rankings (with discussion)., J. R. Stat. Soc. Ser. B. Stat. Methodol. 78 505-562.
  • [17] Elliott, G., Ghanem, D., and Krüger, F. (2016). Forecasting conditional probabilities of binary outcomes under misspecification., Rev. Econ. Stat. 98 742-755.
  • [18] Elliott, G., and Timmermann, A. (2016). Economic forecasting. Princeton University Press, Princeton.
  • [19] Fissler, T., and Ziegel, J. F. (2016). Higher order elicitability and Osband’s principle., Ann. Statist. 44 1680-1707.
  • [20] Giacomini, R., and White, H. (2006). Tests of conditional predictive ability., Econometrica 74 1545-1578.
  • [21] Gneiting, T. (2011). Making and evaluating point forecasts., J. Amer. Statist. Assoc. 106 746-762.
  • [22] Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation., J. Amer. Statist. Assoc. 102 359-378.
  • [23] Gneiting, T., and Ranjan, R. (2013). Combining predictive distributions., Electron. J. Stat. 7 1747-1782.
  • [24] Hansen, B. E. (1996). Inference when a nuisance parameter is not identified under the null hypothesis., Econometrica 64 413-430.
  • [25] Hansen, P. R. (2005). A test for superior predictive ability., J. Business Econ. Statist. 23 365-380.
  • [26] Holzmann, H., and Eulert, M. (2014). The role of the information set for forecasting with applications to risk management., Ann. Appl. Statist 8 595-621.
  • [27] Horn, R. A., and Johnson, C. R. (2013)., Matrix Analysis, 2nd ed. Cambridge University Press, Cambridge.
  • [28] Jin, S., Corradi, V., and Swanson, N. R. (2017). Robust forecast comparison., Econometric Theory 33 1306-1351.
  • [29] Jordan, A., and Krüger, F. (2016)., murphydiagram: Murphy Diagrams for Forecast Comparisons. R package, version 0.11, available at https://cran.r-project.org/web/packages/murphydiagram/index.html.
  • [30] Koenker, R. (2005)., Quantile Regression. Cambridge University Press.
  • [31] Koenker, R. (2017)., quantreg: Quantile Regression. R package, version 5.33, available at https://cran.r-project.org/web/packages/quantreg/index.html.
  • [32] Krüger, F. and Ziegel, J. F. (2018). Generic conditions for forecast dominance. Preprint, available at, https://arxiv.org/abs/1805.09902.
  • [33] Künsch, H. R. (1989). The jackknife abd the bootstrap for general stationary observations., Ann. Statist. 17 2356-2382.
  • [34] Lai, T. L., Gross, S. T., and Shen, D. B. (2011). Evaluating probability forecasts., Ann. Statist. 39 2356-2382.
  • [35] Mammen, E. (1993). Boostrap and wild bootstrap for high dimensional linear models., Ann. Statist. 21 255-285.
  • [36] McLeish, D. L. (1974). Dependent central limit theorems and invariance principles., Ann. Probab. 2 620-628.
  • [37] Merkle, E. C. and Steyvers, M. (2013). Choosing a strictly proper scoring rule., Decis. Anal. 10 292-304.
  • [38] Linton, O., Maasoumi, E., and Whang, Y.-J. (2005). Consistent testing for stochastic dominance under general sampling schemes., Rev. Econom. Stud. 72 735-765.
  • [39] Linton, O., Song, K., and Whang, Y.-J. (2010). An improved bootstrap test of stochastic dominance., J. Econometrics 154 108-202.
  • [40] McNeil, A., Frey, R., and Embrechts, P. (2015)., Quantitative Risk Management: Concepts, Techniques and Tools, revised ed. Princeton University Press, Princeton.
  • [41] Mudholkar, G. S. (1966). The integral of an invariant unimodal function over an invariant convex set–an inequality and applications., Proc. Amer. Math. Soc. 17 1327-1333.
  • [42] Newey, W. K., and Powell, J. L. (1987). Asymmetric least squares estimation and testing., Econometrica 55 819-847.
  • [43] Patton, A. J. (2011). Volatility forecast comparison using imperfect volatility proxies., J. Econometrics 160 246-256.
  • [44] Patton, A. J. (2016). Comparing possibly misspecified forecasts. Preprint, Duke, University.
  • [45] Pollard, D. (1990)., Empirical Processes: Theory and Applications. Institute of Mathematical Statistics, Hayward.
  • [46] R Core Team (2017)., R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna.
  • [47] Rio, E. (1993). Covariance inequalities for strongly mixing processes., Ann. Inst. Henri Poincaré 29 587-597.
  • [48] Rudebusch, G. D., and Williams, J. C. (2009). The puzzle of the enduring power of the yield curve., J. Bus. Econom. Stat. 27 492-503.
  • [49] Savage, L. J. (1971). Elicitation of personal probabilities and expectations., J. Amer. Statist. Assoc. 71 783-801.
  • [50] Schuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability measurement procedures., Psychometrika 31 125-145.
  • [51] Seillier-Moisewitsch, F., and Dawid, A. P. (1993). On testing the validity of sequential probability forecasts., J. Amer. Statist. Assoc. 88 355-359.
  • [52] Strähl, C., and Ziegel, J. (2017). Cross-calibration of probabilistic forecasts., Electron. J. Stat. 11 608-639.
  • [53] Trapani, L. (2016). Testing for (in)finite moments., J. Econometrics 191, 57-68.
  • [54] van der Vaart, A. W. (1998)., Asympotic Statistics. Cambridge University Press, Cambridge.
  • [55] Wu, W.B. (2008). Empirical processes of stationary sequences., Statist. Sinica 18 313-333.
  • [56] Yen, T.-J., and Yen, Y.-M. (2018). Testing forecast accuracy of expectiles and quantiles with the extremal consistent loss functions. Preprint, available at, https://arxiv.org/abs/1707.02048.
  • [57] Yoshihara, K.-I. (1979). Note on an almost sure invariance principle for some empirical processes., Yokohama Math. J. 27 105-110.
  • [58] Ziegel, J. F., Krüger, F., Jordan, A., and Fasciati, F. (2017). Murphy Diagrams: Forecast evaluation of Expected Shortfall. Preprint, available at, https://arxiv.org/abs/1705.04537.
  • [59] Ziegler, K. (1997). Functional central limit theorems for triangular arrays of function-indexed processes under uniformly integrable entropy conditions., J. Multivariate Anal. 62 233-272
  • [60] Žikeš, F., and Baruník, J. (2016). Semi-parametric conditional quantile models for financial returns and realized volatility., J. Financ. Economet. 14 185-226.