Electronic Journal of Statistics

Combining predictive distributions

Tilmann Gneiting and Roopesh Ranjan

Full-text: Open access

Abstract

In probabilistic forecasting combination formulas for the aggregation of predictive distributions need to be estimated based on past experience and training data. We study combination formulas and aggregation methods for predictive cumulative distribution functions from the perspectives of calibration and dispersion, taking an original prediction space approach that applies to discrete, mixed discrete-continuous and continuous predictive distributions alike. The key idea is that aggregation methods ought to be parsimonious, yet sufficiently flexible to accommodate any type of dispersion in the component distributions. Both linear and non-linear aggregation methods are investigated, including generalized, spread-adjusted and beta-transformed linear pools. The effects and techniques are demonstrated theoretically, in simulation examples, and in case studies, where we fit combination formulas for density forecasts of S&P 500 returns and daily maximum temperature at Seattle-Tacoma Airport.

Article information

Source
Electron. J. Statist., Volume 7 (2013), 1747-1782.

Dates
First available in Project Euclid: 3 July 2013

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1372861687

Digital Object Identifier
doi:10.1214/13-EJS823

Mathematical Reviews number (MathSciNet)
MR3080409

Zentralblatt MATH identifier
1294.62220

Subjects
Primary: 62
Secondary: 91B06: Decision theory [See also 62Cxx, 90B50, 91A35]

Keywords
Beta transform conditional calibration density forecast flexibly dispersive forecast aggregation linear pool probability integral transform probabilistic calibration

Citation

Gneiting, Tilmann; Ranjan, Roopesh. Combining predictive distributions. Electron. J. Statist. 7 (2013), 1747--1782. doi:10.1214/13-EJS823. https://projecteuclid.org/euclid.ejs/1372861687


Export citation

References

  • Allard, A., Comunian, A. and Renard, P. (2012). Probability aggregation methods in geoscience., Mathematical Geosciences, 44, 545–581.
  • Arnold, H. M., Moroz, I. M. and Palmer, T. N. (2013). Stochastic parameterizations and model uncertainty in the Lorenz ’96 system., Philiosophical Transactions of the Royal Society Ser. A, 371, 20110479.
  • Berrocal, V. J., Raftery, A. E. and Gneiting, T. (2007). Combining spatial statistical and ensemble information for probabilistic weather forecasting., Monthly Weather Review, 135, 1386–1402.
  • Bollerslev, T. (1986). Generalized autoregressive conditional heteroscedasticity, Journal of Econometrics, 31, 307–327.
  • Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores., Quarterly Journal of the Royal Meteorological Society, 135, 1512–1519.
  • Clemen, R. T. and Winkler, R. L. (2007). Aggregating probability distributions. In Ward, E., Miles, R. F. and von Winterfeldt, D. (eds.), Advances in Decision Analysis: From Foundations to Applications, Cambridge University Press, pp. 154–176.
  • Clements, M. P. and Harvey, D. I. (2011). Combining probability forecasts., International Journal of Forecasting, 27, 208–223.
  • Cox, D. R. (1958). Two further applications of a model for binary regression., Biometrika, 45, 562–565.
  • Czado, C., Gneiting, T. and Held, L. (2009). Predictive model assessment for count data., Biometrics, 65, 1254–1261.
  • Dawid, A. P. (1984). Statistical theory: The prequential approach (with discussion and rejoinder)., Journal of the Royal Statistical Society Ser. A, 147, 278–290.
  • Dawid, A. P. (1986). Probability forecasting, in Kotz, S., Johnson, N. L. and Read, C. B. (eds.), Encyclopedia of Statistical Sciences, Vol. 7, Wiley, New York, pp. 210–218.
  • Dawid, A. P., DeGroot, M. H. and Mortera, J. (1995). Coherent combination of experts’ opinions (with discussion and rejoinder)., Test, 4, 263–313.
  • DelSole, T. and Shukla, J. (2010). Model fidelity versus skill in seasonal forecasting., Journal of Climate, 23, 4794–4806.
  • Diebold, F. X., Gunther, T. A. and Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management., International Economic Review, 39, 863–883.
  • Dutton, J. A. (2002). Opportunities and priorities in a new era for weather and climate services., Bulletin of the American Meteorological Society, 83, 1303–1311.
  • Eckel, F. A. and Mass, C. F. (2005). Aspects of effective short-range ensemble forecasting., Weather and Forecasting, 20, 328–350.
  • Ferguson, T. S. (1996)., A Course in Large Sample Theory. Chapman and Hall.
  • Fraley, C., Raftery, A. E., Gneiting, T., Sloughter, J. M. and Berrocal, V. J. (2011). Probabilistic weather forecasting in R., R Journal, 3(1), 55–63.
  • French, S. and Ríos Insua, D. (2000)., Statistical Decision Theory. Arnold, London.
  • Fricker, T. E., Ferro, C. A. T. and Stephenson, D. B. (2013). Three recommendations for evaluating climate predictions., Meteorological Applications, 20, 246–255.
  • Garratt, A., Mitchell, J., Vahey, S. P. and Wakerly, E. C. (2011). Real-time inflation forecast densities from ensemble Phillips curves., North American Journal of Economics and Finance, 22, 77–87.
  • Genest, C. (1984a). A characterization theorem for externally Bayesian groups., Annals of Statistics, 12, 1100–1105.
  • Genest, C. (1984b). A conflict between two axioms for combining subjective distributions., Journal of the Royal Statistical Society Ser. B, 46, 403–405.
  • Genest, C. and Zidek, J. (1986). Combining probability distributions: A critique and an annotated bibliography., Statistical Science, 1, 114–135.
  • Genest, C., McConway, K. J. and Schervish, M. J. (1986). Characterization of externally Bayesian pooling operators., Annals of Statistics, 14, 487–501.
  • Geweke, J. and Amisano, G. (2011). Optimal prediction pools., Journal of Econometrics, 164, 130–141.
  • Glahn, B., Peroutka, M., Wiedenfeld, J., Wagner, J., Zylstra, G., Schuknecht, B. and Jackson, B. (2009). MOS uncertainty estimates in an ensemble framework., Monthly Weather Review, 137, 246–268.
  • Gneiting, T. (2008). Editorial: Probabilistic forecasting., Journal of the Royal Statistical Society Ser. A, 171, 319–321.
  • Gneiting, T. and Raftery, A. E. (2005). Weather forecasting with ensemble methods., Science, 310, 248–249.
  • Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation., Journal of the American Statistical Association, 102, 359–378.
  • Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness., Journal of the Royal Statistical Society Ser. B, 69, 243–268.
  • Hall, S. G. and Mitchell, J. (2007). Combining density forecasts., International Journal of Forecasting, 23, 1–13.
  • Hamill, T. M. (2001). Interpretation of rank histograms for verifying ensemble forecasts., Monthly Weather Review, 129, 550–560.
  • Held, L., Rufibach, K. and Balabdaoui, F. (2010). A score regression approach to assess calibration of continuous probabilistic predictions., Biometrics, 66, 1295–1305.
  • Hora, S. C. (2004). Probability judgements for continuous quantities: Linear combinations and calibration., Management Science, 50, 597–604.
  • Hora, S. C. (2010). An analytic method for evaluating the performance of aggregation rules for probability densities., Operations Research, 58, 1440–1449.
  • Hoeting, J. A., Madigan, D., Raftery, A. E. and Volinsky, C. T. (1999) Bayesian model averaging: A tutorial (with discussion)., Statistal Science, 4, 382–417.
  • Jore, A. S., Mitchell, J. and Vahey, S. P. (2010). Combining forecast densities from VARs with uncertain stabilities., Journal of Applied Econometrics, 25, 621–634.
  • Jouini, M. N. and Clemen, R. T. (1996). Copula models for aggregating expert opinions., Operations Research, 44, 444–457.
  • Kascha, C. and Ravazzolo, F. (2010). Combining inflation density forecasts., Journal of Forecasting, 29, 231–250.
  • Kleiber, W., Raftery, A. E., Baars, J., Gneiting, T., Mass, C. F. and Grimit, E. (2011). Locally calibrated probabilistic temperature forecasting using geostatistical model averaging and local Bayesian model averaging., Monthly Weather Review, 139, 2630–2649.
  • Krüger, F. (2013). Jensen’s inequality and the success of linear prediction pools. Discussion paper, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2080010.
  • Luss, R., Rosset, S. and Shahar, M. (2012). Efficient regularized isotonic regression with application to gene-gene interaction search., Annals of Applied Statistics, 6, 253–283.
  • Madigan, D. and Raftery, A. E. (1994). Model selection and accounting for model uncertainty in graphical models using Occam’s window., Journal of the American Statistical Association, 89, 1535–1546.
  • Mason, S. J., Galpin, S., Goddard, L., Graham, N. E. and Rajaratnam, B. (2007). Conditional exceedance probabilities., Monthly Weather Review, 135, 363–372.
  • McConway, K. J. (1981). Marginalization and linear opinion pools., Journal of the American Statistical Association, 76, 410–414.
  • Mitchell, J. and Hall, S. G. (2005). Evaluating, comparing and combining density forecasts using the KLIC with an application to the Bank of England and NIESR ‘fan’ charts of inflation., Oxford Bulletin of Economics and Statistics, 67, 995–1033.
  • Murphy, A. H. and Winkler, R. L. (1987). A general framework for forecast verification., Monthly Weather Review, 115, 1330–1338.
  • Murphy, A. H. and Winkler, R. L. (1992). Diagnostic verification of probability forecasts., International Journal of Forecasting, 7, 435–455.
  • Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Smola, A., Bartlett, P., Schölkopf, B. and Schuurmans, D., eds., Advances in Large Margins Classifiers, MIT Press, pp. 61–74.
  • Primo, C., Ferro, C. A. T., Jolliffe, I. T. and Stephenson, D. B. (2009). Calibration of probabilistic forecasts of binary events., Monthly Weather Review, 137, 1142–1149.
  • Raftery, A. E., Gneiting, T., Balabdaoui, F. and Polakowski, M. (2005). Using Bayesian model averaging to calibrate forecast ensembles., Monthly Weather Review, 133, 1155–1174.
  • Ranjan, R. and Gneiting, T. (2010). Combining probability forecasts., Journal of the Royal Statistical Society Ser. B, 72, 71–91.
  • R Development Core Team (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, ISBN 3-900051-07-0, http://www.R-project.org.
  • Rosenblatt, M. (1952). Remarks on a multivariate transformation., Annals of Mathematical Statistics, 23, 470–472.
  • Rüschendorf, L. (1981). Stochastically ordered distributions and monotonicity of the OC-function of sequential probability ratio tests., Mathematische Operationsforschung Ser. Statistics, 12, 327–338.
  • Schervish, M. J. (1989). A general method for comparing probability assessors., Annals of Statistics, 17, 1856–1879.
  • Sloughter, J. M., Raftery, A. E., Gneiting, T. and Fraley, C. (2007). Probabilistic quantitative precipitation forecasting using Bayesian model averaging., Monthly Weather Review, 135, 3209–3220.
  • Stone, M. (1961). The linear pool., Annals of Mathematical Statistics, 32, 1339–1342.
  • Tsyplakov, A. (2011). Evaluating density forecasts: a comment. MPRA paper no. 31233, http://mpra.ub.uni-muenchen.de/32728.
  • Tsyplakov, A. (2013). Evaluation of probabilistic forecasts: proper scoring rules and moments. Preprint, http://dx.doi.org/10.2139/ssrn.2236605.
  • Wallis, K. F. (2005). Combining density and interval forecasts: A modest proposal., Oxford Bulletin of Economics and Statistics, 67, 983–994.
  • Winkler, R. L. (1968). The consensus of subjective probability distributions., Management Science, 15, B61–B75.
  • Winkler, R. L. (1986). Comment on “Combining probability distributions: A critique and an annotated bibliography”., Statistical Science, 1, 138–140.
  • Wuertz, D. and Rmetrics Core Team (2007). The fGarch Package. Reference manual, available at, http://www.mirrorservice.org/sites/lib.stat.cmu.edu/R/CRAN/doc/packages/f Garch.pdf.
  • Zadrozny, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp., 694–699.
  • Zarnowitz, V. (1969). The new ASA-NBER survey of forecasts by economic statisticians., American Statistician, 23, 12–16.