Statistical Science

Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling

Roman Schefzik, Thordis L. Thorarinsdottir, and Tilmann Gneiting

Full-text: Open access


Critical decisions frequently rely on high-dimensional output from complex computer simulation models that show intricate cross-variable, spatial and temporal dependence structures, with weather and climate predictions being key examples. There is a strongly increasing recognition of the need for uncertainty quantification in such settings, for which we propose and review a general multi-stage procedure called ensemble copula coupling (ECC), proceeding as follows:

1. Generate a raw ensemble, consisting of multiple runs of the computer model that differ in the inputs or model parameters in suitable ways.

2. Apply statistical postprocessing techniques, such as Bayesian model averaging or nonhomogeneous regression, to correct for systematic errors in the raw ensemble, to obtain calibrated and sharp predictive distributions for each univariate output variable individually.

3. Draw a sample from each postprocessed predictive distribution.

4. Rearrange the sampled values in the rank order structure of the raw ensemble to obtain the ECC postprocessed ensemble.

The use of ensembles and statistical postprocessing have become routine in weather forecasting over the past decade. We show that seemingly unrelated, recent advances can be interpreted, fused and consolidated within the framework of ECC, the common thread being the adoption of the empirical copula of the raw ensemble. Depending on the use of Quantiles, Random draws or Transformations at the sampling stage, we distinguish the ECC-Q, ECC-R and ECC-T variants, respectively. We also describe relations to the Schaake shuffle and extant copula-based techniques. In a case study, the ECC approach is applied to predictions of temperature, pressure, precipitation and wind over Germany, based on the 50-member European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble.

Article information

Statist. Sci., Volume 28, Number 4 (2013), 616-640.

First available in Project Euclid: 3 December 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Bayesian model averaging empirical copula ensemble calibration nonhomogeneous regression numerical weather prediction probabilistic forecast Schaake shuffle Sklar’s theorem


Schefzik, Roman; Thorarinsdottir, Thordis L.; Gneiting, Tilmann. Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling. Statist. Sci. 28 (2013), no. 4, 616--640. doi:10.1214/13-STS443.

Export citation


  • Aas, K., Czado, C., Frigessi, A. and Bakken, H. (2009). Pair-copula constructions of multiple dependence. Insurance Math. Econom. 44 182–198.
  • Bao, L., Gneiting, T., Grimit, E. P., Guttorp, P. and Raftery, A. E. (2010). Bias correction and Bayesian model averaging for ensemble forecasts of surface wind direction. Monthly Weather Review 138 1811–1821.
  • Ben Bouallègue, Z. (2013). Calibrated short-range ensemble precipitation forecasts using extended logistic regression with interaction terms. Weather and Forecasting 28 515–524.
  • Berrocal, V. J., Raftery, A. E. and Gneiting, T. (2007). Combining spatial statistical and ensemble information in probabilistic weather forecasts. Monthly Weather Review 135 1386–1402.
  • Berrocal, V. J., Raftery, A. E. and Gneiting, T. (2008). Probabilistic quantitative precipitation field forecasting using a two-stage spatial model. Ann. Appl. Stat. 2 1170–1193.
  • Berrocal, V. J., Raftery, A. E., Gneiting, T. and Steed, R. C. (2010). Probabilistic weather forecasting for winter road maintenance. J. Amer. Statist. Assoc. 105 522–537.
  • Bremnes, J. B. (2004). Probabilistic forecasts of precipitation in terms of quantiles using NWP model output. Monthly Weather Review 132 338–347.
  • Bremnes, J. B. (2007). Improved calibration of precipitation forecasts using ensemble techniques. Part 2: Statistical calibration methods. Technical Report 04/2007, Norwegian Meteorological Institute.
  • Bröcker, J. (2012). Evaluating raw ensembles with the continuous ranked probability score. Quarterly Journal of the Royal Meteorological Society 138 1611–1617.
  • Bröcker, J. and Smith, L. A. (2008). From ensemble forecasts to predictive distribution functions. Tellus Ser. A 60 663–678.
  • Chaloulos, G. and Lygeros, J. (2007). Effect of wind correlation on aircraft conflict probability. Journal of Guidance, Control, and Dynamics 30 1742–1752.
  • Chilès, J.-P. and Delfiner, P. (2012). Geostatistics: Modeling Spatial Uncertainty, 2nd ed. Wiley, Hoboken, NJ.
  • Chmielecki, R. M. and Raftery, A. E. (2011). Probabilistic visibility forecasting using Bayesian model averaging. Monthly Weather Review 139 1626–1636.
  • Clark, M., Gangopadhyay, S., Hay, L., Rajagopalan, B. and Wilby, R. (2004). The Schaake shuffle: A method for reconstructing space–time variability in forecasted precipitation and temperature fields. Journal of Hydrometeorology 5 243–262.
  • Cloke, H. L. and Pappenberger, F. (2009). Ensemble flood forecasting: A review. Journal of Hydrology 375 613–626.
  • Cooke, R. (1906). Forecasts and verifications in Western Australia. Monthly Weather Review 34 23–24.
  • Cressie, N. and Wikle, C. K. (2011). Statistics for Spatio-Temporal Data. Wiley, Hoboken, NJ.
  • Czado, C., Gneiting, T. and Held, L. (2009). Predictive model assessment for count data. Biometrics 65 1254–1261.
  • Davison, A. C., Padoan, S. A. and Ribatet, M. (2012). Statistical modeling of spatial extremes. Statist. Sci. 27 161–186.
  • Dawid, A. P. (1984). Statistical theory. The prequential approach. J. Roy. Statist. Soc. Ser. A 147 278–292.
  • Deheuvels, P. (1979). La fonction de dépendance empirique et ses propriétés. Un test non paramétrique d’indépendance. Acad. Roy. Belg. Bull. Cl. Sci. (5) 65 274–292.
  • Delle Monache, L., Hacker, J. P., Zhou, Y., Deng, X. and Stull, R. B. (2006). Probabilistic aspects of meteorological and ozone regional ensemble forecasts. Journal of Geophysical Research 111 D24307.
  • Demarta, S. and McNeil, A. J. (2005). The $t$ copula and related copulas. International Statistical Review 73 111–129.
  • Dette, H. and Volgushev, S. (2008). Non-crossing non-parametric estimates of quantile curves. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 609–627.
  • Di Narzo, A. F. and Cocchi, D. (2010). A Bayesian hierarchical approach to ensemble weather forecasting. J. R. Stat. Soc. Ser. C. Appl. Stat. 59 405–422.
  • Eckel, A. F. and Mass, C. F. (2005). Aspects of effective mesoscale, short-range ensemble forecasting. Weather and Forecasting 20 328–350.
  • Fermanian, J.-D., Radulović, D. and Wegkamp, M. (2004). Weak convergence of empirical copula processes. Bernoulli 10 847–860.
  • Flowerdew, J. (2012). Calibration and combination of medium-range ensemble precipitation forecasts. Technical Report 567, United Kingdom Met Office.
  • Fraley, C., Raftery, A. E. and Gneiting, T. (2010). Calibrating multi-model forecast ensembles with exchangeable and missing members using Bayesian model averaging. Monthly Weather Review 138 190–202.
  • Fraley, C., Raftery, A. E., Gneiting, T., Sloughter, J. M. and Berrocal, V. J. (2011). Probabilistic weather forecasting in R. R Journal 3 55–63.
  • Gel, Y., Raftery, A. E. and Gneiting, T. (2004). Calibrated probabilistic mesoscale weather field forecasting: The geostatistical output perturbation (GOP) method (with discussion and rejoinder). J. Amer. Statist. Assoc. 99 575–590.
  • Genest, C. and Favre, A. C. (2007). Everything you always wanted to know about copula modeling but were afraid to ask. Journal of Hydrologic Engineering 12 347–368.
  • Glahn, H. R., Peroutka, M., Weidenfeld, J., Wagner, J., Zylstra, G. and Schuknecht, B. (2009). MOS uncertainty estimates in an ensemble framework. Monthly Weather Review 137 246–268.
  • Gneiting, T. (2008). Editorial: Probabilistic forecasting. J. Roy. Statist. Soc. Ser. A 171 319–321.
  • Gneiting, T. (2011). Making and evaluating point forecasts. J. Amer. Statist. Assoc. 106 746–762.
  • Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 243–268.
  • Gneiting, T. and Raftery, A. E. (2005). Weather forecasting with ensemble methods. Science 310 248–249.
  • Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102 359–378.
  • Gneiting, T., Raftery, A. E., Westveld, A. H. and Goldman, T. (2005). Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Monthly Weather Review 133 1098–1118.
  • Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L. and Johnson, N. A. (2008). Assessing probabilistic forecasts of multivariate quantities, with applications to ensemble predictions of surface winds (with discussion and rejoinder). TEST 17 211–264.
  • Graf, S. and Luschgy, H. (2000). Foundations of Quantization for Probability Distributions. Lecture Notes in Math. 1730. Springer, Berlin.
  • Hagedorn, R., Hamill, T. M. and Whitaker, J. S. (2008). Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part I: Two-meter temperatures. Monthly Weather Review 136 2608–2619.
  • Hagedorn, R., Buizza, R., Hamill, T. M., Leutbecher, M. and Palmer, T. N. (2012). Comparing TIGGE multimodel forecasts with reforecast-calibrated ECMWF ensemble forecasts. Quarterly Journal of the Royal Meteorological Society 138 1814–1827.
  • Hamill, T. M. (2001). Interpretation of rank histograms for verifying ensemble forecasts. Monthly Weather Review 129 550–560.
  • Hamill, T. M. and Colucci, S. J. (1997). Verification of Eta-RSM short-range ensemble forecasts. Monthly Weather Review 125 1312–1327.
  • Hamill, T. M., Hagedorn, R. and Whitaker, J. S. (2008). Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part II: Precipitation. Monthly Weather Review 136 2620–2632.
  • Hamill, T. M., Whitaker, J. S. and Mullen, S. L. (2006). Reforecasts: An important dataset for improving weather predictions. Bulletin of the American Meteorological Society 87 33–46.
  • Joe, H. (1997). Multivariate Models and Dependence Concepts. Monographs on Statistics and Applied Probability 73. Chapman & Hall, London.
  • Kalnay, E. (2003). Atmospheric Modeling, Data Assimilation and Predictability. Cambridge Univ. Press, Cambridge.
  • Kann, A., Wittmann, C., Wang, Y. and Ma, X. (2009). Calibrating 2-m temperature of limited-area ensemble forecasts using high-resolution analysis. Monthly Weather Review 137 3373–3387.
  • Kleiber, W., Raftery, A. E. and Gneiting, T. (2011). Geostatistical model averaging for locally calibrated probabilistic quantitative precipitation forecasting. J. Amer. Statist. Assoc. 106 1291–1303.
  • Kleiber, W., Raftery, A. E., Baars, J., Gneiting, T., Mass, C. and Grimit, E. P. (2011). Locally calibrated probabilistic temperature foreasting using geostatistical model averaging and local Bayesian model averaging. Monthly Weather Review 139 2630–2649.
  • Kneib, T. (2013). Beyond mean regression (with discussion and rejoinder). Statistical Modelling 13 275–385.
  • Kolesárová, A., Mesiar, R., Mordelová, J. and Sempi, C. (2006). Discrete copulas. IEEE Transactions on Fuzzy Systems 14 698–705.
  • Krzysztofowicz, R. and Toth, Z. (2008). Bayesian processor of ensemble (BPE): Concept and implementation. Slides presented at the 4th NCEP/NWS Ensemble User Workshop, Laurel, MD.
  • Leutbecher, M. and Palmer, T. N. (2008). Ensemble forecasting. J. Comput. Phys. 227 3515–3539.
  • Mayor, G., Suñer, J. and Torrens, J. (2007). Sklar’s theorem in finite settings. IEEE Transactions on Fuzzy Systems 15 410–416.
  • McNeil, A. J., Frey, R. and Embrechts, P. (2005). Quantitative Risk Management: Concepts, Techniques and Tools. Princeton Univ. Press, Princeton, NJ.
  • McNeil, A. J. and Nešlehová, J. (2009). Multivariate Archimedean copulas, $d$-monotone functions and $l_{1}$-norm symmetric distributions. Ann. Statist. 37 3059–3097.
  • Möller, A., Lenkoski, A. and Thorarinsdottir, T. L. (2013). Multivariate probabilistic forecasting using ensemble Bayesian model averaging and copulas. Quarterly Journal of the Royal Meteorological Society 139 982–991.
  • Molteni, F., Buizza, R., Palmer, T. N. and Petroliagis, T. (1996). The new ECMWF ensemble prediction system: Methodology and validation. Quarterly Journal of the Royal Meteorological Society 122 73–119.
  • Nelsen, R. B. (2006). An Introduction to Copulas, 2nd ed. Springer, New York.
  • Palmer, T. N. (2002). The economic value of ensemble forecasts as a tool for risk assessment: From days to decades. Quarterly Journal of the Royal Meteorological Society 128 747–774.
  • Pinson, P. (2012). Adaptive calibration of $(u,v)$-wind ensemble forecasts. Quarterly Journal of the Royal Meteorological Society 138 1273–1284.
  • Pinson, P. (2013). Wind energy: Forecasting challenges for its operational management. Statist. Sci. 28 564–585.
  • Pinson, P., Madsen, H., Nielsen, H. A., Papaefthymiou, G. and Klöckl, B. (2009). From probabilistic forecasts to statistical scenarios of short-term wind power production. Wind Energy 12 51–62.
  • Raftery, A. E., Kárný, M. and Ettler, P. (2010). Online prediction under model uncertainty via dynamic model averaging: Application to a cold rolling mill. Technometrics 52 52–66.
  • Raftery, A. E., Gneiting, T., Balabdaoui, F. and Polakowski, M. (2005). Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Review 133 1155–1174.
  • Roquelaure, S. and Bergot, T. (2008). A local ensemble prediction system for fog and low clouds: Construction, Bayesian model averaging calibration, and validation. Journal of Applied Meteorology and Climatology 47 3072–3088.
  • Roulin, E. and Vannitsem, S. (2012). Postprocessing of ensemble precipitation predictions with extended logistic regression based on hindcasts. Monthly Weather Review 140 874–888.
  • Ruiz, J. J. and Saulo, C. (2012). How sensitive are probabilistic precipitation forecasts to the choice of calibration algorithms and the ensemble generation method? Part I: Sensitivity to calibration methods. Meteorological Applications 19 302–313.
  • Rüschendorf, L. (1976). Asymptotic distributions of multivariate rank order statistics. Ann. Statist. 4 912–923.
  • Rüschendorf, L. (2009). On the distributional transform, Sklar’s theorem, and the empirical copula process. J. Statist. Plann. Inference 139 3921–3927.
  • Schaake, J., Demargne, J., Hartman, R., Mullusky, M., Welles, E., Wu, L., Herr, H., Fan, X. and Seo, D. J. (2007). Precipitation and temperature ensemble forecasts from single-valued forecasts. Hydrology and Earth System Sciences Discussions 4 655–717.
  • Schaake, J., Pailleux, J., Thielen, J., Arritt, R., Hamill, T., Luo, L., Martin, E., McCollor, D. and Pappenberger, F. (2010). Summary of recommendations of the first Workshop on Postprocessing and Downscaling Atmospheric Forecasts for Hydrologic Applications held at Météo–France, Toulouse, France, 15–18 June 2009. Atmospheric Science Letters 11 59–63.
  • Schefzik, R. (2011). Ensemble copula coupling. Diploma thesis, Faculty of Mathematics and Informatics, Heidelberg Univ.
  • Schefzik, R., Thorarinsdottir, T. L. and Gneiting, T. (2013). Supplement to “Uncertainty quantification in complex simulation models using ensemble copula coupling.” DOI:10.1214/13-STS443SUPP.
  • Scheuerer, M. (2013). Probabilistic quantitative precipitation forecasting using ensemble model output statistics. Quarterly Journal of the Royal Meteorological Society. To appear. DOI:10.1002/qj.2183.
  • Schmeits, M. J. and Kok, K. J. (2010). A comparison between raw ensemble output, (modified) Bayesian model averaging, and extended logistic regression using ECMWF ensemble precipitation reforecasts. Monthly Weather Review 138 4199–4211.
  • Schoelzel, C. and Friederichs, P. (2008). Multivariate non-normally distributed random variables in climate research—Introduction to the copula approach. Nonlinear Processes in Geophysics 15 761–772.
  • Schoelzel, C. and Hense, A. (2011). Probabilistic assessment of regional climate change in Southwest Germany by ensemble dressing. Climate Dynamics 36 2003–2014.
  • Schuhen, N., Thorarinsdottir, T. L. and Gneiting, T. (2012). Ensemble model output statistics for wind vectors. Monthly Weather Review 140 3204–3219.
  • Segers, J. (2012). Asymptotics of empirical copula processes under non-restrictive smoothness assumptions. Bernoulli 18 764–782.
  • Sklar, M. (1959). Fonctions de répartition à $n$ dimensions et leurs marges. Publ. Inst. Statist. Univ. Paris 8 229–231.
  • Sloughter, M., Gneiting, T. and Raftery, A. E. (2010). Probabilistic wind spread forecasting using ensembles and Bayesian model averaging. J. Amer. Statist. Assoc. 105 25–35.
  • Sloughter, J. M., Gneiting, T. and Raftery, A. E. (2013). Probabilistic wind vector forecasting using ensembles and Bayesian model averaging. Monthly Weather Review 141 2107–2119.
  • Sloughter, J. M., Raftery, A. E., Gneiting, T. and Fraley, C. (2007). Probabilistic quantitative precipitation forecasting using Bayesian model averaging. Monthly Weather Review 135 3209–3220.
  • Stein, M. L. (1999). Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York.
  • Stute, W. (1984). The oscillation behavior of empirical processes: The multivariate case. Ann. Probab. 12 361–379.
  • Thorarinsdottir, T. L. and Gneiting, T. (2010). Probabilistic forecasts of wind speed: Ensemble model ouput statistics by using heteroscedastic censored regression. J. Roy. Statist. Soc. Ser. A 173 371–388.
  • Thorarinsdottir, T. L. and Johnson, M. S. (2012). Probabilistic wind gust forecasting using non-homogeneous Gaussian regression. Monthly Weather Review 140 889–897.
  • Trefethen, N. (2012). From the President: Discrete or continuous? SIAM News 45 4.
  • van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York.
  • Voisin, N., Pappenberger, F., Lettenmaier, D. P., Buizza, R. and Schaake, J. C. (2011). Application of a medium-range global hydrologic probabilistic forecast scheme to the Ohio River basin. Weather and Forecasting 26 425–446.
  • Wilks, D. S. (2002). Smoothing forecast ensembles with fitted probability distributions. Quarterly Journal of the Royal Meteorological Society 128 2821–2836.
  • Wilks, D. S. (2009). Extending logistic regression to provide full-probability-distribution MOS forecasts. Meteorological Applications 16 361–368.
  • Wilks, D. S. and Hamill, T. M. (2007). Comparison of ensemble-MOS methods using GFS reforecasts. Monthly Weather Review 135 2379–2390.

Supplemental materials

  • Supplementary material: Dynamic version of Figure 5. In this version of Figure 5, the ensemble reordering step in the ECC approach is elucidated when switching back and forth between pages.