Statistical Science

Approximate Models and Robust Decisions

James Watson and Chris Holmes

Full-text: Open access


Decisions based partly or solely on predictions from probabilistic models may be sensitive to model misspecification. Statisticians are taught from an early stage that “all models are wrong, but some are useful”; however, little formal guidance exists on how to assess the impact of model approximation on decision making, or how to proceed when optimal actions appear sensitive to model fidelity. This article presents an overview of recent developments across different disciplines to address this. We review diagnostic techniques, including graphical approaches and summary statistics, to help highlight decisions made through minimised expected loss that are sensitive to model misspecification. We then consider formal methods for decision making under model misspecification by quantifying stability of optimal actions to perturbations to the model within a neighbourhood of model space. This neighbourhood is defined in either one of two ways. First, in a strong sense via an information (Kullback–Leibler) divergence around the approximating model. Second, using a Bayesian nonparametric model (prior) centred on the approximating model, in order to “average out” over possible misspecifications. This is presented in the context of recent work in the robust control, macroeconomics and financial mathematics literature. We adopt a Bayesian approach throughout although the presentation is agnostic to this position.

Article information

Statist. Sci., Volume 31, Number 4 (2016), 465-489.

First available in Project Euclid: 19 January 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Computational decision theory model misspecification D-open problem Kullback–Leibler divergence robustness Bayesian nonparametrics


Watson, James; Holmes, Chris. Approximate Models and Robust Decisions. Statist. Sci. 31 (2016), no. 4, 465--489. doi:10.1214/16-STS592.

Export citation


  • Ahmadi-Javid, A. (2011). An information-theoretic approach to constructing coherent risk measures. In IEEE International Symposium on Information Theory Proceedings (ISIT) 2125–2127. IEEE, New York.
  • Ahmadi-Javid, A. (2012). Entropic value-at-risk: A new coherent risk measure. J. Optim. Theory Appl. 155 1105–1123.
  • Artzner, P., Delbaen, F., Eber, J.-M. and Heath, D. (1999). Coherent measures of risk. Math. Finance 9 203–228.
  • Autier, P. (2015). Breast cancer: Doubtful health benefit of screening from 40 years of age. Nat. Rev. Clin. Oncol. 12 570–572.
  • Baio, G. and Dawid, A. P. (2015). Probabilistic sensitivity analysis in health economics. Stat. Methods Med. Res. 24 615–634.
  • Beaumont, M. A., Zhang, W. and Balding, D. J. (2002). Approximate Bayesian computation in population genetics. Genetics 162 2025–2035.
  • Belsley, D. A., Kuh, E. and Welsch, R. E. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley, New York.
  • Berger, J. O. (1984). The robust Bayesian viewpoint. In Robustness of Bayesian Analyses (J. Kadane, ed.) 63–144. North-Holland, Amsterdam.
  • Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis, 2nd ed. Springer, New York.
  • Berger, J. O. (1994). An overview of robust Bayesian analysis. TEST 3 5–124.
  • Berger, J. and Berliner, L. M. (1986). Robust Bayes and empirical Bayes analysis with $\varepsilon$-contaminated priors. Ann. Statist. 14 461–486.
  • Bernardo, J.-M. and Smith, A. F. M. (1994). Bayesian Theory. Wiley, Chichester.
  • Bissiri, P. G., Holmes, C. C. and Walker, S. G. (2013). A general framework for updating belief distributions. J. R. Stat. Soc. Ser. B. Stat. Methodol. Preprint. Available at arXiv:1306.6430.
  • Bissiri, P. G. and Walker, S. G. (2012). Converting information into probability measures with the Kullback-Leibler divergence. Ann. Inst. Statist. Math. 64 1139–1160.
  • Box, G. E. P. and Draper, N. R. (1987). Empirical Model-Building and Response Surfaces. Wiley, New York.
  • Breuer, T. and Csiszár, I. (2013). Systematic stress tests with entropic plausibility constraints. J. Bank. Financ. 37 1552–1559.
  • Breuer, T. and Csiszár, I. (2016). Measuring distribution model risk. Math. Finance 26 395–411.
  • Carota, C., Parmigiani, G. and Polson, N. G. (1996). Diagnostic measures for model criticism. J. Amer. Statist. Assoc. 91 753–762.
  • Chipman, H. A., George, E. I. and McCulloch, R. E. (1998). Bayesian CART model search. J. Amer. Statist. Assoc. 443 935–948.
  • Basle Committee (1996). Amendment to the capital accord to incorporate market risks. Basle Committee on banking supervision.
  • National Research Council, Committee on the Analysis of Massive Data, Committee on Applied and Theoretical Statistics, Board on Mathematical Sciences and Their Applications and Division on Engineering and Physical Sciences (2013). Frontiers in Massive Data Analysis. The National Academies Press, Washington, DC.
  • Dalalyan, A. and Tsybakov, A. B. (2008). Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Mach. Learn. 72 39–61.
  • Dalalyan, A. S. and Tsybakov, A. B. (2012). Sparse regression learning by aggregation and Langevin Monte-Carlo. J. Comput. System Sci. 78 1423–1443.
  • Del Moral, P., Doucet, A. and Jasra, A. (2006). Sequential Monte Carlo samplers. J. R. Stat. Soc. Ser. B. Stat. Methodol. 68 411–436.
  • Dempster, A. P. (1975). A subjectivist look at robustness. Bull. Int. Stat. Inst. 46 349–374.
  • Denison, D. G. T., Holmes, C. C., Mallick, B. K. and Smith, A. F. M. (2002). Bayesian Methods for Nonlinear Classification and Regression. Wiley, Chichester.
  • Fearnhead, P. and Prangle, D. (2012). Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation. J. R. Stat. Soc. Ser. B. Stat. Methodol. 74 419–474.
  • Gelman, A. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge Univ. Press, Cambridge, MA.
  • Gilboa, I. and Schmeidler, D. (1989). Maxmin expected utility with nonunique prior. J. Math. Econom. 18 141–153.
  • Good, I. J. (1952). Rational decisions. J. R. Stat. Soc. Ser. B. Stat. Methodol. 14 107–114.
  • Grünwald, P. and van Ommen, T. (2014). Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Preprint. Available at arXiv:1412.3730.
  • Hand, D. J. (2006). Classifier technology and the illusion of progress. Statist. Sci. 21 1–34.
  • Hansen, L. P. and Sargent, T. J. (2001a). Acknowledging misspecification in macroeconomic theory. Rev. Econ. Dyn. 4 519–535.
  • Hansen, L. P. and Sargent, T. J. (2001b). Robust control and model uncertainty. Am. Econ. Rev. 91 60–66.
  • Hansen, L. P. and Sargent, T. J. (2008). Robustness. Princeton Univ. Press, Princeton, NJ.
  • Hansen, L. P., Sargent, T. J., Turmuhambetova, G. and Williams, N. (2006). Robust control and model misspecification. J. Econom. Theory 128 45–90.
  • Hastie, T. and Tibshirani, R. (1993). Varying-coefficient models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 55 757–796.
  • Hjort, N. L., Holmes, C. C., Müller, P. and Walker, S. G. (2010). Bayesian Nonparametrics. Cambridge Univ. Press, Cambridge, MA.
  • Huber, P. J. (2011). Robust Statistics. Springer, Berlin.
  • Kadane, J. B., ed. (1984). Robustness of Bayesian Analyses. North-Holland, Amsterdam.
  • Kadane, J. B. and Chuang, D. T. (1978). Stable decision problems. Ann. Statist. 6 1095–1110.
  • Kadane, J. B. and Srinivasan, C. (1994). Discussion of Berger, J. O., An overview of robust Bayesian analysis. TEST 3 116–120.
  • Kerman, J., Gelman, A., Zheng, T. and Ding, Y. (2008). Visualization in Bayesian data analysis. In Handbook of Data Visualization 709–724. Springer, Berlin.
  • Løberg, M., Lousdal, M. L., Bretthauer, M. and Kalager, M. (2015). Benefits and harms of mammography screening. Breast Cancer Res. Treat. 17 63.
  • Marin, J.-M., Pudlo, P., Robert, C. P. and Ryder, R. J. (2012). Approximate Bayesian computational methods. Stat. Comput. 22 1167–1180.
  • Marjoram, P., Molitor, J., Plagnol, V. and Tavaré, S. (2003). Markov chain Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 100 15324–15328.
  • Marmot, M. G. et al. (2012). The benefits and harms of breast cancer screening: An independent review. Lancet 380 1778–1786.
  • McCulloch, R. E. (1989). Local model influence. J. Amer. Statist. Assoc. 84 473–478.
  • Miller, J. W. and Dunson, D. B. (2015). Robust Bayesian inference via coarsening. Preprint. Available at arXiv:1506.06101.
  • Minka, T. P. (2001). Expectation propagation for approximate Bayesian inference. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence 362–369. Morgan Kaufmann, San Mateo, CA.
  • Moss, S. M., Wale, S., Smith, R., Evans, A., Cuckle, H. and Duffy, S. W. (2015). Effect of mammographic screening from age 40 years on breast cancer mortality in the UK age trial at 17 years’ follow-up: A randomised controlled trial. Lancet Oncol. 16 1123–1132.
  • Parmigiani, G. (1993). On optimal screening ages. J. Amer. Statist. Assoc. 88 622–628.
  • Parmigiani, G. and Inoue, L. Y. T. (2009). Decision Theory. Wiley, Chichester.
  • Pritsker, M. (1997). Evaluating value at risk methodologies: Accuracy versus computational time. J. Financ. Serv. Res. 12 201–242.
  • Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.
  • Ratmann, O., Andrieu, C., Wiuf, C. and Richardson, S. (2009). Model criticism based on likelihood-free inference, with an application to protein network evolution. Proc. Natl. Acad. Sci. USA 106 10576–10581.
  • Ríos Insua, D. and Ruggeri, F., eds. (2000). Robust Bayesian Analysis. Springer, New York.
  • Robbins, H. (1951). Asymptotically subminimax solutions of compound statistical decision problems. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability 131–148. University of California Press, Berkeley and Los Angeles.
  • Robert, C. P. and Casella, G. (2004). Monte Carlo Statistical Methods, 2nd ed. Springer, New York.
  • Rockafellar, R. T. and Uryasev, S. (2000). Optimization of conditional value-at-risk. The Journal of Risk 2 21–42.
  • Rostek, M. (2010). Quantile maximization in decision theory. Rev. Econ. Stud. 77 339–371.
  • Rue, H., Martino, S. and Chopin, N. (2009). Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. J. R. Stat. Soc. Ser. B. Stat. Methodol. 71 319–392.
  • Ruggeri, F., Insua, D. R. and Martín, J. (2005). Robust Bayesian analysis. In Bayesian Thinking: Modeling and Computation. Handbook of Statistics 25 623–667. Elsevier, Amsterdam.
  • Ruggeri, F. and Wasserman, L. (1993). Infinitesimal sensitivity of posterior distributions. Canad. J. Statist. 21 195–203.
  • Savage, L. J. (1954). The Foundations of Statistics. Wiley, New York.
  • Shapiro, S., Venet, W., Strax, P. and Venet, L. (1988). Periodic Screening for Breast Cancer: The Health Insurance Plan Project and Its Sequelae, 19631986. The John Hopkins Univ. Press, Baltimore, MD.
  • Sivaganesan, S. (1994). Discussion of Berger, J. O., An overview of robust Bayesian analysis. TEST 3 116–120.
  • Sivaganesan, S. (2000). Global and local robustness approaches: Uses and limitations. In Robust Bayesian Analysis (D. Rios Insua and F. Ruggeri, eds.) 89–108. Springer, New York.
  • Varin, C., Reid, N. and Firth, D. (2011). An overview of composite likelihood methods. Statist. Sinica 21 5–42.
  • Vickers, A. J. and Elkin, E. B. (2006). Decision curve analysis: A novel method for evaluating prediction models. Med. Decis. Mak. 26 565–574.
  • Vidakovic, B. (2000). $\Gamma$-minimax: A paradigm for conservative robust Bayesians. In Robust Bayesian Analysis (D. Rios Insua and F. Ruggeri, eds.) 241–259. Springer, New York.
  • von Neumann, J. and Morgenstern, O. (1947). Theory of Games and Economic Behavior, 2nd ed. Princeton Univ. Press, Princeton, NJ.
  • Wainwright, M. and Jordan, M. I. (2003). Graphical models, exponential families and variational inference. Faund. Trends Mach. Learn. 1–305.
  • Wald, A. (1950). Statistical Decision Functions. Wiley, New York.
  • Walker, S. and Hjort, N. L. (2001). On Bayesian consistency. J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 811–821.
  • Wasserman, L. (1992). Recent methodological advances in robust Bayesian inference. In Bayesian Statistics, 4 (PeñíScola, 1991) 483–502. Oxford Univ. Press, New York.
  • Watson, J., Nieto-Barajas, L. and Holmes, C. (2016). Characterising variation of nonparametric random probability models using the Kullback–Leibler divergence. Statistics. To appear. Available at 1411.6578.
  • Whittle, P. (1990). Risk-Sensitive Optimal Control. Wiley, Chichester.
  • Wu, D., Rosner, G. L. and Broemeling, L. D. (2007). Bayesian inference for the lead time in periodic cancer screening. Biometrics 63 873–880.
  • Zhang, T. (2006a). From $\varepsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation. Ann. Statist. 34 2180–2210.
  • Zhang, T. (2006b). Information-theoretic upper and lower bounds for statistical estimation. IEEE Trans. Inform. Theory 52 1307–1321.

See also