Bayesian Analysis

Bayesian Solution Uncertainty Quantification for Differential Equations

Oksana A. Chkrebtii, David A. Campbell, Ben Calderhead, and Mark A. Girolami

Full-text: Open access


We explore probability modelling of discretization uncertainty for system states defined implicitly by ordinary or partial differential equations. Accounting for this uncertainty can avoid posterior under-coverage when likelihoods are constructed from a coarsely discretized approximation to system equations. A formalism is proposed for inferring a fixed but a priori unknown model trajectory through Bayesian updating of a prior process conditional on model information. A one-step-ahead sampling scheme for interrogating the model is described, its consistency and first order convergence properties are proved, and its computational complexity is shown to be proportional to that of numerical explicit one-step solvers. Examples illustrate the flexibility of this framework to deal with a wide variety of complex and large-scale systems. Within the calibration problem, discretization uncertainty defines a layer in the Bayesian hierarchy, and a Markov chain Monte Carlo algorithm that targets this posterior distribution is presented. This formalism is used for inference on the JAK-STAT delay differential equation model of protein dynamics from indirectly observed measurements. The discussion outlines implications for the new field of probabilistic numerics.

Article information

Bayesian Anal. Volume 11, Number 4 (2016), 1239-1267.

First available in Project Euclid: 7 September 2016

Permanent link to this document

Digital Object Identifier

Zentralblatt MATH identifier

Bayesian numerical analysis uncertainty quantification Gaussian processes differential equation models uncertainty in computer models

Creative Commons Attribution 4.0 International License.


Chkrebtii, Oksana A.; Campbell, David A.; Calderhead, Ben; Girolami, Mark A. Bayesian Solution Uncertainty Quantification for Differential Equations. Bayesian Anal. 11 (2016), no. 4, 1239--1267. doi:10.1214/16-BA1017.

Export citation


  • Arridge, S. R., Kaipio, J. P., Kolehmainen, V., Schweiger, M., Somersalo, E., Tarvainen, T., and Vauhkonen, M. (2006). “Approximation errors and model reduction with an application in optical diffusion tomography.” Inverse Problems, 22(1): 175–195.
  • Berliner, L. M. (1991). “Likelihood and Bayesian prediction of chaotic systems.” Journal of the American Statistical Association, 86(416): 938–952.
  • Bock, H. G. (1983). “Recent advances in parameter identification techniques for ODE.” In Deuflhard, P. and Harrier, E. (eds.), Numerical Treatment of Inverse Problems in Differential and Integral Equations, 95–121. Birkhäuser.
  • Brunel, N. J. (2008). “Parameter estimation of ODEs via nonparametric estimators.” Electronic Journal of Statistics, 2: 1242–1267.
  • Calderhead, B. and Girolami, M. (2011). “Statistical analysis of nonlinear dynamical systems using differential geometric sampling methods.” Interface Focus, 1(6): 821–835.
  • Campbell, D. and Steele, R. J. (2011). “Smooth functional tempering for nonlinear differential equation models.” Statistics and Computing, 22: 429–443.
  • Campbell, D. A. and Chkrebtii, O. (2013). “Maximum profile likelihood estimation of differential equation parameters through model based smoothing state estimates.” Mathematical Biosciences, 246: 283–292.
  • Chkrebtii, O. (2013). “Probabilistic solution of differential equations for Bayesian uncertainty quantification and inference.” Ph.D. thesis, Simon Fraser University.
  • Chkrebtii, O. A., Campbell, D. A., Calderhead, B., and Girolami, M. A. (2016). “Supplementary material for “Bayesian solution uncertainty quantification for differential equations”.” Bayesian Analysis.
  • Dowd, M. (2007). “Bayesian statistical data assimilation for ecosystem models using Markov chain Monte Carlo.” Journal of Marine Systems, 68(3–4): 439–456.
  • Geyer, C. (1991). “Markov chain Monte Carlo maximum likelihood.” In Computing Science and Statistics, Proceedings of the 23rd Symposium on the Interface, 156. American Statistical Association.
  • Ghanem, R. and Spanos, P. (2003). Stochastic Finite Elements: A Spectral Approach. Springer-Verlag.
  • Golightly, A. and Wilkinson, D. J. (2011). “Bayesian parameter inference for stochastic biochemical network models using particle Markov chain Monte Carlo.” Interface Focus, 1: 807–820.
  • Gramacy, R. and Lee, H. (2012). “Cases for the nugget in modeling computer experiments.” Statistics and Computing, 22: 713–722.
  • Gugushvili, S. and Klaassen, C. A. (2012). “Root n-consistent parameter estimation for systems of ordinary differential equations: bypassing numerical integration via smoothing.” Bernoulli, 18: 1061–1098.
  • Higdon, D. (1998). “A process-convolution approach to modeling temperatures in the North Atlantic Ocean.” Journal of Environmental and Ecological Statistics, 5: 173–190.
  • Higdon, D. (2002). Space and Space–Time Modeling Using Process Convolutions, 37–54. London, UK: Springer Verlag.
  • Huttunen, J. M. J. and Kaipio, J. P. (2007). “Approximation error analysis in nonlinear state estimation with an application to state–space identification.” Inverse Problems, 23(5): 2141.
  • Ionides, E., Bretó, C., and King, A. (2006). “Inference for nonlinear dynamical systems.” Proceedings of the National Academy of Sciences of the United States of America, 103(49): 18438–18443.
  • Kaipio, J. and Somersalo, E. (2007). “Statistical inverse problems: Discretization, model reduction and inverse crimes.” Journal of Computational and Applied Mathematics, 198: 493–504.
  • Kaipio, J. P., Seppanen, A., Somersalo, E., and Haario, H. (2004). “Posterior covariance related optimal current patterns in electrical impedance tomography.” Inverse Problems, 20(3): 919.
  • Kennedy, M. C. and O’Hagan, A. (2001). “Bayesian calibration of computer models.” Journal of the Royal Statistical Society B, 63(3): 425–464.
  • Kim, J.-h. et al. (2014). “The AGORA high-resolution galaxy simulations comparison project.” The Astrophysical Journal Supplement Series, 210(1): 14.
  • Liang, H. and Wu, H. (2008). “Parameter estimation for differential equation models using a framework of measurement error in regression models.” Journal of the American Statistical Association, 103(484): 1570–1583.
  • Lorenz, E. N. (1963). “Deterministic nonperiodic flow.” Journal of the Atmospheric Sciences, 20: 130–141.
  • Marjoram, P., Molitor, J., Plagnol, V., and Tavaré, S. (2003). “Markov chain Monte Carlo without likelihoods.” Proceedings of the National Academy of Sciences, 100(26): 15324–15328.
  • Marzouk, Y. and Najm, H. (2009). “Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems.” Journal of Computational Physics, 228(6): 1862–1902.
  • Marzouk, Y., Najm, H., and Rahn, L. (2007). “Stochastic spectral methods for efficient Bayesian solution of inverse problems.” Journal of Computational Physics, 224(2): 560–586.
  • Neal, R. M. (2011). “MCMC Using Ensembles of States for Problems with Fast and Slow Variables such as Gaussian Process Regression.” arXiv:1101.0387.
  • Oberkampf, W. and Roy, C. (2010). Verification and Validation in Scientific Computing. Cambridge University Press.
  • O’Hagan, A. O. (1992). “Some Bayesian numerical analysis.” Bayesian Statistics, 4: 345–363.
  • Pellegrini, S. and Dusanter-Fourt, I. (1997). “The structure, regulation and function of the Janus Kinases (JAKs) and the signal transducers and activators of transcription (STATs).” European Journal of Biochemistry, 248(3): 615–633.
  • Ramsay, J., Hooker, G., Campbell, D., and Cao, J. (2007). “Parameter estimation for differential equations: a generalized smoothing approach.” Journal of the Royal Statistical Society B, 69: 741–796.
  • Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. Cambridge, Massachusetts: MIT Press.
  • Raue, A., Kreutz, C., Maiwald, T., Bachmann, J., Schilling, M., Klingmüller, U., and Timmer, J. (2009). “Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood.” Bioinformatics, 25: 1923–1929.
  • Robinson, J. (2001). Infinite-Dimensional Dynamical Systems: An Introduction to Dissipative Parabolic PDEs and the Theory of Global Attractors. Cambridge Texts in Applied Mathematics. Cambridge University Press.
  • Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). “Design and analysis of computer experiments.” Statistical Science, 4(4): 409–423.
  • Schmidl, D., Czado, C., Hug, S., and Theis, F. J. (2013). “A vine-copula based adaptive MCMC sampler for efficient inference of dynamical systems.” Bayesian Analysis, 8(1): 1–22.
  • Schober, M., Duvenaud, D. K., and Hennig, P. (2014). “Probabilistic ODE solvers with Runge–Kutta means.” In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems 27, 739–747. Curran Associates, Inc.
  • Shampine, L. (2003). “Singular boundary value problems for ODEs.” Applied Mathematics and Computation, 138: 99–112.
  • Skilling, J. (1991). Bayesian Solution of Ordinary Differential Equations, 23–37. Seattle: Kluwer Academic Publishers.
  • Solak, E., Murray-Smith, R., Leithead, W. E., Leith, D. J., and Rasmussen, C. E. (2003). “Derivative observations in Gaussian process models of dynamic systems.” In Becker, S., Thrun, S., and Obermayer, K. (eds.), Advances in Neural Information Processing Systems 15, 1057–1064. MIT Press.
  • Stuart, A. M. (2010). “Inverse problems: A Bayesian perspective.” Acta Numerica, 19: 451–559.
  • Swameye, I., Muller, T., Timmer, J., Sandra, O., and Klingmuller, U. (2003). “Identification of nucleocytoplasmic cycling as a remote sensor in cellular signaling by databased modeling.” Proceedings of the National Academy of Sciences, 100: 1028–1033.
  • Xue, H., Miao, H., and Wu, H. (2010). “Sieve estimation of constant and time-varying coefficients in nonlinear ordinary differential equation models by considering both numerical error and measurement error.” The Annals of Statistics, 38: 2351–2387.
  • Xun, X., Cao, J., Mallick, B., Carroll, R. J., and Maity, A. (2013). “Parameter estimation of partial differential equation models.”

See also

  • Related item: Martin Lysy (2016). Comment on Article by Chkrebtii, Campbell, Calderhead, and Girolami. Bayesian Anal. Vol. 11, Iss. 4, 1269–1273.
  • Related item: Sarat C. Dass (2016). Comment on Article by Chkrebtii, Campbell, Calderhead, and Girolami. Bayesian Anal. Vol. 11, Iss. 4, 1275–1277.
  • Related item: Bani K. Mallick, Keren Yang, Nilabja Guha, Yalchin Efendiev (2016). Comment on Article by Chkrebtii, Campbell, Calderhead, and Girolami. Bayesian Anal. Vol. 11, Iss. 4, 1279–1284.
  • Related item: François-Xavier Briol, Jon Cockayne, Onur Teymur, William Weimin Yoo, Jon Cockayne, Michael Schober, Philipp Hennig (2016). Contributed Discussion on Article by Chkrebtii, Campbell, Calderhead, and Girolami. Bayesian Anal. Vol. 11, Iss. 4, 1285–1293.
  • Related item: Oksana A. Chkrebtii, David A. Campbell, Ben Calderhead, Mark A. Girolami (2016). Rejoinder. Bayesian Anal. Vol. 11, Iss. 4, 1295–1299.

Supplemental materials