Statistical Science

Probabilistic Integration: A Role in Statistical Computation?

Abstract

A research frontier has emerged in scientific computation, wherein discretisation error is regarded as a source of epistemic uncertainty that can be modelled. This raises several statistical challenges, including the design of statistical methods that enable the coherent propagation of probabilities through a (possibly deterministic) computational work-flow, in order to assess the impact of discretisation error on the computer output. This paper examines the case for probabilistic numerical methods in routine statistical computation. Our focus is on numerical integration, where a probabilistic integrator is equipped with a full distribution over its output that reflects the fact that the integrand has been discretised. Our main technical contribution is to establish, for the first time, rates of posterior contraction for one such method. Several substantial applications are provided for illustration and critical evaluation, including examples from statistical modelling, computer graphics and a computer model for an oil reservoir.

Article information

Source
Statist. Sci., Volume 34, Number 1 (2019), 1-22.

Dates
First available in Project Euclid: 12 April 2019

https://projecteuclid.org/euclid.ss/1555056025

Digital Object Identifier
doi:10.1214/18-STS660

Mathematical Reviews number (MathSciNet)
MR3938958

Citation

Briol, François-Xavier; Oates, Chris J.; Girolami, Mark; Osborne, Michael A.; Sejdinovic, Dino. Probabilistic Integration: A Role in Statistical Computation?. Statist. Sci. 34 (2019), no. 1, 1--22. doi:10.1214/18-STS660. https://projecteuclid.org/euclid.ss/1555056025

References

• Bauer, B., Devroye, L., Kohler, M., Krzyżak, A. and Walk, H. (2017). Nonparametric estimation of a function from noiseless observations at random points. J. Multivariate Anal. 160 93–104.
• Berlinet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic, Boston, MA.
• Bogachev, V. I. (1998). Gaussian Measures. Mathematical Surveys and Monographs 62. Amer. Math. Soc., Providence, RI.
• Bondarenko, A., Radchenko, D. and Viazovska, M. (2013). Optimal asymptotic bounds for spherical designs. Ann. of Math. (2) 178 443–452.
• Briol, F.-X., Oates, C. J., Girolami, M. and Osborne, M. A. (2015). Frank-Wolfe Bayesian quadrature: Probabilistic integration with theoretical guarantees. In Proc. Adv. Neur. In. (NIPS) 1162–1170.
• Briol, F.-X., Cockayne, J., Teymur, O., Yoo, W. W., Schober, M. and Hennig, P. (2016a). Contributed discussion on article by Chkrebtii, Campbell, Calderhead, and Girolami [MR3577378]. Bayesian Anal. 11 1285–1293.
• Briol, F.-X., Oates, C. J., Girolami, M., Osborne, M. A and Sejdinovic, D. (2019). Supplement to “Probabilistic integration: A role in statistical computation?” DOI:10.1214/18-STS660SUPP.
• Brouillat, J., Bouville, C., Loos, B., Hansen, C. and Bouatouch, K. (2009). A Bayesian Monte Carlo approach to global illumination. Comput. Graph. Forum 28 2315–2329.
• Buchholz, A. and Chopin, N. (2017). Improving approximate Bayesian computation via quasi Monte Carlo. arXiv:1710:01057.
• Calderhead, B. and Girolami, M. (2009). Estimating Bayes factors via thermodynamic integration and population MCMC. Comput. Statist. Data Anal. 53 4028–4045.
• Cialenco, I., Fasshauer, G. E. and Ye, Q. (2012). Approximation of stochastic partial differential equations by a kernel-based collocation method. Int. J. Comput. Math. 89 2543–2561.
• Diaconis, P. (1988). Bayesian numerical analysis. In Statistical Decision Theory and Related Topics, IV, Vol. 1 (West Lafayette, Ind., 1986) 163–175. Springer, New York.
• Dick, J., Kuo, F. Y. and Sloan, I. H. (2013). High-dimensional integration: The quasi-Monte Carlo way. Acta Numer. 22 133–288.
• Dick, J. and Pillichshammer, F. (2010). Digital Nets and Sequences. Cambridge Univ. Press, Cambridge. Discrepancy theory and quasi-Monte Carlo integration.
• Duvenaud, D. (2014). Automatic model construction with Gaussian processes Ph.D. thesis, Univ. Cambridge.
• Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability 57. CRC Press, New York.
• Eftang, J. L. and Stamm, B. (2012). Parameter multi-domain ‘$hp$’ empirical interpolation. Internat. J. Numer. Methods Engrg. 90 412–428.
• Friel, N., Hurn, M. and Wyse, J. (2014). Improving power posterior estimation of statistical evidence. Stat. Comput. 24 709–723.
• Friel, N. and Pettitt, A. N. (2008). Marginal likelihood estimation via power posteriors. J. R. Stat. Soc. Ser. B. Stat. Methodol. 70 589–607.
• Gelman, A. and Meng, X.-L. (1998). Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statist. Sci. 13 163–185.
• Gerber, M. and Chopin, N. (2015). Sequential quasi Monte Carlo. J. R. Stat. Soc. Ser. B. Stat. Methodol. 77 509–579.
• Gunter, T., Garnett, R., Osborne, M., Hennig, P. and Roberts, S. (2014). Sampling for inference in probabilistic models with fast Bayesian quadrature. In Proc. Adv. Neur. In 2789–2797.
• Hajizadeh, Y., Christie, M. and Demyanov, V. (2011). Ant colony optimization for history matching and uncertainty quantification of reservoir models. J. Pet. Sci. Eng. 77 78–92.
• Hennig, P., Osborne, M. A. and Girolami, M. (2015). Probabilistic numerics and uncertainty in computations. Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 471 20150142, 17.
• Hickernell, F. J. (1998). A generalized discrepancy and quadrature error bound. Math. Comp. 67 299–322.
• Hickernell, F. J., Lemieux, C. and Owen, A. B. (2005). Control variates for quasi-Monte Carlo. Statist. Sci. 20 1–31.
• Hug, S., Schwarzfischer, M., Hasenauer, J., Marr, C. and Theis, F. J. (2016). An adaptive scheduling scheme for calculating Bayes factors with thermodynamic integration using Simpson’s rule. Stat. Comput. 26 663–677.
• Huszar, F. and Duvenaud, D. (2012). Optimally-weighted herding is Bayesian quadrature. In Proc. Uncertainty in Artificial Intelligence (UAI) 377–385.
• Kadane, J. B. (1985). Parallel and sequential computation: A statistician’s view. J. Complexity 1 256–263.
• Kadane, J. B. and Wasilkowski, G. W. (1985). Average case $\epsilon$-complexity in computer science. A Bayesian view. In Bayesian Statistics, 2 (Valencia, 1983) 361–374. North-Holland, Amsterdam.
• Kallenberg, O. (2002). Foundations of Modern Probability, 2nd ed. Probability and Its Applications (New York). Springer, New York.
• Kanagawa, M., Sriperumbudur, B. K. and Fukumizu, K. (2016). Convergence guarantees for kernel-based quadrature rules in misspecified settings. In Proc. Adv. Neur. In. (NIPS).
• Kanagawa, M., Sriperumbudur, B. K. and Fukumizu, K. (2017). Convergence analysis of deterministic kernel-based quadrature rules in misspecified settings. arXiv:1709.00147.
• Karvonen, T. and Särkkä, S. (2017). Classical quadrature rules via Gaussian processes. IEEE I Workshop Mach. Learn. Signal Proc. to appear.
• Karvonen, T. and Särkkä, S. (2018). Fully symmetric kernel quadrature. SIAM J. Sci. Comput. 40 A697–A720.
• Kennedy, M. (1998). Bayesian quadrature with non-normal approximating functions. Stat. Comput. 8 365–375.
• Kristoffersen, S. (2013). The empirical interpolation method Master’s thesis, Dept. Mathematical Sciences, Norwegian Univ. Science and Technology, Trondheim.
• Kuo, F. Y., Dunsmuir, W. T. M., Sloan, I. H., Wand, M. P. and Womersley, R. S. (2008). Quasi-Monte Carlo for highly structured generalised response models. Methodol. Comput. Appl. Probab. 10 239–275.
• Lan, S., Bui-Thanh, T., Christie, M. and Girolami, M. (2016). Emulation of higher-order tensors in manifold Monte Carlo methods for Bayesian inverse problems. J. Comput. Phys. 308 81–101.
• Larkin, F. M. (1972). Gaussian measure in Hilbert space and applications in numerical analysis. Rocky Mountain J. Math. 2 379–421.
• Marques, R., Bouville, C., Ribardiere, M., Santos, L. P. and Bouatouch, K. (2013). A spherical Gaussian framework for Bayesian Monte Carlo rendering of glossy surfaces. IEEE Trans. Vis. Comput. Graph. 19 1619–1632.
• Marques, R., Bouville, C., Santos, L. and Bouatouch, K. (2015). Efficient quadrature rules for illumination integrals: From quasi Monte Carlo to Bayesian Monte Carlo. Synth. Lect. Comput. Graph. Animat. 7 1–92.
• Minka, T. (2000). Deriving quadrature rules from Gaussian processes Technical report, Statistics Dept., Carnegie Mellon Univ.
• Mohamed, L., Christie, M. and Demyanov, V. (2010). Comparison of stochastic sampling algorithms for uncertainty quantification. SPE J. 15 31–38.
• Mosbach, S. and Turner, A. G. (2009). A quantitative probabilistic investigation into the accumulation of rounding errors in numerical ODE solution. Comput. Math. Appl. 57 1157–1167.
• Novak, E. and Woźniakowski, H. (2008). Tractability of Multivariate Problems. Vol. 1: Linear Information. EMS Tracts in Mathematics 6. European Mathematical Society (EMS), Zürich.
• Novak, E. and Woźniakowski, H. (2010). Tractability of Multivariate Problems. Volume II: Standard Information for Functionals. EMS Tracts in Mathematics 12. European Mathematical Society (EMS), Zürich.
• O’Hagan, A. (1991). Bayes-Hermite quadrature. J. Statist. Plann. Inference 29 245–260.
• O’Hagan, A. (1992). Some Bayesian numerical analysis. In Bayesian Statistics, 4 (Peñíscola, 1991) 345–363. Oxford Univ. Press, New York.
• Oates, C. J., Cockayne, J. and Aykroyd, R. G. (2017). Bayesian probabilistic numerical methods for industrial process monitoring. arXiv:1707.06107.
• Oates, C. J., Girolami, M. and Chopin, N. (2017). Control functionals for Monte Carlo integration. J. R. Stat. Soc. Ser. B. Stat. Methodol. 79 695–718.
• Oates, C. J., Papamarkou, T. and Girolami, M. (2016). The controlled thermodynamic integral for Bayesian model evidence evaluation. J. Amer. Statist. Assoc. 111 634–645.
• Oates, C. J., Cockayne, J., Briol, F.-X. and Girolami, M. (2016). Convergence rates for a class of estimators based on Stein’s identity. Bernoulli (to appear), arXiv:1603.03220.
• Oates, C. J., Niederer, S., Lee, A., Briol, F. X. and Girolami, M. (2017). Probabilistic models for integration error in assessment of functional cardiac models. In Proc. Adv. Neur. In. (NIPS). to appear.
• Oettershagen, J. (2017). Construction of optimal cubature algorithms with applications to econometrics and uncertainty quantification Ph.D. Thesis, Univ. Bonn.
• Osborne, M. A. (2010). Bayesian Gaussian processes for sequential prediction, optimisation and quadrature Ph.D. thesis, Univ. Oxford.
• Osborne, M. A., Duvenaud, D., Garnett, R., Rasmussen, C. E., Roberts, S. and Ghahramani, Z. (2012). Active learning of model evidence using Bayesian quadrature. In Proc. Adv. Neur. In. (NIPS) 46–54.
• Park, H., Scheidt, C., Fenwick, D., Boucher, A. and Caers, J. (2013). History matching and uncertainty quantification of facies models with multiple geological interpretations. Comput. Geosci. 17 609–621.
• Pharr, M. and Humphreys, G. (2004). Physically Based Rendering: From Theory to Implementation. Kaufmann, Los Altos.
• Rasmussen, C. E. and Ghahramani, Z. (2002). Bayesian Monte Carlo. In Proc. Adv. Neur. Inf. (NIPS) 489–496.
• Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA.
• Robert, C. P. and Casella, G. (1999). Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer, New York.
• Särkka, S., Hartikainen, J., Svensson, L. and Sandblom, F. (2016). On the relation between Gaussian process quadratures and sigma-point methods. J. Adv. Inf. Fusion 11 31–46.
• Schölkopf, B. and Smola, A. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond. MIT Press, Cambridge.
• Sickel, W. and Ullrich, T. (2009). Tensor products of Sobolev–Besov spaces and applications to approximation from the hyperbolic cross. J. Approx. Theory 161 748–786.
• Sinescu, V., Kuo, F. Y. and Sloan, I. H. (2012). On the choice of weights in a function space for quasi-Monte Carlo methods for a class of generalised response models in statistics. In Proc. Monte Carlo and Quasi-Monte Carlo Methods.
• Smola, A., Gretton, A., Song, L. and Schölkopf, B. (2007). A Hilbert space embedding for distributions. In Proc. 18th I. Conf. Algorithmic Learn. Theory 13–31.
• Sommariva, A. and Vianello, M. (2006). Numerical cubature on scattered data by radial basis functions. Computing 76 295–310.
• Song, L. (2008). Learning via Hilbert space embedding of distributions Ph.D. thesis, School of Information Technologies, Univ. Sydney.
• Stein, M. L. (1995a). Predicting integrals of random fields using observations on a lattice. Ann. Statist. 23 1975–1990.
• Stein, M. L. (1995b). Locally lattice sampling designs for isotropic random fields. Ann. Statist. 23 1991–2012.
• Stein, M. L. (1999). Interpolation of Spatial Data. Springer Series in Statistics. Springer, New York. Some theory for Kriging.
• Steinwart, I. and Christmann, A. (2008). Support Vector Machines. Information Science and Statistics. Springer, New York.
• Suldin, A. B. (1959). Wiener measure and its applications to approximation methods. I. Izvestiya Vysshikh Uchebnykh Zavedenii. Matematika 6 145–158.
• Szabó, B., van der Vaart, A. W. and van Zanten, J. H. (2015). Frequentist coverage of adaptive nonparametric Bayesian credible sets. Ann. Statist. 43 1391–1428.
• Traub, J. F., Wasilkowski, G. W. and Woźniakowski, H. (1988). Information-Based Complexity. Computer Science and Scientific Computing. Academic Press, Boston, MA.
• Wahba, G. (1990). Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics 59. SIAM, Philadelphia, PA.
• Wendland, H. (2005). Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathematics 17. Cambridge Univ. Press, Cambridge.
• Xi, X., Briol, F.-X. and Girolami, M. (2018). Bayesian quadrature for multiple related integrals. International Conference on Machine Learning, PMLR 80:5369–5378, arXiv:1801.04153.
• Xu, W. and Stein, M. L. (2017). Maximum likelihood estimation for a smooth Gaussian random field model. SIAM/ASA J. Uncertain. Quantificat. 5 138–175.
• Yang, Y. and Dunson, D. B. (2016). Bayesian manifold regression. Ann. Statist. 44 876–905.
• Yu, Y., Debevec, P., Malik, J. and Hawkins, T. (1999). Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proc. Ann. Conf. Comput. Graph. Int. Tech. 215–224.