Bayesian Analysis

Efficient Acquisition Rules for Model-Based Approximate Bayesian Computation

Marko Järvenpää, Michael U. Gutmann, Arijus Pleska, Aki Vehtari, and Pekka Marttinen

Advance publication

This article is in its final form and can be cited using the date of online publication and the DOI.

Full-text: Open access

Abstract

Approximate Bayesian computation (ABC) is a method for Bayesian inference when the likelihood is unavailable but simulating from the model is possible. However, many ABC algorithms require a large number of simulations, which can be costly. To reduce the computational cost, Bayesian optimisation (BO) and surrogate models such as Gaussian processes have been proposed. Bayesian optimisation enables one to intelligently decide where to evaluate the model next but common BO strategies are not designed for the goal of estimating the posterior distribution. Our paper addresses this gap in the literature. We propose to compute the uncertainty in the ABC posterior density, which is due to a lack of simulations to estimate this quantity accurately, and define a loss function that measures this uncertainty. We then propose to select the next evaluation location to minimise the expected loss. Experiments show that the proposed method often produces the most accurate approximations as compared to common BO strategies.

Article information

Source
Bayesian Anal., Advance publication (2018), 28 pages.

Dates
First available in Project Euclid: 18 September 2018

Permanent link to this document
https://projecteuclid.org/euclid.ba/1537258134

Digital Object Identifier
doi:10.1214/18-BA1121

Keywords
approximate Bayesian computation intractable likelihood Gaussian processes Bayesian optimisation sequential experiment design

Rights
Creative Commons Attribution 4.0 International License.

Citation

Järvenpää, Marko; Gutmann, Michael U.; Pleska, Arijus; Vehtari, Aki; Marttinen, Pekka. Efficient Acquisition Rules for Model-Based Approximate Bayesian Computation. Bayesian Anal., advance publication, 18 September 2018. doi:10.1214/18-BA1121. https://projecteuclid.org/euclid.ba/1537258134


Export citation

References

  • Beaumont, M. A., Cornuet, J.-M., Marin, J.-M., and Robert, C. P. (2009). “Adaptive approximate Bayesian computation.” Biometrika, 96(4): 983–990.
  • Beaumont, M. A., Zhang, W., and Balding, D. J. (2002). “Approximate Bayesian computation in population genetics.” Genetics, 162(4): 2025–2035.
  • Bijl, H., Schön, T. B., van Wingerden, J., and Verhaegen, M. (2016). “A sequential Monte Carlo approach to Thompson sampling for Bayesian optimization.” Available at https://arxiv.org/abs/1604.00169.
  • Blum, M. G. B. (2010). “Approximate Bayesian Computation: a nonparametric perspective.” Journal of American Statistical Association, 105(491): 1178–1187.
  • Blum, M. G. B. and François, O. (2010). “Non-linear regression models for Approximate Bayesian Computation.” Statistics and Computing, 20(1): 63–73.
  • Blum, M. G. B., Nunes, M. A., Prangle, D., and Sisson, S. A. (2013). “A comparative review of dimension reduction methods in approximate Bayesian computation.” Statistical Science, 28(2): 189–208.
  • Chevalier, C., Bect, J., Ginsbourger, D., Vazquez, E., Picheny, V., and Richet, Y. (2014). “Fast Parallel Kriging-Based Stepwise Uncertainty Reduction With Application to the Identification of an Excursion Set.” Technometrics, 56(4): 455–465.
  • Diggle, P. J. and Gratton, R. J. (1984). “Monte Carlo Methods of Inference for Implicit Statistical Models.” Journal of the Royal Statistical Society. Series B (Methodological), 46(2): 193–227.
  • Drovandi, C. C., Moores, M. T., and Boys, R. J. (2015). “Accelerating pseudo-marginal MCMC using Gaussian processes.” Available at http://eprints.qut.edu.au/90973/. Accessed 11-9-2017.
  • Fan, Y., Nott, D. J., and Sisson, S. A. (2013). “Approximate Bayesian computation via regression density estimation.” Stat, 2(1): 34–48.
  • Fearnhead, P. and Prangle, D. (2012). “Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation.” Journal of the Royal Statistical Society. Series B: Statistical Methodology, 74(3): 419–474.
  • González, J., Osborne, M., and Lawrence, N. D. (2016). “GLASSES: Relieving The Myopia Of Bayesian Optimisation.” In Proceedings of the Nineteenth International Workshop on Artificial Intelligence and Statistics.
  • Gutmann, M. U. and Corander, J. (2016). “Bayesian optimization for likelihood-free inference of simulator-based statistical models.” Journal of Machine Learning Research, 17(125): 1–47.
  • Gutmann, M. U., Dutta, R., Kaski, S., and Corander, J. (2017). “Likelihood-free inference via classification.” Statistics and Computing.
  • Hartig, F., Calabrese, J. M., Reineking, B., Wiegand, T., and Huth, A. (2011). “Statistical inference for stochastic simulation models–theory and application.” Ecology Letters, 14(8): 816–27.
  • Hennig, P., Osborne, M. A., and Girolami, M. (2015). “Probabilistic numerics and uncertainty in computations.” Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 471(2179): 20150142.
  • Hennig, P. and Schuler, C. J. (2012). “Entropy Search for Information-Efficient Global Optimization.” Journal of Machine Learning Research, 13(1999): 1809–1837.
  • Hernández-Lobato, J. M., Hoffman, M. W., and Ghahramani, Z. (2014). “Predictive Entropy Search for Efficient Global Optimization of Black-box Functions.” Advances in Neural Information Processing Systems 28, 1–9.
  • Jabot, F., Lagarrigues, G., Courbaud, B., and Dumoulin, N. (2014). “A comparison of emulation methods for Approximate Bayesian Computation.” Available at http://arxiv.org/abs/1412.7560.
  • Järvenpää, M., Gutmann, M. U., Pleska, A., Vehtari, A., and Marttinen, P. (2018). “Supplementary material of “Efficient acquisition rules for model-based approximate Bayesian computation”.” Bayesian Analysis.
  • Järvenpää, M., Gutmann, M., Vehtari, A., and Marttinen, P. (2017). “Gaussian process modeling in approximate Bayesian computation to estimate horizontal gene transfer in bacteria.” Available at https://arxiv.org/abs/1610.06462.
  • Kandasamy, K., Schneider, J., and Póczos, B. (2015). “Bayesian active learning for posterior estimation.” In International Joint Conference on Artificial Intelligence, 3605–3611.
  • Lenormand, M., Jabot, F., and Deffuant, G. (2013). “Adaptive approximate Bayesian computation for complex models.” Computational Statistics, 28(6): 2777–2796.
  • Lintusaari, J., Gutmann, M. U., Dutta, R., Kaski, S., and Corander, J. (2017). “Fundamentals and Recent Developments in Approximate Bayesian Computation.” Systematic Biology, 66(1): e66–e82.
  • Lintusaari, J., Vuollekoski, H., Kangasrääsiö, A., Skytén, K., Järvenpää, M., Gutmann, M. U., Vehtari, A., Corander, J., and Kaski, S. (2018). “ELFI: Engine for Likelihood Free Inference.” Journal of Machine Learning Research. Accepted for publication.
  • Marin, J. M., Pudlo, P., Robert, C. P., and Ryder, R. J. (2012). “Approximate Bayesian computational methods.” Statistics and Computing, 22(6): 1167–1180.
  • Marjoram, P., Molitor, J., Plagnol, V., and Tavare, S. (2003). “Markov chain Monte Carlo without likelihoods.” Proceedings of the National Academy of Sciences of the United States of America, 100(26): 15324–8.
  • Marttinen, P., Gutmann, M. U., Croucher, N. J., Hanage, W. P., and Corander, J. (2015). “Recombination produces coherent bacterial species clusters in both core and accessory genomes.” Microbial Genomics, 1(5).
  • Meeds, E. and Welling, M. (2014). “GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computation.” In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence.
  • Murray, I. and Adams, R. P. (2010). “Slice sampling covariance hyperparameters of latent Gaussian models.” Advances in Neural Information Processing Systems, 2(1): 9.
  • Numminen, E., Cheng, L., Gyllenberg, M., and Corander, J. (2013). “Estimating the Transmission Dynamics of Streptococcus pneumoniae from Strain Prevalence Data.” Biometrics, 69(3): 748–757.
  • Osborne, M. A., Duvenaud, D., Garnett, R., Rasmussen, C. E., Roberts, S. J., and Ghahramani, Z. (2012). “Active Learning of Model Evidence Using Bayesian Quadrature.” Advances in Neural Information Processing Systems 26, 1–9.
  • Owen, D. B. (1980). “A table of normal integrals.” Communications in Statistics – Simulation and Computation, 9(4): 389–419.
  • Papamakarios, G. and Murray, I. (2016). “Fast e-free inference of simulation models with Bayesian conditional density estimation.” In Advances in Neural Information Processing Systems 29.
  • Patefield, M. and Tandy, D. (2000). “Fast and accurate Calculation of Owen’s T-Function.” Journal of Statistical Software, 5(5): 1–25.
  • Price, L. F., Drovandi, C. C., Lee, A., and Nott, D. J. (2017). “Bayesian synthetic likelihood.” Journal of Computational and Graphical Statistics. (In Press).
  • Rasmussen, C. E. (2003). “Gaussian Processes to Speed up Hybrid Monte Carlo for Expensive Bayesian Integrals.” Bayesian Statistics, 7, 651–659.
  • Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. The MIT Press.
  • Robert, C. P. and Casella, G. (2004). Monte Carlo Statistical Methods. New York: Springer, second edition.
  • Rue, H., Martino, S., and Chopin, N. (2009). “Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations.” Journal of the Royal Statistical Society. Series B: Statistical Methodology, 71(2): 319–392.
  • Ryan, E. G., Drovandi, C. C., Mcgree, J. M., and Pettitt, A. N. (2016). “A Review of Modern Computational Algorithms for Bayesian Optimal Design.” International Statistical Review, 84(1): 128–154.
  • Shahriari, B., Bouchard-Côté, A., and de Freitas, N. (2016). “Unbounded Bayesian Optimization via Regularization.” In International Conference on Artificial Intelligence and Statistics.
  • Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. (2015). “Taking the human out of the loop: A review of Bayesian optimization.” Proceedings of the IEEE, 104(1).
  • Siivola, E., Vehtari, A., Vanhatalo, J., and González, J. (2017). “Bayesian optimization with virtual derivative sign observations.” Available at https://arxiv.org/abs/1704.00963.
  • Sisson, S. A., Fan, Y., and Tanaka, M. M. (2007). “Sequential Monte Carlo without likelihoods.” Proceedings of the National Academy of Sciences of the United States of America, 104(6): 1760–5.
  • Snoek, J., Larochelle, H., and Adams, R. P. (2012). “Practical Bayesian optimization of machine learning algorithms.” In Advances in Neural Information Processing Systems 25, 1–9.
  • Srinivas, N., Krause, A., Kakade, S., and Seeger, M. (2010). “Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design.” In Proceedings of the 27th International Conference on International Conference on Machine Learning, 1015–1022.
  • Toni, T., Welch, D., Strelkowa, N., Ipsen, A., and Stumpf, M. P. H. (2009). “Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems.” Journal of the Royal Society, Interface, 6(31): 187–202.
  • Turner, B. M. and Van Zandt, T. (2012). “A tutorial on approximate Bayesian computation.” Journal of Mathematical Psychology, 56(2): 69–85.
  • Vanhatalo, J., Pietiläinen, V., and Vehtari, A. (2010). “Approximate inference for disease mapping with sparse Gaussian processes.” Statistics in Medicine, 29(15): 1580–1607.
  • Vanhatalo, J., Riihimäki, J., Hartikainen, J., Jylänki, P., Tolvanen, V., and Vehtari, A. (2013). “GPstuff: Bayesian modeling with Gaussian processes.” Journal of Machine Learning Research, 14: 1175–1179.
  • Wang, Z. and Jegelka, S. (2017). “Max-value Entropy Search for Efficient Bayesian Optimization.” In Proceedings of the 34th International Conference on Machine Learning, 3627–3635.
  • Wang, Z., Zhou, B., and Jegelka, S. (2016). “Optimization as Estimation with Gaussian Processes in Bandit Settings.” In In proceedings of the 19th International Conference on Artificial Intelligence and Statistics.
  • Wilkinson, R. D. (2013). “Approximate Bayesian computation (ABC) gives exact results under the assumption of model error.” Statistical Applications in Genetics and Molecular Biology, 12(2): 129–141.
  • Wilkinson, R. D. (2014). “Accelerating ABC methods using Gaussian processes.” In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics.
  • Wood, S. N. (2010). “Statistical inference for noisy nonlinear ecological dynamic systems.” Nature, 466: 1102–1104.

Supplemental materials

  • Supplementary material of “Efficient acquisition rules for model-based approximate Bayesian computation”. The supplementary material contains proofs and derivations. Additional experimental results are also presented.