Statistical Science

A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation

M. G. B. Blum, M. A. Nunes, D. Prangle, and S. A. Sisson

Full-text: Open access

Abstract

Approximate Bayesian computation (ABC) methods make use of comparisons between simulated and observed summary statistics to overcome the problem of computationally intractable likelihood functions. As the practical implementation of ABC requires computations based on vectors of summary statistics, rather than full data sets, a central question is how to derive low-dimensional summary statistics from the observed data with minimal loss of information. In this article we provide a comprehensive review and comparison of the performance of the principal methods of dimension reduction proposed in the ABC literature. The methods are split into three nonmutually exclusive classes consisting of best subset selection methods, projection techniques and regularization. In addition, we introduce two new methods of dimension reduction. The first is a best subset selection method based on Akaike and Bayesian information criteria, and the second uses ridge regression as a regularization procedure. We illustrate the performance of these dimension reduction techniques through the analysis of three challenging models and data sets.

Article information

Source
Statist. Sci. Volume 28, Number 2 (2013), 189-208.

Dates
First available in Project Euclid: 21 May 2013

Permanent link to this document
https://projecteuclid.org/euclid.ss/1369147911

Digital Object Identifier
doi:10.1214/12-STS406

Mathematical Reviews number (MathSciNet)
MR3112405

Zentralblatt MATH identifier
1331.62123

Keywords
Approximate Bayesian computation dimension reduction likelihood-free inference regularization variable selection

Citation

Blum, M. G. B.; Nunes, M. A.; Prangle, D.; Sisson, S. A. A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation. Statist. Sci. 28 (2013), no. 2, 189--208. doi:10.1214/12-STS406. https://projecteuclid.org/euclid.ss/1369147911.


Export citation

References

  • Abdi, H. and Williams, L. J. (2010). Partial least square regression, projection on latent structure regression. Wiley Interdiscip. Rev. Comput. Stat. 2 433–459.
  • Aeschbacher, S., Beaumont, M. A. and Futschik, A. (2012). A novel approach for choosing summary statistics in approximate Bayesian computation. Genetics 192 1027–1047.
  • Akaike, H. (1974). A new look at the statistical model identification. IEEE Trans. Automat. Control AC-19 716–723.
  • Allingham, D., King, R. A. R. and Mengersen, K. L. (2009). Bayesian estimation of quantile distributions. Stat. Comput. 19 189–201.
  • Baddeley, A. and Jensen, E. B. V. (2004). Stereology for Statisticians. Chapman & Hall/CRC, Boca Raton, FL.
  • Barnes, C., Filippi, S., Stumpf, M. P. H. and Thorne, T. (2012). Considerate approaches to constructing summary statistics for ABC model selection. Stat. Comput. 22 1181–1197.
  • Barthelmé, S. and Chopin, N. (2011). Expectation-propagation for summary-less, likelihood-free inference. Available at http://arxiv.org/abs/1107.5959.
  • Beaumont, M. A. (2010). Approximate Bayesian computation in evolution and ecology. Annual Review of Ecology, Evolution, and Systematics 41 379–406.
  • Beaumont, M. A., Zhang, W. and Balding, D. J. (2002). Approximate Bayesian computation in population genetics. Genetics 162 2025–2035.
  • Beaumont, M. A., Marin, J. M., Cornuet, J. M. and Robert, C. P. (2009). Adaptivity for ABC algorithms: The ABC–PMC scheme. Biometrika 96 983–990.
  • Bertorelle, G., Benazzo, A. and Mona, S. (2010). ABC as a flexible framework to estimate demography over space and time: Some cons, many pros. Mol. Ecol. 19 2609–2625.
  • Blum, M. G. B. (2010a). Approximate Bayesian computation: A nonparametric perspective. J. Amer. Statist. Assoc. 105 1178–1187.
  • Blum, M. G. B. (2010b). Choosing the summary statistics and the acceptance rate in approximate Bayesian computation. In COMPSTAT 2010: Proceedings in Computational Statistics (G. Saporta and Y. Lechevallier, eds.) 47–56. Springer, New York.
  • Blum, M. G. B. and François, O. (2010). Non-linear regression models for approximate Bayesian computation. Stat. Comput. 20 63–73.
  • Blum, M. G. B., Nunes, M. A., Prangle, D. and Sisson, S. A. (2013). Supplement to “A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation.” DOI:10.1214/12-STS406SUPP.
  • Bonassi, F. V., You, L. and West, M. (2011). Bayesian learning from marginal data in bionetwork models. Stat. Appl. Genet. Mol. Biol. 10 Art. 49, 29.
  • Bortot, P., Coles, S. G. and Sisson, S. A. (2007). Inference for stereological extremes. J. Amer. Statist. Assoc. 102 84–92.
  • Boulesteix, A.-L. and Strimmer, K. (2007). Partial least squares: A versatile tool for the analysis of high-dimensional genomic data. Brief. Bioinformatics 8 32–44.
  • Coles, S. (2001). An Introduction to Statistical Modeling of Extreme Values. Springer, London.
  • Csilléry, K., François, O. and Blum, M. G. B. (2012). abc: An R package for approximate Bayesian computation. Methods in Ecology and Evolution 3 475–479.
  • Csilléry, K., Blum, M. G. B., Gaggiotti, O. and François, O. (2010). Approximate Bayesian computation in practice. Trends in Ecology and Evolution 25 410–418.
  • Del Moral, P., Doucet, A. and Jasra, A. (2012). An adaptive sequential Monte Carlo method for approximate Bayesian computation. Stat. Comput. 22 1009–1020.
  • Drovandi, C. C. and Pettitt, A. N. (2011). Estimation of parameters for macroparasite population evolution using approximate Bayesian computation. Biometrics 67 225–233.
  • Drovandi, C. C., Pettitt, A. N. and Faddy, M. J. (2011). Approximate Bayesian computation using indirect inference. J. R. Stat. Soc. Ser. C. Appl. Stat. 60 317–337.
  • Estoup, A., Lombaert, E., Marin, J. M., Guillemaud, T., Pudlo, P., Robert, C. and Cornuet, J. M. (2012). Estimation of demo-genetic model probabilities with approximate Bayesian computation using linear discriminant analysis on summary statistics. Molecular Ecology Resources 12 846–855.
  • Fan, Y., Nott, D. J. and Sisson, S. A. (2012). Regression density estimation ABC. Unpublished manuscript.
  • Fearnhead, P. and Prangle, D. (2012). Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation (with discussion). J. R. Stat. Soc. Ser. B Stat. Methodol. 74 419–474.
  • Filippi, S., Barnes, C. P. and Stumpf, M. P. H. (2012). Contribution to the discussion of Fearnhead and Prangle (2012). Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation. J. R. Stat. Soc. Ser. B Stat. Methodol. 74 459–460.
  • Geman, S., Bienenstock, E. and Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Comput. 4 1–58.
  • Golub, G. H., Heath, M. and Wahba, G. (1979). Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21 215–223.
  • Heggland, K. and Frigessi, A. (2004). Estimating functions in indirect inference. J. R. Stat. Soc. Ser. B Stat. Methodol. 66 447–462.
  • Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12 55–67.
  • Hudson, R. R. (2002). Generating samples under a Wright–Fisher neutral model of genetic variation. Bioinformatics 18 337–338.
  • Hurvich, C. M., Simonoff, J. S. and Tsai, C.-L. (1998). Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. J. R. Stat. Soc. Ser. B Stat. Methodol. 60 271–293.
  • Hurvich, C. M. and Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika 76 297–307.
  • Irizarry, R. A. (2001). Information and posterior probability criteria for model selection in local likelihood estimation. J. Amer. Statist. Assoc. 96 303–315.
  • Jasra, A., Singh, S. S., Martin, J. S. and McCoy, E. (2012). Filtering via approximate Bayesian computation. Statist. Comput. 22 1223–1237.
  • Jeremiah, E., Sisson, S. A., Marshall, L., Mehrotra, R. and Sharma, A. (2011). Bayesian calibration and uncertainty analysis for hydrological models: A comparison of adaptive-Metropolis and sequential Monte Carlo samplers. Water Resources Research 47 W07547, 13pp.
  • Joyce, P. and Marjoram, P. (2008). Approximately sufficient statistics and Bayesian computation. Stat. Appl. Genet. Mol. Biol. 7 Art. 26, 18.
  • Jung, H. and Marjoram, P. (2011). Choice of summary statistic weights in approximate Bayesian computation. Stat. Appl. Genet. Mol. Biol. 10 Art. 45, 25.
  • Konishi, S., Ando, T. and Imoto, S. (2004). Bayesian information criteria and smoothing parameter selection in radial basis function networks. Biometrika 91 27–43.
  • Leuenberger, C. and Wegmann, D. (2010). Bayesian computation and model selection without likelihoods. Genetics 184 243–252.
  • Lopes, J. S. and Beaumont, M. A. (2010). ABC: A useful Bayesian tool for the analysis of population data. Infect. Genet. Evol. 10 826–833.
  • Luciani, F., Sisson, S. A., Jiang, H., Francis, A. R. and Tanaka, M. M. (2009). The epidemiological fitness cost of drug resistance in Mycobacterium tuberculosis. Proc. Natl. Acad. Sci. USA 106 14711–14715.
  • Marjoram, P., Molitor, J., Plagnol, V. and Tavare, S. (2003). Markov chain Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 100 15324–15328.
  • Mevik, B.-H. and Cederkvist, H. R. (2004). Mean squared error of prediction (MSEP) estimates for principal component regression (PCR) and partial least squares regression (PLSR). Journal of Chemometrics 18 422–429.
  • Mevik, B.-H. and Wehrens, R. (2007). The pls package: Principal component and partial least squares regression in R. Journal of Statistical Software 18 1–24.
  • Minka, T. (2001). Expectation propagation for approximate Bayesian inference. Proceedings of Uncertainty in Artificial Intelligence 17 362–369.
  • Nakagome, S., Fukumizu, K. and Mano, S. (2012). Kernel approximate Bayesian computation for population genetic inferences. Available at http://arxiv.org/abs/1205.3246.
  • Nix, D. A. and Weigend, A. S. (1995). Learning local error bars for nonlinear regression. In Advances in Neural Information Processing Systems 7 (NIPS‘94) (G. Tesauo, D. Touretzky and T. Leen, eds.) 489–496. MIT Press, Cambridge.
  • Nordborg, M. (2007). Coalescent theory. In Handbook of Statistical Genetics, 3rd ed. (D. J. Balding, M. J. Bishop and C. Cannings, eds.) 179–208. Wiley, Chichester.
  • Nott, D. J., Fan, Y. and Sisson, S. A. (2012). Contribution to the discussion of Fearnhead and Prangle (2012). Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation. J. R. Stat. Soc. Ser. B Stat. Methodol. 74 466.
  • Nott, D. J., Fan, Y., Marshall, L. and Sisson, S. A. (2013). Approximate Bayesian computation and Bayes linear analysis: Towards high-dimensional approximate Bayesian computation. J. Comput. Graph. Statist. To appear.
  • Nunes, M. A. and Balding, D. J. (2010). On optimal selection of summary statistics for approximate Bayesian computation. Stat. Appl. Genet. Mol. Biol. 9 Art. 34, 16.
  • Peters, G. W., Fan, Y. and Sisson, S. A. (2012). On sequential Monte Carlo, partial rejection control and approximate Bayesian computation. Stat. Comput. 22 1209–1222.
  • Pritchard, J. K., Seielstad, M. T., Perez-Lezaun, A. and Feldman, M. W. (1999). Population growth of human Y chromosomes: A study of Y chromosome microsatellites. Mol. Biol. Evol. 16 1791–1798.
  • Ripley, B. D. (1994). Neural networks and related methods for classification. J. R. Stat. Soc. Ser. B Stat. Methodol. 56 409–456.
  • Schwarz, G. (1978). Estimating the dimension of a model. Ann. Statist. 6 461–464.
  • Sedki, M. A. and Pudlo, P. (2012). Contribution to the discussion of Fearnhead and Prangle (2012). Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation. J. R. Stat. Soc. Ser. B Stat. Methodol. 74 466–467.
  • Shannon, C. E. (1948). A mathematical theory of communication. Bell System Tech. J. 27 379–423, 623–656.
  • Singh, H., Misra, N., Hnizdo, V., Fedorowicz, A. and Demchuk, E. (2003). Nearest neighbor estimates of entropy. Amer. J. Math. Management Sci. 23 301–321.
  • Sisson, S. A., Fan, Y. and Tanaka, M. M. (2007). Sequential Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 104 1760–1765 (electronic).
  • Sisson, S. A. and Fan, Y. (2011). Likelihood-free Markov chain Monte Carlo. In Handbook of Markov Chain Monte Carlo (S. P. Brooks, A. Gelman, G. Jones and X. L. Meng, eds.) 319–341. CRC Press, Boca Raton, FL.
  • Taniguchi, M. and Tresp, V. (1997). Averaging regularized estimators. Neural Comput. 9 1163–1178.
  • Toni, T., Welch, D., Strelkowa, N., Ipsen, A. and Stumpf, M. P. (2009). Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems. Journal of the Royal Society Interface 6 187–202.
  • Vinzi, V. E., Chin, W. W., Henseler, J. and Wang, H., eds. (2010). Handbook of Partial Least Squares: Concepts, Methods and Applications. Springer, Heidelberg.
  • Wegmann, D., Leuenberger, C. and Excoffier, L. (2009). Efficient approximate Bayesian computation coupled with Markov chain Monte Carlo without likelihood. Genetics 182 1207–1218.

Supplemental materials

  • Supplementary material: Supplement to “A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation”. The supplement contains for each of the three examples a comprehensive comparison of the errors obtained with the different methods of dimension reduction.