We derive the explicit form of expectation propagation for approximate deterministic Bayesian inference in a simple statistical model. The model corresponds to a random sample from the Normal distribution. The explicit forms, and their derivation, allow a deeper understanding of the issues and challenges involved in practical implementation of expectation propagation for statistical analyses. No auxiliary approximations are used: we follow the expectation propagation prescription exactly. A simulation study shows expectation propagation to be more accurate than mean field variational Bayes for larger sample sizes, but at the cost of considerably more algebraic and computational effort.
References
[1] Bakker, B. & Heskes, T. (2007), ‘Learning and approximate inference in dynamic hierarchical models’, Computational Statistics and Data Analysis, 52, 821–839. MR2418531[1] Bakker, B. & Heskes, T. (2007), ‘Learning and approximate inference in dynamic hierarchical models’, Computational Statistics and Data Analysis, 52, 821–839. MR2418531
[3] Frey, B.J. & MacKay, D.J.C. (1998). ‘A revolution: Belief propagation in graphs with cycles’, In Jordan, M.I., Kearns, M.J., &, Solla S.A. (eds.), Advances in Neural Information Processing Systems 10, pp. 479–485.[3] Frey, B.J. & MacKay, D.J.C. (1998). ‘A revolution: Belief propagation in graphs with cycles’, In Jordan, M.I., Kearns, M.J., &, Solla S.A. (eds.), Advances in Neural Information Processing Systems 10, pp. 479–485.
[4] Guo, B.-N. & Qi, F. (2013), ‘Refinements of lower bounds for polygamma functions’, Proceedings of the American Mathematical Society, 141, 1007–1015. MR3003692 10.1090/S0002-9939-2012-11387-5[4] Guo, B.-N. & Qi, F. (2013), ‘Refinements of lower bounds for polygamma functions’, Proceedings of the American Mathematical Society, 141, 1007–1015. MR3003692 10.1090/S0002-9939-2012-11387-5
[5] Luts, J., Broderick, T. & Wand, M.P. (2014), ‘Real-time semiparametric regression’, Journal of Computational and Graphical Statistics, 23, 589–615. MR3224647 10.1080/10618600.2013.810150[5] Luts, J., Broderick, T. & Wand, M.P. (2014), ‘Real-time semiparametric regression’, Journal of Computational and Graphical Statistics, 23, 589–615. MR3224647 10.1080/10618600.2013.810150
[7] Minka, T.P. (2001), ‘Expectation propagation for approximate Bayesian inference’. In Breese, J.S. & Koller, D. (eds), Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pp. 362–369. Morgan Kaufmann, Burlington, Massachusetts.[7] Minka, T.P. (2001), ‘Expectation propagation for approximate Bayesian inference’. In Breese, J.S. & Koller, D. (eds), Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pp. 362–369. Morgan Kaufmann, Burlington, Massachusetts.
[10] Minka, T., Winn, J., Guiver, J. & Knowles, D. (2013), Infer.NET 2.5, http://research.microsoft.com/infernet.[10] Minka, T., Winn, J., Guiver, J. & Knowles, D. (2013), Infer.NET 2.5, http://research.microsoft.com/infernet.
[11] R Development Core Team (2015), ‘R: A language and environment for statistical computing’, R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org[11] R Development Core Team (2015), ‘R: A language and environment for statistical computing’, R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org
[12] Smyth, G. (2013), ‘statmod 1.4. Statistical modeling’, http://cran.r-project.org[12] Smyth, G. (2013), ‘statmod 1.4. Statistical modeling’, http://cran.r-project.org
[13] Stan Development Team (2016), ‘rstan 2.9: R interface to Stan’, Version 2.9, http://mc-stan.org[13] Stan Development Team (2016), ‘rstan 2.9: R interface to Stan’, Version 2.9, http://mc-stan.org
[14] Wainwright, M.J. & Jordan, M.I. (2008), ‘Graphical models, exponential families, and variational inference’, Foundations and Trends in Machine Learning, 1, 1–305.[14] Wainwright, M.J. & Jordan, M.I. (2008), ‘Graphical models, exponential families, and variational inference’, Foundations and Trends in Machine Learning, 1, 1–305.
[15] Wand, M.P. & Jones, M.C. (1993), ‘Comparison of smoothing parameterizations in bivariate kernel density estimation’, Journal of the American Statistical Association, 88, 520–528. MR1224377 10.1080/01621459.1993.10476303[15] Wand, M.P. & Jones, M.C. (1993), ‘Comparison of smoothing parameterizations in bivariate kernel density estimation’, Journal of the American Statistical Association, 88, 520–528. MR1224377 10.1080/01621459.1993.10476303
[16] Wand M.P. & Ripley, B.D. (2010), ‘KernSmooth 2.23. Functions for kernel smoothing for Wand & Jones (1995)’, http://cran.r-project.org[16] Wand M.P. & Ripley, B.D. (2010), ‘KernSmooth 2.23. Functions for kernel smoothing for Wand & Jones (1995)’, http://cran.r-project.org
[17] Wand, M.P., Ormerod, J.T., Padoan, S.A. & Frühwirth, R. (2011), ‘Mean field variational Bayes for elaborate distributions’, Bayesian Analysis, 6, 847–900. MR2869967 10.1214/11-BA631 euclid.ba/1339616546
[17] Wand, M.P., Ormerod, J.T., Padoan, S.A. & Frühwirth, R. (2011), ‘Mean field variational Bayes for elaborate distributions’, Bayesian Analysis, 6, 847–900. MR2869967 10.1214/11-BA631 euclid.ba/1339616546
[19] Zoeter, O. & Heskes, T. (2005), ‘Gaussian quadrature based expectation propagation’, In Ghahramani, Z. and Cowell, R. (eds.), Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics pp. 445–452. The Society for Artificial Intelligence and Statistics.[19] Zoeter, O. & Heskes, T. (2005), ‘Gaussian quadrature based expectation propagation’, In Ghahramani, Z. and Cowell, R. (eds.), Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics pp. 445–452. The Society for Artificial Intelligence and Statistics.
