Bayesian Analysis

Variational Inference for Count Response Semiparametric Regression

J. Luts and M. P. Wand

Full-text: Open access


Fast variational approximate algorithms are developed for Bayesian semiparametric regression when the response variable is a count, i.e., a non-negative integer. We treat both the Poisson and Negative Binomial families as models for the response variable. Our approach utilizes recently developed methodology known as non-conjugate variational message passing. For concreteness, we focus on generalized additive mixed models, although our variational approximation approach extends to a wide class of semiparametric regression models such as those containing interactions and elaborate random effect structure.

Article information

Bayesian Anal. Volume 10, Number 4 (2015), 991-1023.

First available in Project Euclid: 4 February 2015

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

approximate Bayesian inference generalized additive mixed models mean field variational Bayes penalized splines real-time semiparametric regression


Luts, J.; Wand, M. P. Variational Inference for Count Response Semiparametric Regression. Bayesian Anal. 10 (2015), no. 4, 991--1023. doi:10.1214/14-BA932.

Export citation


  • Albert, J. H. and Chib, S. (1993). “Bayesian analysis of binary and polychotomous response data.” Journal of the American Statistical Association, 88: 669–679.
  • Barber, D. and Bishop, C. M. (1998). “Ensemble learning for multi-layer networks.” In: Jordan, M. I., Kearns, K. J., and Solla, S. A. (eds.), Advances in Neural Information Processing Systems, 10, 395–401. Cambridge, Massachusetts: MIT Press.
  • Bishop, C. M. (2008). “A new framework for machine learning.” In: J. M. Zurada et al., (eds.), World Congress on Computational Intelligence, 2008 Plenary/Invited Lectures, Lecture Notes in Computer Science, 5050, 1–24. Heidelberg, Germany: Springer-Verlag,.
  • Chopin, N., Jacob, P. E., and Papaspiliopoulos, O. (2013). “SMC2: an efficient algorithm for sequential analysis of state space models.” Journal of the Royal Statistical Society: Series B, 75: 397–426.
  • Consonni, G. and Marin, J.-M. (2007). “Mean-field variational approximate Bayesian inference for latent variable models.” Computational Statistics and Data Analysis, 52: 790–798.
  • Damien, P., Wakefield, J., and Walker, S. (1999). “Gibbs sampling for Bayesian non-conjugate and hierarchical models by using auxiliary variables.” Journal of the Royal Statistical Society, Series B, 61: 331–345.
  • Efron, B. (1979). “Bootstrap methods: another look at the jackknife.” The Annals of Statistics, 7: 1–26.
  • Fan, Y., Leslie, D. S. and Wand, M. P. (2008). “Generalised linear mixed model analysis via sequential Monte Carlo sampling.” Electronic Journal of Statistics, 2: 916–938.
  • Frühwirth-Schnatter, S., Frühwirth, R., Held, L., and Rue, H. (2009). “Improved auxiliary mixture sampling for hierarchical models of non-Gaussian data.” Statistics and Computing, 19: 479–492.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models.” Bayesian Analysis, 1: 515–533.
  • Girolami, M. and Rogers, S. (2006). “Variational Bayesian multinomial probit regression.” Neural Computation, 18: 1790–1817.
  • Honkela, A., Raiko, T., Kuusela, M., Tornio, M. and Karhunen, J. (2010). “Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes.” Journal of Machine Learning Research, 11: 3235–3268.
  • Jaakkola, T. S. and Jordan, M. I. (2000). “Bayesian parameter estimation via variational methods.” Statistics and Computing, 10: 25–37.
  • Knowles, D. A. and Minka, T. P. (2011). “Non-conjugate message passing for multinomial and binary regression.” In: Shawe-Taylor, J., Zamel, R. S., Bartlett, P., Pereira, F., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 24, 1701–1709.
  • Lawless, J. F. (1987). “Negative Binomial and mixed Poisson regression.” Canadian Journal of Statistics, 15: 209–225.
  • Leetaru, K. H. and Schrodt, P. A. (2013). “A 30-year georeferenced global event database: The Global Database of Events, Language, and Tone (GDELT).” In: International Studies Association Conference. San Francisco, USA.
  • Li, Y. and Ruppert, D. (2008). “On the asymptotics of penalized splines.” Biometrika, 95: 415–436.
  • Luts, J., Broderick, T., and Wand, M. P. (2014). “Real-time semiparametric regression.” Journal of Computational and Graphical Statistics, 23: 589–615.
  • Magnus, J. R. and Neudecker, H. (1999). Matrix Differential Calculus with Applications in Statistics and Econometrics, Revised Edition. Chichester UK: Wiley.
  • Marley, J. K. and Wand, M. P. (2010). “Non-standard semiparametric regression via BRugs.” Journal of Statistical Software, 37: 1–30.
  • Menictas, M. and Wand, M. P. (2015). “Variational inference for heteroscedastic semiparametric regression.” Australian and New Zealand Journal of Statistics, 57: in press.
  • Michalak, S., DuBois, A., DuBois, D., Vander Wiel, S., and Hogden, J. (2012). “Developing systems for real-time streaming analysis.” Journal of Computational and Graphical Statistics, 21: 561–580.
  • Minka, T. P. (2001). “Expectation propagation for approximate Bayesian inference.” In: Proceedings of Conference on Uncertainty in Artificial Intelligence, 362–369.
  • Mittal, S., Madigan, D., Burd, R. S., and Suchard, M. A. (2013). “High-dimensional, massive sample-size Cox proportional hazards regression for survival analysis.” Biostatistics, 15: 287–294.
  • O’Sullivan, F. (1986). “A statistical perspective on ill-posed inverse problems (with discussion).” Statistical Science, 1: 502–527.
  • R Development Core Team (2015). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
  • Rue, H., Martino, S., and Chopin, N. (2009). “Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion).” Journal of the Royal Statistical Society, Series B, 71: 319–392.
  • Ruppert, D., Wand, M. P., and Carroll, R. J. (2003). Semiparametric Regression. New York: Cambridge University Press.
  • — (2009). “Semiparametric regression during 2003–2007.” Electronic Journal of Statistics, 3: 1193–1256.
  • Spiegelhalter, D. J., Thomas, A., Best, N. G., Gilks, W. R., and Lunn, D. (2003). BUGS: Bayesian inference using Gibbs sampling. Medical Research Council Biostatistics Unit, Cambridge, UK.
  • Tan, L. S. L. and Nott, D. J. (2013). “Variational inference for generalized linear mixed models using partially noncentred parametrizations.” Statistical Science, 28: 168–188.
  • Thurston, S. W., Wand, M. P., and Weincke, J. K. (2000). “Negative binomial additive models.” Biometrics, 56: 139–144.
  • Wainwright, M. J. and Jordan, M. I. (2008). “Graphical models, exponential families, and variational inference.” Foundation and Trends in Machine Learning, 1: 1–305.
  • Wand, M. P. (2014). “Fully simplified Multivariate Normal updates in non-conjugate variational message passing.” Journal of Machine Learning Research, 15: 1351–1369.
  • Wand, M. P. and Ormerod, J. T. (2008). “On O’Sullivan penalised splines and semiparametric regression.” Australian and New Zealand Journal of Statistics, 50: 179–198.
  • Wand, M. P., Ormerod, J. T., Padoan, S. A., and Frühwirth, R. (2011). “Mean field variational Bayes for elaborate distributions.” Bayesian Analysis, 6: 847–900.
  • Wiencke, J., Thurston, S. W., Kelsey, K. T., Varkonyi, A., Wain, J. C., Mark, E. J., and Christiani, D. C. (1999). “Early age at smoking initiation and tobacco carcinogen DNA damage in the lung.” Journal of the National Cancer Institute, 91: 614–619.