Bayesian Analysis

Variational Inference for Count Response Semiparametric Regression

J. Luts and M. P. Wand

Full-text: Open access

Abstract

Fast variational approximate algorithms are developed for Bayesian semiparametric regression when the response variable is a count, i.e., a non-negative integer. We treat both the Poisson and Negative Binomial families as models for the response variable. Our approach utilizes recently developed methodology known as non-conjugate variational message passing. For concreteness, we focus on generalized additive mixed models, although our variational approximation approach extends to a wide class of semiparametric regression models such as those containing interactions and elaborate random effect structure.

Article information

Source
Bayesian Anal. Volume 10, Number 4 (2015), 991-1023.

Dates
First available in Project Euclid: 4 February 2015

Permanent link to this document
https://projecteuclid.org/euclid.ba/1423083636

Digital Object Identifier
doi:10.1214/14-BA932

Mathematical Reviews number (MathSciNet)
MR3432247

Zentralblatt MATH identifier
1335.62054

Keywords
approximate Bayesian inference generalized additive mixed models mean field variational Bayes penalized splines real-time semiparametric regression

Citation

Luts, J.; Wand, M. P. Variational Inference for Count Response Semiparametric Regression. Bayesian Anal. 10 (2015), no. 4, 991--1023. doi:10.1214/14-BA932. https://projecteuclid.org/euclid.ba/1423083636


Export citation

References

  • Albert, J. H. and Chib, S. (1993). “Bayesian analysis of binary and polychotomous response data.” Journal of the American Statistical Association, 88: 669–679.
  • Barber, D. and Bishop, C. M. (1998). “Ensemble learning for multi-layer networks.” In: Jordan, M. I., Kearns, K. J., and Solla, S. A. (eds.), Advances in Neural Information Processing Systems, 10, 395–401. Cambridge, Massachusetts: MIT Press.
  • Bishop, C. M. (2008). “A new framework for machine learning.” In: J. M. Zurada et al., (eds.), World Congress on Computational Intelligence, 2008 Plenary/Invited Lectures, Lecture Notes in Computer Science, 5050, 1–24. Heidelberg, Germany: Springer-Verlag,.
  • Chopin, N., Jacob, P. E., and Papaspiliopoulos, O. (2013). “SMC2: an efficient algorithm for sequential analysis of state space models.” Journal of the Royal Statistical Society: Series B, 75: 397–426.
  • Consonni, G. and Marin, J.-M. (2007). “Mean-field variational approximate Bayesian inference for latent variable models.” Computational Statistics and Data Analysis, 52: 790–798.
  • Damien, P., Wakefield, J., and Walker, S. (1999). “Gibbs sampling for Bayesian non-conjugate and hierarchical models by using auxiliary variables.” Journal of the Royal Statistical Society, Series B, 61: 331–345.
  • Efron, B. (1979). “Bootstrap methods: another look at the jackknife.” The Annals of Statistics, 7: 1–26.
  • Fan, Y., Leslie, D. S. and Wand, M. P. (2008). “Generalised linear mixed model analysis via sequential Monte Carlo sampling.” Electronic Journal of Statistics, 2: 916–938.
  • Frühwirth-Schnatter, S., Frühwirth, R., Held, L., and Rue, H. (2009). “Improved auxiliary mixture sampling for hierarchical models of non-Gaussian data.” Statistics and Computing, 19: 479–492.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models.” Bayesian Analysis, 1: 515–533.
  • Girolami, M. and Rogers, S. (2006). “Variational Bayesian multinomial probit regression.” Neural Computation, 18: 1790–1817.
  • Honkela, A., Raiko, T., Kuusela, M., Tornio, M. and Karhunen, J. (2010). “Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes.” Journal of Machine Learning Research, 11: 3235–3268.
  • Jaakkola, T. S. and Jordan, M. I. (2000). “Bayesian parameter estimation via variational methods.” Statistics and Computing, 10: 25–37.
  • Knowles, D. A. and Minka, T. P. (2011). “Non-conjugate message passing for multinomial and binary regression.” In: Shawe-Taylor, J., Zamel, R. S., Bartlett, P., Pereira, F., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 24, 1701–1709.
  • Lawless, J. F. (1987). “Negative Binomial and mixed Poisson regression.” Canadian Journal of Statistics, 15: 209–225.
  • Leetaru, K. H. and Schrodt, P. A. (2013). “A 30-year georeferenced global event database: The Global Database of Events, Language, and Tone (GDELT).” In: International Studies Association Conference. San Francisco, USA.
  • Li, Y. and Ruppert, D. (2008). “On the asymptotics of penalized splines.” Biometrika, 95: 415–436.
  • Luts, J., Broderick, T., and Wand, M. P. (2014). “Real-time semiparametric regression.” Journal of Computational and Graphical Statistics, 23: 589–615.
  • Magnus, J. R. and Neudecker, H. (1999). Matrix Differential Calculus with Applications in Statistics and Econometrics, Revised Edition. Chichester UK: Wiley.
  • Marley, J. K. and Wand, M. P. (2010). “Non-standard semiparametric regression via BRugs.” Journal of Statistical Software, 37: 1–30.
  • Menictas, M. and Wand, M. P. (2015). “Variational inference for heteroscedastic semiparametric regression.” Australian and New Zealand Journal of Statistics, 57: in press.
  • Michalak, S., DuBois, A., DuBois, D., Vander Wiel, S., and Hogden, J. (2012). “Developing systems for real-time streaming analysis.” Journal of Computational and Graphical Statistics, 21: 561–580.
  • Minka, T. P. (2001). “Expectation propagation for approximate Bayesian inference.” In: Proceedings of Conference on Uncertainty in Artificial Intelligence, 362–369.
  • Mittal, S., Madigan, D., Burd, R. S., and Suchard, M. A. (2013). “High-dimensional, massive sample-size Cox proportional hazards regression for survival analysis.” Biostatistics, 15: 287–294.
  • O’Sullivan, F. (1986). “A statistical perspective on ill-posed inverse problems (with discussion).” Statistical Science, 1: 502–527.
  • R Development Core Team (2015). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/.
  • Rue, H., Martino, S., and Chopin, N. (2009). “Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion).” Journal of the Royal Statistical Society, Series B, 71: 319–392.
  • Ruppert, D., Wand, M. P., and Carroll, R. J. (2003). Semiparametric Regression. New York: Cambridge University Press.
  • — (2009). “Semiparametric regression during 2003–2007.” Electronic Journal of Statistics, 3: 1193–1256.
  • Spiegelhalter, D. J., Thomas, A., Best, N. G., Gilks, W. R., and Lunn, D. (2003). BUGS: Bayesian inference using Gibbs sampling. Medical Research Council Biostatistics Unit, Cambridge, UK. http://www.mrc-bsu.cam.ac.uk/bugs
  • Tan, L. S. L. and Nott, D. J. (2013). “Variational inference for generalized linear mixed models using partially noncentred parametrizations.” Statistical Science, 28: 168–188.
  • Thurston, S. W., Wand, M. P., and Weincke, J. K. (2000). “Negative binomial additive models.” Biometrics, 56: 139–144.
  • Wainwright, M. J. and Jordan, M. I. (2008). “Graphical models, exponential families, and variational inference.” Foundation and Trends in Machine Learning, 1: 1–305.
  • Wand, M. P. (2014). “Fully simplified Multivariate Normal updates in non-conjugate variational message passing.” Journal of Machine Learning Research, 15: 1351–1369.
  • Wand, M. P. and Ormerod, J. T. (2008). “On O’Sullivan penalised splines and semiparametric regression.” Australian and New Zealand Journal of Statistics, 50: 179–198.
  • Wand, M. P., Ormerod, J. T., Padoan, S. A., and Frühwirth, R. (2011). “Mean field variational Bayes for elaborate distributions.” Bayesian Analysis, 6: 847–900.
  • Wiencke, J., Thurston, S. W., Kelsey, K. T., Varkonyi, A., Wain, J. C., Mark, E. J., and Christiani, D. C. (1999). “Early age at smoking initiation and tobacco carcinogen DNA damage in the lung.” Journal of the National Cancer Institute, 91: 614–619.