Bayesian Analysis

Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model

D. M. Titterington and Bo Wang

Full-text: Open access

Abstract

In this paper we propose a generalised iterative algorithm for calculating variational Bayesian estimates for a normal mixture model and investigate its convergence properties. It is shown theoretically that the variational Bayesian estimator converges locally to the maximum likelihood estimator at the rate of $O(1/{n})$ in the large sample limit.

Article information

Source
Bayesian Anal. Volume 1, Number 3 (2006), 625-650.

Dates
First available in Project Euclid: 22 June 2012

Permanent link to this document
https://projecteuclid.org/euclid.ba/1340371055

Digital Object Identifier
doi:10.1214/06-BA121

Mathematical Reviews number (MathSciNet)
MR2221291

Zentralblatt MATH identifier
1331.62168

Keywords
Mixture model Variational Bayes Local convergence Laplace approximation

Citation

Wang, Bo; Titterington, D. M. Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model. Bayesian Anal. 1 (2006), no. 3, 625--650. doi:10.1214/06-BA121. https://projecteuclid.org/euclid.ba/1340371055


Export citation

References

  • Attias, H. (1999). "Inferring parameters and structure of latent variable models by variational Bayes." In Prade, H. and Laskey, K. (eds.), Proc. 15th Conference on Uncertainty in Artificial Intelligence, 21–30. Stockholm, Sweden: Morgan Kaufmann Publishers.
  • –- (2000). "A variational Bayesian framework for graphical models." In Solla, S., Leen, T., and Muller, K.-R. (eds.), Advances in Neural Information Processing Systems 12, 209–215. Cambridge, MA: MIT Press.
  • Beal, M. J. (2003). "Variational Algorithms for Approximate Bayesian Inference." Ph.D. thesis, University College London.
  • Bhatia, R. (1997). Matrix Analysis. New York: Springer-Verlag.
  • Celeux, G., Hurn, M., and Robert, C. P. (2000). "Computational and inferential difficulties with mixture posterior distributions." Journal of the American Statistical Association, 95(957-979).
  • Corduneanu, A. and Bishop, C. M. (2001). "Variational Bayesian model selection for mixture distributions." In Richardson, T. and Jaakkola, T. (eds.), Proceedings Eighth International Conference on Artificial Intelligence and Statistics, 27–34. Morgan Kaufmann.
  • Duda, R. O. and Hart, P. E. (1973). Pattern Classification and Scene Analysis. New York: John Wiley.
  • Evans, M. and Swartz, T. (2000). Approximating Integrals via Monte Carlo and Deterministic Methods. New York: Oxford University Press.
  • Ghahramani, Z. and Beal, M. J. (2000). "Variational inference for Bayesian mixtures of factor analysers." In Solla, S., Leen, T., and Muller, K.-R. (eds.), Advances in Neural Information Processing Systems 12, 449–455. Cambridge, MA: MIT Press.
  • –- (2001). "Propagation algorithms for variational Bayesian learning." In Leen, T., Dietterich, T., and Tresp, V. (eds.), Advances in Neural Information Processing Systems 13, 507–513. Cambridge, MA: MIT Press.
  • Hall, P., Humphreys, K., and Titterington, D. M. (2002). "On the adequacy of variational lower bound functions for likelihood-based inference in Markovian models with missing values." Journal of the Royal Statistical Society Series B, 64: 549–564.
  • Humphreys, K. and Titterington, D. M. (2000). "Approximate Bayesian inference for simple mixtures." In Bethlehem, J. G. and van der Heijden, P. G. M. (eds.), COMPSTAT2000, 331–336. Heidelberg: Physica-Verlag.
  • Jordan, M. I. (2004). "Graphical Models." Statistical Science, 19(1): 140–155.
  • MacKay, D. J. C. (1997). "Ensemble learning for hidden Markov models." Technical report, Cavendish Laboratory, University of Cambridge.
  • McLachlan, G. J. and Peel, D. (2000). Finite Mixture Models. New York: Wiley.
  • Penny, W. D. and Roberts, S. J. (2000). "Variational Bayes for 1-dimensional mixture models." Technical Report PARG-2000-01, Oxford University.
  • Peters, B. C. and Walker, H. F. (1978). "An iterative procedure for obtaining maximum-likelihood estimates of the parameters for a mixture of normal distributions." SIAM J. Appl. Math., 35: 362–378.
  • Redner, R. A. and Walker, H. F. (1984). "Mixture densities, Maximum likelihood and the EM" Algorithm. SIAM Review, 26: 195–239.
  • Salakhutdinov, R. and Roweis, S. (2003). "Adaptive overrelaxed bound optimization methods." In Fawcett, T. and Mishra, N. (eds.), Proceedings of the Twentieth International Conference on Machine Learning, 664–671. AAAI Press.
  • Titterington, D. M. (2004). "Bayesian methods for neural networks and related models." Statistical Science, 19(1): 128–139.
  • Ueda, N. and Ghahramani, Z. (2003). "Bayesian model search for mixture models based on optimizing variational bounds." Neural Networks, 15: 1223–1241.
  • Wang, B. and Titterington, D. M. (2003). "Local convergence of variational Bayes estimators for mixing coefficients." Technical Report 03-4, University of Glasgow. Http://www.stats.gla.ac.uk/Research/TechRep2003/03-4.pdf.
  • –- (2004). "Lack of consistency of mean field and variational Bayes approximations for state space models." Neural Processing Letters, 20: 151–170.
  • –- (2005). "Inadequacy of interval estimates corresponding to variational Bayesian approximations." In Cowell, R. G. and Ghahramani, Z. (eds.), Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Jan 6-8, 2005, Savannah Hotel, Barbados, 373–380. Society for Artificial Intelligence and Statistics. (Available electronically at http://www.gatsby.ucl.ac.uk/aistats/).