Open Access
December 2006 Variational Bayesian learning of directed graphical models with hidden variables
Matthew J. Beal, Zoubin Ghahramani
Bayesian Anal. 1(4): 793-831 (December 2006). DOI: 10.1214/06-BA126

Abstract

A key problem in statistics and machine learning is inferring suitable structure of a model given some observed data. A Bayesian approach to model comparison makes use of the marginal likelihood of each candidate model to form a posterior distribution over models; unfortunately for most models of interest, notably those containing hidden or latent variables, the marginal likelihood is intractable to compute.

We present the variational Bayesian (VB) algorithm for directed graphical models, which optimises a lower bound approximation to the marginal likelihood in a procedure similar to the standard EM algorithm. We show that for a large class of models, which we call conjugate exponential, the VB algorithm is a straightforward generalisation of the EM algorithm that incorporates uncertainty over model parameters. In a thorough case study using a small class of bipartite DAGs containing hidden variables, we compare the accuracy of the VB approximation to existing asymptotic-data approximations such as the Bayesian Information Criterion (BIC) and the Cheeseman-Stutz (CS) criterion, and also to a sampling based gold standard, Annealed Importance Sampling (AIS). We find that the VB algorithm is empirically superior to CS and BIC, and much faster than AIS. Moreover, we prove that a VB approximation can always be constructed in such a way that guarantees it to be more accurate than the CS approximation.

Citation

Download Citation

Matthew J. Beal. Zoubin Ghahramani. "Variational Bayesian learning of directed graphical models with hidden variables." Bayesian Anal. 1 (4) 793 - 831, December 2006. https://doi.org/10.1214/06-BA126

Information

Published: December 2006
First available in Project Euclid: 22 June 2012

zbMATH: 1331.68181
MathSciNet: MR2282207
Digital Object Identifier: 10.1214/06-BA126

Keywords: Approximate Bayesian inference , Bayes factors , directed acyclic graphs , EM algorithm , graphical models , Markov chain Monte Carlo , Model selection , variational Bayes

Rights: Copyright © 2006 International Society for Bayesian Analysis

Vol.1 • No. 4 • December 2006
Back to Top