Bayesian Analysis

Variational Bayesian learning of directed graphical models with hidden variables

Matthew J. Beal and Zoubin Ghahramani

Full-text: Open access


A key problem in statistics and machine learning is inferring suitable structure of a model given some observed data. A Bayesian approach to model comparison makes use of the marginal likelihood of each candidate model to form a posterior distribution over models; unfortunately for most models of interest, notably those containing hidden or latent variables, the marginal likelihood is intractable to compute.

We present the variational Bayesian (VB) algorithm for directed graphical models, which optimises a lower bound approximation to the marginal likelihood in a procedure similar to the standard EM algorithm. We show that for a large class of models, which we call conjugate exponential, the VB algorithm is a straightforward generalisation of the EM algorithm that incorporates uncertainty over model parameters. In a thorough case study using a small class of bipartite DAGs containing hidden variables, we compare the accuracy of the VB approximation to existing asymptotic-data approximations such as the Bayesian Information Criterion (BIC) and the Cheeseman-Stutz (CS) criterion, and also to a sampling based gold standard, Annealed Importance Sampling (AIS). We find that the VB algorithm is empirically superior to CS and BIC, and much faster than AIS. Moreover, we prove that a VB approximation can always be constructed in such a way that guarantees it to be more accurate than the CS approximation.

Article information

Bayesian Anal. Volume 1, Number 4 (2006), 793-831.

First available in Project Euclid: 22 June 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Approximate Bayesian Inference Bayes Factors Directed Acyclic Graphs EM Algorithm Graphical Models Markov Chain Monte Carlo Model Selection Variational Bayes


Beal, Matthew J.; Ghahramani, Zoubin. Variational Bayesian learning of directed graphical models with hidden variables. Bayesian Anal. 1 (2006), no. 4, 793--831. doi:10.1214/06-BA126.

Export citation