Bayesian Analysis

Variational Bayesian learning of directed graphical models with hidden variables

Matthew J. Beal and Zoubin Ghahramani

Full-text: Open access

Abstract

A key problem in statistics and machine learning is inferring suitable structure of a model given some observed data. A Bayesian approach to model comparison makes use of the marginal likelihood of each candidate model to form a posterior distribution over models; unfortunately for most models of interest, notably those containing hidden or latent variables, the marginal likelihood is intractable to compute.

We present the variational Bayesian (VB) algorithm for directed graphical models, which optimises a lower bound approximation to the marginal likelihood in a procedure similar to the standard EM algorithm. We show that for a large class of models, which we call conjugate exponential, the VB algorithm is a straightforward generalisation of the EM algorithm that incorporates uncertainty over model parameters. In a thorough case study using a small class of bipartite DAGs containing hidden variables, we compare the accuracy of the VB approximation to existing asymptotic-data approximations such as the Bayesian Information Criterion (BIC) and the Cheeseman-Stutz (CS) criterion, and also to a sampling based gold standard, Annealed Importance Sampling (AIS). We find that the VB algorithm is empirically superior to CS and BIC, and much faster than AIS. Moreover, we prove that a VB approximation can always be constructed in such a way that guarantees it to be more accurate than the CS approximation.

Article information

Source
Bayesian Anal. Volume 1, Number 4 (2006), 793-831.

Dates
First available in Project Euclid: 22 June 2012

Permanent link to this document
https://projecteuclid.org/euclid.ba/1340370943

Digital Object Identifier
doi:10.1214/06-BA126

Mathematical Reviews number (MathSciNet)
MR2282207

Zentralblatt MATH identifier
1331.68181

Keywords
Approximate Bayesian Inference Bayes Factors Directed Acyclic Graphs EM Algorithm Graphical Models Markov Chain Monte Carlo Model Selection Variational Bayes

Citation

Beal, Matthew J.; Ghahramani, Zoubin. Variational Bayesian learning of directed graphical models with hidden variables. Bayesian Anal. 1 (2006), no. 4, 793--831. doi:10.1214/06-BA126. https://projecteuclid.org/euclid.ba/1340370943


Export citation