Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models

Abstract

Bayesian regression trees are flexible non-parametric models that are well suited to many modern statistical regression problems. Many such tree models have been proposed, from the simple single-tree model to more complex tree ensembles. Their nonparametric formulation allows one to model datasets exhibiting complex non-linear relationships between the model predictors and observations. However, the mixing behavior of the Markov Chain Monte Carlo (MCMC) sampler is sometimes poor, frequently suffering from local mode stickiness and poor mixing. This is because existing Metropolis–Hastings proposals do not allow for efficient traversal of the model space. We develop novel Metropolis–Hastings proposals that account for the topological structure of regression trees. The first is a novel tree rotation proposal that only requires local changes to the regression tree structure, yet efficiently traverses disparate regions of the model space along contours of high likelihood. The second is a rule perturbation proposal which can be seen as an efficient variation of the change proposal found in existing literature. We implement these samplers and demonstrate their effectiveness on a prediction problem from computer experiments, a test function where structural tree variability is needed to fully explore the posterior and data from a heart rate study.