Bayesian Analysis

Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models

Matthew T. Pratola

Full-text: Open access

Abstract

Bayesian regression trees are flexible non-parametric models that are well suited to many modern statistical regression problems. Many such tree models have been proposed, from the simple single-tree model to more complex tree ensembles. Their nonparametric formulation allows one to model datasets exhibiting complex non-linear relationships between the model predictors and observations. However, the mixing behavior of the Markov Chain Monte Carlo (MCMC) sampler is sometimes poor, frequently suffering from local mode stickiness and poor mixing. This is because existing Metropolis–Hastings proposals do not allow for efficient traversal of the model space. We develop novel Metropolis–Hastings proposals that account for the topological structure of regression trees. The first is a novel tree rotation proposal that only requires local changes to the regression tree structure, yet efficiently traverses disparate regions of the model space along contours of high likelihood. The second is a rule perturbation proposal which can be seen as an efficient variation of the change proposal found in existing literature. We implement these samplers and demonstrate their effectiveness on a prediction problem from computer experiments, a test function where structural tree variability is needed to fully explore the posterior and data from a heart rate study.

Article information

Source
Bayesian Anal., Volume 11, Number 3 (2016), 885-911.

Dates
First available in Project Euclid: 7 March 2016

Permanent link to this document
https://projecteuclid.org/euclid.ba/1457383101

Digital Object Identifier
doi:10.1214/16-BA999

Mathematical Reviews number (MathSciNet)
MR3543912

Zentralblatt MATH identifier
1357.62178

Keywords
Markov chain Monte Carlo proposal distribution computer experiments uncertainty quantification credible interval coverage probability

Citation

Pratola, Matthew T. Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models. Bayesian Anal. 11 (2016), no. 3, 885--911. doi:10.1214/16-BA999. https://projecteuclid.org/euclid.ba/1457383101


Export citation

References

  • Bleich, J., Kapelner, A., George, E. I., and Jensen, S. (2014). “Variable Selection for BART: An Application to Gene Regulation.” The Annals of Applied Statistics, 8(3): 1750–1781.
  • Chipman, H., George, E., and McCulloch, R. (1998). “Bayesian CART Model Search.” Journal of the American Statistical Association, 93(443): 935–960.
  • Chipman, H., George, E., and McCulloch, R. (2002). “Bayesian Treed Models.” Machine Learning, 48: 299–320.
  • Chipman, H., George, E., and McCulloch, R. (2010). “BART: Bayesian Additive Regression Trees.” The Annals of Applied Statistics, 4(1): 266–298.
  • Denison, D., Mallick, B., and Smith, A. (1998). “A Bayesian CART Algorithm.” Biometrika, 85(2): 363–377.
  • Friedman, J. H. (1991). “Multivariate Adaptive Regression Splines.” The Annals of Statistics, 19: 1–67.
  • Geweke, J. (1992). “Evaluating the Accuracy of Sampling-Based Approaches to the Calculation of Posterior Moments.” In: Bayesian Statistics 4. Oxford University Press.
  • Gramacy, R. and Lee, H. (2008). “Bayesian Treed Gaussian Process Models with an Application to Computer Modeling.” Journal of the American Statistical Association, 103(483): 1119–1130.
  • Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008). “Computer Model Calibration Using High-Dimensional Output.” Journal of the American Statistical Association, 103(482): 570–583.
  • Kennedy, M. and O’Hagan, A. (2001). “Bayesian Calibration of Computer Models (with Discussion).” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68: 425–464.
  • Lakshminarayanan, B., Roy, D. M., and Teh, Y. W. (2013). “Top-Down Particle Filtering for Bayesian Decision Trees.” In: Proceedings of the International Conference on Machine Learning. ICML.
  • Lakshminarayanan, B., Roy, D. M., and Teh, Y. W. (2015). “Particle Gibbs for Bayesian Additive Regression Trees.” arXiv:1502.04622v1.
  • Løland, A. and Høst, G. (2003). “Spatial Covariance Modeling in a Complex Coastal Domain by Multidimensional Scaling.” Environmetrics, 14: 307–321.
  • Oakley, J. and O’Hagan, A. (2002). “Bayesian Inference for the Uncertainty Distribution of Computer Model Outputs.” Biometrika, 89(4): 769–784.
  • Pratola, M. T. (2016). “Supplementary Material of “Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models”.” Bayesian Analysis.
  • Pratola, M., Chipman, H., Gattiker, J., Higdon, D., McCulloch, R., and Rust, W. (2014). “Parallel Bayesian Additive Regression Trees.” Journal of Computational and Graphical Statistics, 23: 830–852.
  • Pratola, M. T., Harari, O., Bingham, D., and Flowers, G. E. (2015). “Design and Analysis of Experiments on Non-Convex Regions.” Technometrics.
  • Rathbun, S. L. (1998). “Spatial Modeling in Irregularly Shaped Regions: Kriging Estuaries.” Environmetrics, 9: 109–129.
  • Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). “Design and Analysis of Computer Experiments (with Discussion).” Statistical Science, 4: 409–423.
  • Sleator, D., Tarjan, R., and Thurston, W. (1988). “Rotation Distance, Triangulations, and Hyperbolic Geometry.” Journal of the American Mathematical Society, 3: 647–681.
  • Taddy, M., Gramacy, R., and Polson, N. (2011). “Dynamic Trees for Learning and Design.” Journal of the American Statistical Association, 106(493): 109–123.
  • Wu, Y., Tjelmeland, H., and West, M. (2007). “Bayesian CART: Prior Specification and Posterior Simulation.” Journal of Computational and Graphical Statistics, 16(1): 44–66.

See also

  • Related item: Robert B. Gramacy (2016). Comment on Article by Pratola. Bayesian Anal. Vol. 11, Iss. 3, 913–919.
  • Related item: Christopher M. Hans (2016). Comment on Article by Pratola. Bayesian Anal. Vol. 11, Iss. 3, 921–927.
  • Related item: Oksana A. Chkrebtii, Scotland Leman, Andrew Hoegh, Reihaneh Entezari, Radu V. Craiu, Jeffrey S. Rosenthal, Abdolreza Mohammadi, Maurits Kaptein, Luca Martino, Rafael B. Stern, Francisco Louzada (2016). Contributed Discussion on Article by Pratola. Bayesian Anal. Vol. 11, Iss. 3, 929–943.
  • Related item: Matthew T. Pratola (2016). Rejoinder. Bayesian Anal. Vol. 11, Iss. 3, 945–955.

Supplemental materials