Bayesian Analysis

Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models

Matthew T. Pratola

Full-text: Open access

Enhanced PDF (537 KB)

Abstract

Bayesian regression trees are flexible non-parametric models that are well suited to many modern statistical regression problems. Many such tree models have been proposed, from the simple single-tree model to more complex tree ensembles. Their nonparametric formulation allows one to model datasets exhibiting complex non-linear relationships between the model predictors and observations. However, the mixing behavior of the Markov Chain Monte Carlo (MCMC) sampler is sometimes poor, frequently suffering from local mode stickiness and poor mixing. This is because existing Metropolis–Hastings proposals do not allow for efficient traversal of the model space. We develop novel Metropolis–Hastings proposals that account for the topological structure of regression trees. The first is a novel tree rotation proposal that only requires local changes to the regression tree structure, yet efficiently traverses disparate regions of the model space along contours of high likelihood. The second is a rule perturbation proposal which can be seen as an efficient variation of the change proposal found in existing literature. We implement these samplers and demonstrate their effectiveness on a prediction problem from computer experiments, a test function where structural tree variability is needed to fully explore the posterior and data from a heart rate study.

Article information

Source
Bayesian Anal., Volume 11, Number 3 (2016), 885-911.

Dates
First available in Project Euclid: 7 March 2016

Permanent link to this document
https://projecteuclid.org/euclid.ba/1457383101

Digital Object Identifier
doi:10.1214/16-BA999

Mathematical Reviews number (MathSciNet)
MR3543912

Zentralblatt MATH identifier
1357.62178

Keywords
Markov chain Monte Carlo proposal distribution computer experiments uncertainty quantification credible interval coverage probability

Citation

Pratola, Matthew T. Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models. Bayesian Anal. 11 (2016), no. 3, 885--911. doi:10.1214/16-BA999. https://projecteuclid.org/euclid.ba/1457383101


Export citation

References

  • Bleich, J., Kapelner, A., George, E. I., and Jensen, S. (2014). “Variable Selection for BART: An Application to Gene Regulation.” The Annals of Applied Statistics, 8(3): 1750–1781.
    Mathematical Reviews (MathSciNet): MR3271352
    Digital Object Identifier: doi:10.1214/14-AOAS755
    Project Euclid: euclid.aoas/1414091233
  • Chipman, H., George, E., and McCulloch, R. (1998). “Bayesian CART Model Search.” Journal of the American Statistical Association, 93(443): 935–960.
    Mathematical Reviews (MathSciNet): MR1631325
    Digital Object Identifier: doi:10.1080/01621459.1998.10473707
  • Chipman, H., George, E., and McCulloch, R. (2002). “Bayesian Treed Models.” Machine Learning, 48: 299–320.
  • Chipman, H., George, E., and McCulloch, R. (2010). “BART: Bayesian Additive Regression Trees.” The Annals of Applied Statistics, 4(1): 266–298.
    Mathematical Reviews (MathSciNet): MR2758172
    Digital Object Identifier: doi:10.1214/09-AOAS285
    Project Euclid: euclid.aoas/1273584455
  • Denison, D., Mallick, B., and Smith, A. (1998). “A Bayesian CART Algorithm.” Biometrika, 85(2): 363–377.
    Mathematical Reviews (MathSciNet): MR1649118
    Digital Object Identifier: doi:10.1093/biomet/85.2.363
  • Friedman, J. H. (1991). “Multivariate Adaptive Regression Splines.” The Annals of Statistics, 19: 1–67.
    Mathematical Reviews (MathSciNet): MR1091842
    Digital Object Identifier: doi:10.1214/aos/1176347963
    Project Euclid: euclid.aos/1176347963
  • Geweke, J. (1992). “Evaluating the Accuracy of Sampling-Based Approaches to the Calculation of Posterior Moments.” In: Bayesian Statistics 4. Oxford University Press.
    Mathematical Reviews (MathSciNet): MR1380276
  • Gramacy, R. and Lee, H. (2008). “Bayesian Treed Gaussian Process Models with an Application to Computer Modeling.” Journal of the American Statistical Association, 103(483): 1119–1130.
    Mathematical Reviews (MathSciNet): MR2528830
    Digital Object Identifier: doi:10.1198/016214508000000689
  • Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008). “Computer Model Calibration Using High-Dimensional Output.” Journal of the American Statistical Association, 103(482): 570–583.
    Mathematical Reviews (MathSciNet): MR2523994
    Digital Object Identifier: doi:10.1198/016214507000000888
  • Kennedy, M. and O’Hagan, A. (2001). “Bayesian Calibration of Computer Models (with Discussion).” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68: 425–464.
    Mathematical Reviews (MathSciNet): MR1858398
    Digital Object Identifier: doi:10.1111/1467-9868.00294
  • Lakshminarayanan, B., Roy, D. M., and Teh, Y. W. (2013). “Top-Down Particle Filtering for Bayesian Decision Trees.” In: Proceedings of the International Conference on Machine Learning. ICML.
  • Lakshminarayanan, B., Roy, D. M., and Teh, Y. W. (2015). “Particle Gibbs for Bayesian Additive Regression Trees.” arXiv:1502.04622v1.
  • Løland, A. and Høst, G. (2003). “Spatial Covariance Modeling in a Complex Coastal Domain by Multidimensional Scaling.” Environmetrics, 14: 307–321.
  • Oakley, J. and O’Hagan, A. (2002). “Bayesian Inference for the Uncertainty Distribution of Computer Model Outputs.” Biometrika, 89(4): 769–784.
  • Pratola, M. T. (2016). “Supplementary Material of “Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models”.” Bayesian Analysis.
  • Pratola, M., Chipman, H., Gattiker, J., Higdon, D., McCulloch, R., and Rust, W. (2014). “Parallel Bayesian Additive Regression Trees.” Journal of Computational and Graphical Statistics, 23: 830–852.
    Mathematical Reviews (MathSciNet): MR3224658
    Digital Object Identifier: doi:10.1080/10618600.2013.841584
  • Pratola, M. T., Harari, O., Bingham, D., and Flowers, G. E. (2015). “Design and Analysis of Experiments on Non-Convex Regions.” Technometrics.
  • Rathbun, S. L. (1998). “Spatial Modeling in Irregularly Shaped Regions: Kriging Estuaries.” Environmetrics, 9: 109–129.
  • Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989). “Design and Analysis of Computer Experiments (with Discussion).” Statistical Science, 4: 409–423.
    Mathematical Reviews (MathSciNet): MR1041765
    Digital Object Identifier: doi:10.1214/ss/1177012413
    Project Euclid: euclid.ss/1177012413
  • Sleator, D., Tarjan, R., and Thurston, W. (1988). “Rotation Distance, Triangulations, and Hyperbolic Geometry.” Journal of the American Mathematical Society, 3: 647–681.
    Mathematical Reviews (MathSciNet): MR928904
    Digital Object Identifier: doi:10.1090/S0894-0347-1988-0928904-4
  • Taddy, M., Gramacy, R., and Polson, N. (2011). “Dynamic Trees for Learning and Design.” Journal of the American Statistical Association, 106(493): 109–123.
    Mathematical Reviews (MathSciNet): MR2816706
    Digital Object Identifier: doi:10.1198/jasa.2011.ap09769
  • Wu, Y., Tjelmeland, H., and West, M. (2007). “Bayesian CART: Prior Specification and Posterior Simulation.” Journal of Computational and Graphical Statistics, 16(1): 44–66.
    Mathematical Reviews (MathSciNet): MR2345747
    Digital Object Identifier: doi:10.1198/106186007X180426

See also

  • Related item: Robert B. Gramacy (2016). Comment on Article by Pratola. Bayesian Anal. Vol. 11, Iss. 3, 913–919.
    Digital Object Identifier: doi:10.1214/16-BA999A
  • Related item: Christopher M. Hans (2016). Comment on Article by Pratola. Bayesian Anal. Vol. 11, Iss. 3, 921–927.
    Digital Object Identifier: doi:10.1214/16-BA999B
  • Related item: Oksana A. Chkrebtii, Scotland Leman, Andrew Hoegh, Reihaneh Entezari, Radu V. Craiu, Jeffrey S. Rosenthal, Abdolreza Mohammadi, Maurits Kaptein, Luca Martino, Rafael B. Stern, Francisco Louzada (2016). Contributed Discussion on Article by Pratola. Bayesian Anal. Vol. 11, Iss. 3, 929–943.
    Digital Object Identifier: doi:10.1214/16-BA999H
  • Related item: Matthew T. Pratola (2016). Rejoinder. Bayesian Anal. Vol. 11, Iss. 3, 945–955.
    Digital Object Identifier: doi:10.1214/16-BA999REJ

Supplemental materials

International Society for Bayesian Analysis

Bayesian Analysis

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more