Open Access
August 2020 Posterior concentration for Bayesian regression trees and forests
Veronika Ročková, Stéphanie van der Pas
Ann. Statist. 48(4): 2108-2131 (August 2020). DOI: 10.1214/19-AOS1879


Since their inception in the 1980s, regression trees have been one of the more widely used nonparametric prediction methods. Tree-structured methods yield a histogram reconstruction of the regression surface, where the bins correspond to terminal nodes of recursive partitioning. Trees are powerful, yet susceptible to overfitting. Strategies against overfitting have traditionally relied on pruning greedily grown trees. The Bayesian framework offers an alternative remedy against overfitting through priors. Roughly speaking, a good prior charges smaller trees where overfitting does not occur. While the consistency of random histograms, trees and their ensembles has been studied quite extensively, the theoretical understanding of the Bayesian counterparts has been missing. In this paper, we take a step toward understanding why/when do Bayesian trees and forests not overfit. To address this question, we study the speed at which the posterior concentrates around the true smooth regression function. We propose a spike-and-tree variant of the popular Bayesian CART prior and establish new theoretical results showing that regression trees (and forests) (a) are capable of recovering smooth regression surfaces (with smoothness not exceeding one), achieving optimal rates up to a log factor, (b) can adapt to the unknown level of smoothness and (c) can perform effective dimension reduction when $p>n$. These results provide a piece of missing theoretical evidence explaining why Bayesian trees (and additive variants thereof?) have worked so well in practice.


Download Citation

Veronika Ročková. Stéphanie van der Pas. "Posterior concentration for Bayesian regression trees and forests." Ann. Statist. 48 (4) 2108 - 2131, August 2020.


Received: 1 September 2017; Revised: 1 May 2019; Published: August 2020
First available in Project Euclid: 14 August 2020

MathSciNet: MR4134788
Digital Object Identifier: 10.1214/19-AOS1879

Primary: 62G08 , 62G20

Keywords: additive regression , asymptotic minimaxity , BART , Bayesian CART , posterior concentration , recursive partitioning , regression trees

Rights: Copyright © 2020 Institute of Mathematical Statistics

Vol.48 • No. 4 • August 2020
Back to Top