Open Access
April 2019 Estimating the algorithmic variance of randomized ensembles via the bootstrap
Miles E. Lopes
Ann. Statist. 47(2): 1088-1112 (April 2019). DOI: 10.1214/18-AOS1707


Although the methods of bagging and random forests are some of the most widely used prediction methods, relatively little is known about their algorithmic convergence. In particular, there are not many theoretical guarantees for deciding when an ensemble is “large enough”—so that its accuracy is close to that of an ideal infinite ensemble. Due to the fact that bagging and random forests are randomized algorithms, the choice of ensemble size is closely related to the notion of “algorithmic variance” (i.e., the variance of prediction error due only to the training algorithm). In the present work, we propose a bootstrap method to estimate this variance for bagging, random forests and related methods in the context of classification. To be specific, suppose the training dataset is fixed, and let the random variable $\mathrm{ERR}_{t}$ denote the prediction error of a randomized ensemble of size $t$. Working under a “first-order model” for randomized ensembles, we prove that the centered law of $\mathrm{ERR}_{t}$ can be consistently approximated via the proposed method as $t\to\infty$. Meanwhile, the computational cost of the method is quite modest, by virtue of an extrapolation technique. As a consequence, the method offers a practical guideline for deciding when the algorithmic fluctuations of $\mathrm{ERR}_{t}$ are negligible.


Download Citation

Miles E. Lopes. "Estimating the algorithmic variance of randomized ensembles via the bootstrap." Ann. Statist. 47 (2) 1088 - 1112, April 2019.


Received: 1 October 2017; Revised: 1 February 2018; Published: April 2019
First available in Project Euclid: 11 January 2019

zbMATH: 07033162
MathSciNet: MR3909961
Digital Object Identifier: 10.1214/18-AOS1707

Primary: 62F40
Secondary: 60G25 , 65B05 , 68W20

Keywords: bagging , bootstrap , random forests , randomized algorithms

Rights: Copyright © 2019 Institute of Mathematical Statistics

Vol.47 • No. 2 • April 2019
Back to Top