Annals of Statistics

Consistency of random forests

Erwan Scornet, Gérard Biau, and Jean-Philippe Vert

Full-text: Open access


Random forests are a learning algorithm proposed by Breiman [ Mach. Learn. 45 (2001) 5–32] that combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly data-dependent tree structure. In the present paper, we take a step forward in forest exploration by proving a consistency result for Breiman’s [ Mach. Learn. 45 (2001) 5–32] original algorithm in the context of additive regression models. Our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity.

Article information

Ann. Statist., Volume 43, Number 4 (2015), 1716-1741.

Received: May 2014
Revised: February 2015
First available in Project Euclid: 17 June 2015

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G05: Estimation
Secondary: 62G20: Asymptotic properties

Random forests randomization consistency additive model sparsity dimension reduction


Scornet, Erwan; Biau, Gérard; Vert, Jean-Philippe. Consistency of random forests. Ann. Statist. 43 (2015), no. 4, 1716--1741. doi:10.1214/15-AOS1321.

Export citation


  • Amaratunga, D., Cabrera, J. and Lee, Y.-S. (2008). Enriched random forests. Bioinformatics 24 2010–2014.
  • Bai, Z.-D., Devroye, L., Hwang, H.-K. and Tsai, T.-H. (2005). Maxima in hypercubes. Random Structures Algorithms 27 290–309.
  • Barndorff-Nielsen, O. and Sobel, M. (1966). On the distribution of the number of admissible points in a vector random sample. Teor. Verojatnost. i Primenen. 11 283–305.
  • Biau, G. (2012). Analysis of a random forests model. J. Mach. Learn. Res. 13 1063–1095.
  • Biau, G. and Devroye, L. (2010). On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification. J. Multivariate Anal. 101 2499–2518.
  • Biau, G., Devroye, L. and Lugosi, G. (2008). Consistency of random forests and other averaging classifiers. J. Mach. Learn. Res. 9 2015–2033.
  • Boucheron, S., Lugosi, G. and Massart, P. (2013). Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford Univ. Press, Oxford.
  • Breiman, L. (1996). Bagging predictors. Mach. Learn. 24 123–140.
  • Breiman, L. (2001). Random forests. Mach. Learn. 45 5–32.
  • Breiman, L. (2004). Consistency for a simple model of random forests. Technical Report 670, Univ. California, Berkeley, CA.
  • Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth Advanced Books and Software, Belmont, CA.
  • Bühlmann, P. and Yu, B. (2002). Analyzing bagging. Ann. Statist. 30 927–961.
  • Clémençon, S., Depecker, M. and Vayatis, N. (2013). Ranking forests. J. Mach. Learn. Res. 14 39–73.
  • Cutler, D. R., Edwards, T. C. Jr, Beard, K. H., Cutler, A., Hess, K. T., Gibson, J. and Lawler, J. J. (2007). Random forests for classification in ecology. Ecology 88 2783–2792.
  • Denil, M., Matheson, D. and Freitas, N. d. (2013). Consistency of online random forests. In Proceedings of the ICML Conference. Available at arXiv:1302.4853.
  • Devroye, L., Györfi, L. and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Applications of Mathematics (New York) 31. Springer, New York.
  • Díaz-Uriarte, R. and Alvarez de Andrés, S. (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7 1–13.
  • Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. CBMS-NSF Regional Conference Series in Applied Mathematics 38. SIAM, Philadelphia.
  • Genuer, R. (2012). Variance reduction in purely random forests. J. Nonparametr. Stat. 24 543–562.
  • Geurts, P., Ernst, D. and Wehenkel, L. (2006). Extremely randomized trees. Mach. Learn. 63 3–42.
  • Györfi, L., Kohler, M., Krzyżak, A. and Walk, H. (2002). A Distribution-Free Theory of Nonparametric Regression. Springer, New York.
  • Hastie, T. and Tibshirani, R. (1986). Generalized additive models. Statist. Sci. 1 297–318.
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.
  • Ishwaran, H. and Kogalur, U. B. (2010). Consistency of random survival forests. Statist. Probab. Lett. 80 1056–1064.
  • Ishwaran, H., Kogalur, U. B., Blackstone, E. H. and Lauer, M. S. (2008). Random survival forests. Ann. Appl. Stat. 2 841–860.
  • Kleiner, A., Talwalkar, A., Sarkar, P. and Jordan, M. I. (2014). A scalable bootstrap for massive data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 795–816.
  • Lin, Y. and Jeon, Y. (2006). Random forests and adaptive nearest neighbors. J. Amer. Statist. Assoc. 101 578–590.
  • Meinshausen, N. (2006). Quantile regression forests. J. Mach. Learn. Res. 7 983–999.
  • Mentch, L. and Hooker, G. (2014). Ensemble trees and clts: Statistical inference for supervised learning. Available at arXiv:1404.6473.
  • Nobel, A. (1996). Histogram regression estimation using data-dependent partitions. Ann. Statist. 24 1084–1105.
  • Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer, New York.
  • Prasad, A. M., Iverson, L. R. and Liaw, A. (2006). Newer classification and regression tree techniques: Bagging and random forests for ecological prediction. Ecosystems 9 181–199.
  • Scornet, E. (2014). On the asymptotics of random forests. Available at arXiv:1409.2090.
  • Scornet, E., Biau, G. and Vert, J. (2015). Supplement to “Consistency of random forests.” DOI:10.1214/15-AOS1321SUPP.
  • Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M. and Moore, R. (2013). Real-time human pose recognition in parts from single depth images. Comm. ACM 56 116–124.
  • Stone, C. J. (1977). Consistent nonparametric regression. Ann. Statist. 5 595–645.
  • Stone, C. J. (1985). Additive regression and other nonparametric models. Ann. Statist. 13 689–705.
  • Svetnik, V., Liaw, A., Tong, C., Culberson, J. C., Sheridan, R. P. and Feuston, B. P. (2003). Random forest: A classification and regression tool for compound classification and QSAR modeling. J. Chem. Inf. Comput. Sci. 43 1947–1958.
  • Wager, S. (2014). Asymptotic theory for random forests. Available at arXiv:1405.0352.
  • Wager, S., Hastie, T. and Efron, B. (2014). Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. J. Mach. Learn. Res. 15 1625–1651.
  • Zhu, R., Zeng, D. and Kosorok, M. R. (2012). Reinforcement learning trees. Technical report, Univ. North Carolina.

Supplemental materials