The Annals of Statistics

Consistency of random forests

Erwan Scornet, Gérard Biau, and Jean-Philippe Vert

Full-text: Access denied (no subscription detected) We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

Random forests are a learning algorithm proposed by Breiman [ Mach. Learn. 45 (2001) 5–32] that combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly data-dependent tree structure. In the present paper, we take a step forward in forest exploration by proving a consistency result for Breiman’s [ Mach. Learn. 45 (2001) 5–32] original algorithm in the context of additive regression models. Our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity.

Article information

Source
Ann. Statist. Volume 43, Number 4 (2015), 1716-1741.

Dates
Received: May 2014
Revised: February 2015
First available in Project Euclid: 17 June 2015

Permanent link to this document
https://projecteuclid.org/euclid.aos/1434546220

Digital Object Identifier
doi:10.1214/15-AOS1321

Mathematical Reviews number (MathSciNet)
MR3357876

Zentralblatt MATH identifier
1317.62028

Subjects
Primary: 62G05: Estimation
Secondary: 62G20: Asymptotic properties

Keywords
Random forests randomization consistency additive model sparsity dimension reduction

Citation

Scornet, Erwan; Biau, Gérard; Vert, Jean-Philippe. Consistency of random forests. Ann. Statist. 43 (2015), no. 4, 1716--1741. doi:10.1214/15-AOS1321. https://projecteuclid.org/euclid.aos/1434546220.


Export citation

References

  • Amaratunga, D., Cabrera, J. and Lee, Y.-S. (2008). Enriched random forests. Bioinformatics 24 2010–2014.
  • Bai, Z.-D., Devroye, L., Hwang, H.-K. and Tsai, T.-H. (2005). Maxima in hypercubes. Random Structures Algorithms 27 290–309.
  • Barndorff-Nielsen, O. and Sobel, M. (1966). On the distribution of the number of admissible points in a vector random sample. Teor. Verojatnost. i Primenen. 11 283–305.
  • Biau, G. (2012). Analysis of a random forests model. J. Mach. Learn. Res. 13 1063–1095.
  • Biau, G. and Devroye, L. (2010). On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification. J. Multivariate Anal. 101 2499–2518.
  • Biau, G., Devroye, L. and Lugosi, G. (2008). Consistency of random forests and other averaging classifiers. J. Mach. Learn. Res. 9 2015–2033.
  • Boucheron, S., Lugosi, G. and Massart, P. (2013). Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford Univ. Press, Oxford.
  • Breiman, L. (1996). Bagging predictors. Mach. Learn. 24 123–140.
  • Breiman, L. (2001). Random forests. Mach. Learn. 45 5–32.
  • Breiman, L. (2004). Consistency for a simple model of random forests. Technical Report 670, Univ. California, Berkeley, CA.
  • Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth Advanced Books and Software, Belmont, CA.
  • Bühlmann, P. and Yu, B. (2002). Analyzing bagging. Ann. Statist. 30 927–961.
  • Clémençon, S., Depecker, M. and Vayatis, N. (2013). Ranking forests. J. Mach. Learn. Res. 14 39–73.
  • Cutler, D. R., Edwards, T. C. Jr, Beard, K. H., Cutler, A., Hess, K. T., Gibson, J. and Lawler, J. J. (2007). Random forests for classification in ecology. Ecology 88 2783–2792.
  • Denil, M., Matheson, D. and Freitas, N. d. (2013). Consistency of online random forests. In Proceedings of the ICML Conference. Available at arXiv:1302.4853.
  • Devroye, L., Györfi, L. and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Applications of Mathematics (New York) 31. Springer, New York.
  • Díaz-Uriarte, R. and Alvarez de Andrés, S. (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7 1–13.
  • Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. CBMS-NSF Regional Conference Series in Applied Mathematics 38. SIAM, Philadelphia.
  • Genuer, R. (2012). Variance reduction in purely random forests. J. Nonparametr. Stat. 24 543–562.
  • Geurts, P., Ernst, D. and Wehenkel, L. (2006). Extremely randomized trees. Mach. Learn. 63 3–42.
  • Györfi, L., Kohler, M., Krzyżak, A. and Walk, H. (2002). A Distribution-Free Theory of Nonparametric Regression. Springer, New York.
  • Hastie, T. and Tibshirani, R. (1986). Generalized additive models. Statist. Sci. 1 297–318.
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.
  • Ishwaran, H. and Kogalur, U. B. (2010). Consistency of random survival forests. Statist. Probab. Lett. 80 1056–1064.
  • Ishwaran, H., Kogalur, U. B., Blackstone, E. H. and Lauer, M. S. (2008). Random survival forests. Ann. Appl. Stat. 2 841–860.
  • Kleiner, A., Talwalkar, A., Sarkar, P. and Jordan, M. I. (2014). A scalable bootstrap for massive data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 795–816.
  • Lin, Y. and Jeon, Y. (2006). Random forests and adaptive nearest neighbors. J. Amer. Statist. Assoc. 101 578–590.
  • Meinshausen, N. (2006). Quantile regression forests. J. Mach. Learn. Res. 7 983–999.
  • Mentch, L. and Hooker, G. (2014). Ensemble trees and clts: Statistical inference for supervised learning. Available at arXiv:1404.6473.
  • Nobel, A. (1996). Histogram regression estimation using data-dependent partitions. Ann. Statist. 24 1084–1105.
  • Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer, New York.
  • Prasad, A. M., Iverson, L. R. and Liaw, A. (2006). Newer classification and regression tree techniques: Bagging and random forests for ecological prediction. Ecosystems 9 181–199.
  • Scornet, E. (2014). On the asymptotics of random forests. Available at arXiv:1409.2090.
  • Scornet, E., Biau, G. and Vert, J. (2015). Supplement to “Consistency of random forests.” DOI:10.1214/15-AOS1321SUPP.
  • Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M. and Moore, R. (2013). Real-time human pose recognition in parts from single depth images. Comm. ACM 56 116–124.
  • Stone, C. J. (1977). Consistent nonparametric regression. Ann. Statist. 5 595–645.
  • Stone, C. J. (1985). Additive regression and other nonparametric models. Ann. Statist. 13 689–705.
  • Svetnik, V., Liaw, A., Tong, C., Culberson, J. C., Sheridan, R. P. and Feuston, B. P. (2003). Random forest: A classification and regression tool for compound classification and QSAR modeling. J. Chem. Inf. Comput. Sci. 43 1947–1958.
  • Wager, S. (2014). Asymptotic theory for random forests. Available at arXiv:1405.0352.
  • Wager, S., Hastie, T. and Efron, B. (2014). Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. J. Mach. Learn. Res. 15 1625–1651.
  • Zhu, R., Zeng, D. and Kosorok, M. R. (2012). Reinforcement learning trees. Technical report, Univ. North Carolina.

Supplemental materials

  • Supplement to “Consistency of random forests”. Proofs of technical results.