The Annals of Statistics

Consistency of random forests

Erwan Scornet, Gérard Biau, and Jean-Philippe Vert

Full-text: Access has been disabled (more information)

Abstract

Random forests are a learning algorithm proposed by Breiman [ Mach. Learn. 45 (2001) 5–32] that combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly data-dependent tree structure. In the present paper, we take a step forward in forest exploration by proving a consistency result for Breiman’s [ Mach. Learn. 45 (2001) 5–32] original algorithm in the context of additive regression models. Our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity.

Article information

Source
Ann. Statist. Volume 43, Number 4 (2015), 1716-1741.

Dates
Received: May 2014
Revised: February 2015
First available in Project Euclid: 17 June 2015

Permanent link to this document
http://projecteuclid.org/euclid.aos/1434546220

Digital Object Identifier
doi:10.1214/15-AOS1321

Mathematical Reviews number (MathSciNet)
MR3357876

Zentralblatt MATH identifier
1317.62028

Subjects
Primary: 62G05: Estimation
Secondary: 62G20: Asymptotic properties

Keywords
Random forests randomization consistency additive model sparsity dimension reduction

Citation

Scornet, Erwan; Biau, Gérard; Vert, Jean-Philippe. Consistency of random forests. Ann. Statist. 43 (2015), no. 4, 1716--1741. doi:10.1214/15-AOS1321. http://projecteuclid.org/euclid.aos/1434546220.


Export citation

References

  • Amaratunga, D., Cabrera, J. and Lee, Y.-S. (2008). Enriched random forests. Bioinformatics 24 2010–2014.
  • Bai, Z.-D., Devroye, L., Hwang, H.-K. and Tsai, T.-H. (2005). Maxima in hypercubes. Random Structures Algorithms 27 290–309.
    Mathematical Reviews (MathSciNet): MR2162600
  • Barndorff-Nielsen, O. and Sobel, M. (1966). On the distribution of the number of admissible points in a vector random sample. Teor. Verojatnost. i Primenen. 11 283–305.
    Mathematical Reviews (MathSciNet): MR207003
  • Biau, G. (2012). Analysis of a random forests model. J. Mach. Learn. Res. 13 1063–1095.
    Mathematical Reviews (MathSciNet): MR2930634
    Zentralblatt MATH: 1283.62127
  • Biau, G. and Devroye, L. (2010). On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification. J. Multivariate Anal. 101 2499–2518.
    Mathematical Reviews (MathSciNet): MR2719877
    Zentralblatt MATH: 1198.62048
    Digital Object Identifier: doi:10.1016/j.jmva.2010.06.019
  • Biau, G., Devroye, L. and Lugosi, G. (2008). Consistency of random forests and other averaging classifiers. J. Mach. Learn. Res. 9 2015–2033.
    Mathematical Reviews (MathSciNet): MR2447310
    Zentralblatt MATH: 1225.62081
  • Boucheron, S., Lugosi, G. and Massart, P. (2013). Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford Univ. Press, Oxford.
    Mathematical Reviews (MathSciNet): MR3185193
  • Breiman, L. (1996). Bagging predictors. Mach. Learn. 24 123–140.
    Mathematical Reviews (MathSciNet): MR1425957
    Zentralblatt MATH: 0867.62055
    Digital Object Identifier: doi:10.1214/aos/1032181158
    Project Euclid: euclid.aos/1032181158
  • Breiman, L. (2001). Random forests. Mach. Learn. 45 5–32.
  • Breiman, L. (2004). Consistency for a simple model of random forests. Technical Report 670, Univ. California, Berkeley, CA.
  • Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth Advanced Books and Software, Belmont, CA.
    Mathematical Reviews (MathSciNet): MR726392
    Zentralblatt MATH: 0541.62042
  • Bühlmann, P. and Yu, B. (2002). Analyzing bagging. Ann. Statist. 30 927–961.
    Mathematical Reviews (MathSciNet): MR1926165
    Zentralblatt MATH: 1029.62037
    Digital Object Identifier: doi:10.1214/aos/1031689014
    Project Euclid: euclid.aos/1031689014
  • Clémençon, S., Depecker, M. and Vayatis, N. (2013). Ranking forests. J. Mach. Learn. Res. 14 39–73.
    Mathematical Reviews (MathSciNet): MR3033325
    Zentralblatt MATH: 1307.68065
  • Cutler, D. R., Edwards, T. C. Jr, Beard, K. H., Cutler, A., Hess, K. T., Gibson, J. and Lawler, J. J. (2007). Random forests for classification in ecology. Ecology 88 2783–2792.
  • Denil, M., Matheson, D. and Freitas, N. d. (2013). Consistency of online random forests. In Proceedings of the ICML Conference. Available at arXiv:1302.4853.
    arXiv: arXiv:1302.4853
  • Devroye, L., Györfi, L. and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Applications of Mathematics (New York) 31. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1383093
    Zentralblatt MATH: 0853.68150
  • Díaz-Uriarte, R. and Alvarez de Andrés, S. (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7 1–13.
  • Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. CBMS-NSF Regional Conference Series in Applied Mathematics 38. SIAM, Philadelphia.
    Mathematical Reviews (MathSciNet): MR659849
    Zentralblatt MATH: 0496.62036
  • Genuer, R. (2012). Variance reduction in purely random forests. J. Nonparametr. Stat. 24 543–562.
    Mathematical Reviews (MathSciNet): MR2968888
    Zentralblatt MATH: 1254.62050
    Digital Object Identifier: doi:10.1080/10485252.2012.677843
  • Geurts, P., Ernst, D. and Wehenkel, L. (2006). Extremely randomized trees. Mach. Learn. 63 3–42.
  • Györfi, L., Kohler, M., Krzyżak, A. and Walk, H. (2002). A Distribution-Free Theory of Nonparametric Regression. Springer, New York.
  • Hastie, T. and Tibshirani, R. (1986). Generalized additive models. Statist. Sci. 1 297–318.
    Mathematical Reviews (MathSciNet): MR858512
    Digital Object Identifier: doi:10.1214/ss/1177013604
    Project Euclid: euclid.ss/1177013604
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.
    Mathematical Reviews (MathSciNet): MR2722294
  • Ishwaran, H. and Kogalur, U. B. (2010). Consistency of random survival forests. Statist. Probab. Lett. 80 1056–1064.
    Mathematical Reviews (MathSciNet): MR2651045
  • Ishwaran, H., Kogalur, U. B., Blackstone, E. H. and Lauer, M. S. (2008). Random survival forests. Ann. Appl. Stat. 2 841–860.
    Mathematical Reviews (MathSciNet): MR2516796
    Zentralblatt MATH: 1149.62331
    Digital Object Identifier: doi:10.1214/08-AOAS169
    Project Euclid: euclid.aoas/1223908043
  • Kleiner, A., Talwalkar, A., Sarkar, P. and Jordan, M. I. (2014). A scalable bootstrap for massive data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 795–816.
    Mathematical Reviews (MathSciNet): MR3248677
    Digital Object Identifier: doi:10.1111/rssb.12050
  • Lin, Y. and Jeon, Y. (2006). Random forests and adaptive nearest neighbors. J. Amer. Statist. Assoc. 101 578–590.
    Mathematical Reviews (MathSciNet): MR2256176
    Zentralblatt MATH: 1119.62304
    Digital Object Identifier: doi:10.1198/016214505000001230
  • Meinshausen, N. (2006). Quantile regression forests. J. Mach. Learn. Res. 7 983–999.
    Mathematical Reviews (MathSciNet): MR2274394
    Zentralblatt MATH: 1222.68262
  • Mentch, L. and Hooker, G. (2014). Ensemble trees and clts: Statistical inference for supervised learning. Available at arXiv:1404.6473.
    arXiv: arXiv:1404.6473
  • Nobel, A. (1996). Histogram regression estimation using data-dependent partitions. Ann. Statist. 24 1084–1105.
    Mathematical Reviews (MathSciNet): MR1401839
    Zentralblatt MATH: 0862.62038
    Digital Object Identifier: doi:10.1214/aos/1032526958
    Project Euclid: euclid.aos/1032526958
  • Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1707286
  • Prasad, A. M., Iverson, L. R. and Liaw, A. (2006). Newer classification and regression tree techniques: Bagging and random forests for ecological prediction. Ecosystems 9 181–199.
  • Scornet, E. (2014). On the asymptotics of random forests. Available at arXiv:1409.2090.
    arXiv: arXiv:1409.2090
  • Scornet, E., Biau, G. and Vert, J. (2015). Supplement to “Consistency of random forests.” DOI:10.1214/15-AOS1321SUPP.
  • Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M. and Moore, R. (2013). Real-time human pose recognition in parts from single depth images. Comm. ACM 56 116–124.
  • Stone, C. J. (1977). Consistent nonparametric regression. Ann. Statist. 5 595–645.
    Mathematical Reviews (MathSciNet): MR443204
    Zentralblatt MATH: 0366.62051
    Digital Object Identifier: doi:10.1214/aos/1176343886
    Project Euclid: euclid.aos/1176343886
  • Stone, C. J. (1985). Additive regression and other nonparametric models. Ann. Statist. 13 689–705.
    Mathematical Reviews (MathSciNet): MR790566
    Zentralblatt MATH: 0605.62065
    Digital Object Identifier: doi:10.1214/aos/1176349548
    Project Euclid: euclid.aos/1176349548
  • Svetnik, V., Liaw, A., Tong, C., Culberson, J. C., Sheridan, R. P. and Feuston, B. P. (2003). Random forest: A classification and regression tool for compound classification and QSAR modeling. J. Chem. Inf. Comput. Sci. 43 1947–1958.
  • Wager, S. (2014). Asymptotic theory for random forests. Available at arXiv:1405.0352.
    arXiv: arXiv:1405.0352
  • Wager, S., Hastie, T. and Efron, B. (2014). Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. J. Mach. Learn. Res. 15 1625–1651.
    Mathematical Reviews (MathSciNet): MR3225243
    Zentralblatt MATH: 06378123
  • Zhu, R., Zeng, D. and Kosorok, M. R. (2012). Reinforcement learning trees. Technical report, Univ. North Carolina.

Supplemental materials

  • Supplement to “Consistency of random forests”. Proofs of technical results.
    Digital Object Identifier: doi:10.1214/15-AOS1321SUPP
    Supplemental files available for subscribers.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more