The Annals of Statistics

Classification algorithms using adaptive partitioning

Peter Binev, Albert Cohen, Wolfgang Dahmen, and Ronald DeVore

Full-text: Access has been disabled (more information)

Abstract

Algorithms for binary classification based on adaptive tree partitioning are formulated and analyzed for both their risk performance and their friendliness to numerical implementation. The algorithms can be viewed as generating a set approximation to the Bayes set and thus fall into the general category of set estimators. In contrast with the most studied tree-based algorithms, which utilize piecewise constant approximation on the generated partition [IEEE Trans. Inform. Theory 52 (2006) 1335–1353; Mach. Learn. 66 (2007) 209–242], we consider decorated trees, which allow us to derive higher order methods. Convergence rates for these methods are derived in terms the parameter $\alpha$ of margin conditions and a rate $s$ of best approximation of the Bayes set by decorated adaptive partitions. They can also be expressed in terms of the Besov smoothness $\beta$ of the regression function that governs its approximability by piecewise polynomials on adaptive partition. The execution of the algorithms does not require knowledge of the smoothness or margin conditions. Besov smoothness conditions are weaker than the commonly used Hölder conditions, which govern approximation by nonadaptive partitions, and therefore for a given regression function can result in a higher rate of convergence. This in turn mitigates the compatibility conflict between smoothness and margin parameters.

Article information

Source
Ann. Statist. Volume 42, Number 6 (2014), 2141-2163.

Dates
First available in Project Euclid: 20 October 2014

Permanent link to this document
http://projecteuclid.org/euclid.aos/1413810724

Digital Object Identifier
doi:10.1214/14-AOS1234

Mathematical Reviews number (MathSciNet)
MR3269976

Zentralblatt MATH identifier
1310.62074

Subjects
Primary: 62M45: Neural nets and related approaches 65D05: Interpolation 68Q32: Computational learning theory [See also 68T05] 97N50: Interpolation and approximation

Keywords
Binary classification adaptive methods set estimators tree-based algorithms

Citation

Binev, Peter; Cohen, Albert; Dahmen, Wolfgang; DeVore, Ronald. Classification algorithms using adaptive partitioning. Ann. Statist. 42 (2014), no. 6, 2141--2163. doi:10.1214/14-AOS1234. http://projecteuclid.org/euclid.aos/1413810724.


Export citation

References

  • [1] Akakpo, N. (2012). Adaptation to anisotropy and inhomogeneity via dyadic piecewise polynomial selection. Math. Methods Statist. 21 1–28.
    Mathematical Reviews (MathSciNet): MR2901269
    Digital Object Identifier: doi:10.3103/S1066530712010012
  • [2] Audibert, J.-Y. and Tsybakov, A. B. (2007). Fast learning rates for plug-in classifiers. Ann. Statist. 35 608–633.
    Mathematical Reviews (MathSciNet): MR2336861
    Digital Object Identifier: doi:10.1214/009053606000001217
    Project Euclid: euclid.aos/1183667286
  • [3] Binev, P., Cohen, A., Dahmen, W. and DeVore, R. (2007). Universal algorithms for learning theory. II. Piecewise polynomial functions. Constr. Approx. 26 127–152.
    Mathematical Reviews (MathSciNet): MR2327596
    Digital Object Identifier: doi:10.1007/s00365-006-0658-z
  • [4] Binev, P., Cohen, A., Dahmen, W. and DeVore, R. (2014). Supplement to “Classification algorithms using adaptive partitioning.” DOI:10.1214/14-AOS1234SUPP.
    Mathematical Reviews (MathSciNet): MR3269976
    Digital Object Identifier: doi:10.1214/14-AOS1234
    Project Euclid: euclid.aos/1413810724
  • [5] Blanchard, G. and Massart, P. (2006). Discussion: “Local Rademacher complexities and oracle inequalities in risk minimization” [Ann. Statist. 34 (2006), no. 6, 2593–2656; 2329442] by V. Koltchinskii. Ann. Statist. 34 2664–2671.
    Mathematical Reviews (MathSciNet): MR2329460
    Digital Object Identifier: doi:10.1214/009053606000001037
    Project Euclid: euclid.aos/1179935057
  • [6] Blanchard, G., Schäfer, C., Rozenholc, Y. and Müller, K.-R. (2007). Optimal dyadic decision trees. Mach. Learn. 66 209–242.
  • [7] Boucheron, S., Bousquet, O. and Lugosi, G. (2005). Theory of classification: A survey of some recent advances. ESAIM Probab. Stat. 9 323–375.
    Mathematical Reviews (MathSciNet): MR2182250
    Digital Object Identifier: doi:10.1051/ps:2005018
  • [8] Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth, Belmont, CA.
    Mathematical Reviews (MathSciNet): MR726392
  • [9] Cohen, A., Dahmen, W., Daubechies, I. and DeVore, R. (2001). Tree approximation and optimal encoding. Appl. Comput. Harmon. Anal. 11 192–226.
    Mathematical Reviews (MathSciNet): MR1848303
    Digital Object Identifier: doi:10.1006/acha.2001.0336
  • [10] DeVore, R. A. (1998). Nonlinear approximation. In Acta Numerica, 1998. Acta Numer. 7 51–150. Cambridge Univ. Press, Cambridge.
    Mathematical Reviews (MathSciNet): MR1689432
  • [11] Devroye, L., Györfi, L. and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Applications of Mathematics (New York) 31. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1383093
  • [12] Györfy, L., Kohler, M., Krzyzak, A. and Walk, H. (2002). A Distribution-Free Theory of Nonparametric Regression. Springer, Berlin.
  • [13] Massart, P. and Nédélec, É. (2006). Risk bounds for statistical learning. Ann. Statist. 34 2326–2366.
    Mathematical Reviews (MathSciNet): MR2291502
    Digital Object Identifier: doi:10.1214/009053606000000786
    Project Euclid: euclid.aos/1169571799
  • [14] Scott, C. and Nowak, R. D. (2006). Minimax-optimal classification with dyadic decision trees. IEEE Trans. Inform. Theory 52 1335–1353.
    Mathematical Reviews (MathSciNet): MR2241192
    Digital Object Identifier: doi:10.1109/TIT.2006.871056
  • [15] Tsybakov, A. B. (2004). Optimal aggregation of classifiers in statistical learning. Ann. Statist. 32 135–166.
    Mathematical Reviews (MathSciNet): MR2051002
    Digital Object Identifier: doi:10.1214/aos/1079120131
    Project Euclid: euclid.aos/1079120131

Supplemental materials

  • Supplementary material: Proof of Theorem 2.1. This supplement contains the detailed proof of Theorem 2.1 [4].
    Digital Object Identifier: doi:10.1214/14-AOS1234SUPP
    Supplemental files available for subscribers.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more