A great many tools have been developed for supervised classification, ranging from early methods such as linear discriminant analysis through to modern developments such as neural networks and support vector machines. A large number of comparative studies have been conducted in attempts to establish the relative superiority of these methods. This paper argues that these comparisons often fail to take into account important aspects of real problems, so that the apparent superiority of more sophisticated methods may be something of an illusion. In particular, simple methods typically yield performance almost as good as more sophisticated methods, to the extent that the difference in performance may be swamped by other sources of uncertainty that generally are not considered in the classical supervised classification paradigm.
References
Adams, N. M. and Hand, D. J. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition 32 1139--1147. 1059.62065 Adams, N. M. and Hand, D. J. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition 32 1139--1147. 1059.62065
Breiman, L. (2001). Statistical modeling: The two cultures (with discussion). Statist. Sci. 16 199--231. MR1874152 10.1214/ss/1009213726 euclid.ss/1009213726
1059.62505 Breiman, L. (2001). Statistical modeling: The two cultures (with discussion). Statist. Sci. 16 199--231. MR1874152 10.1214/ss/1009213726 euclid.ss/1009213726
1059.62505
Efron, B. (2001). Comment on ``Statistical modeling: The two cultures,'' by L. Breiman. Statist. Sci. 16 218--219. MR1861072 10.1214/ss/1009213290 euclid.ss/1009213290
1059.01542 Efron, B. (2001). Comment on ``Statistical modeling: The two cultures,'' by L. Breiman. Statist. Sci. 16 218--219. MR1861072 10.1214/ss/1009213290 euclid.ss/1009213290
1059.01542
Hand, D. J. (1981). Discrimination and Classification. Wiley, Chichester. MR634676 0587.62119 Hand, D. J. (1981). Discrimination and Classification. Wiley, Chichester. MR634676 0587.62119
Hand, D. J. (1996). Classification and computers: Shifting the focus. In COMPSTAT-96: Proceedings in Computational Statistics (A. Prat, ed.) 77--88. Physica, Berlin. MR1602244 Hand, D. J. (1996). Classification and computers: Shifting the focus. In COMPSTAT-96: Proceedings in Computational Statistics (A. Prat, ed.) 77--88. Physica, Berlin. MR1602244
Hand, D. J. (1997). Construction and Assessment of Classification Rules. Wiley, Chichester. 0997.62500 Hand, D. J. (1997). Construction and Assessment of Classification Rules. Wiley, Chichester. 0997.62500
Hand, D. J. (1999). Intelligent data analysis and deep understanding. In Causal Models and Intelligent Data Management (A. Gammerman, ed.) 67--80. Springer, Berlin. MR1722705 Hand, D. J. (1999). Intelligent data analysis and deep understanding. In Causal Models and Intelligent Data Management (A. Gammerman, ed.) 67--80. Springer, Berlin. MR1722705
Hand, D. J. (2004). Academic obsessions and classification realities: Ignoring practicalities in supervised classification. In Classification, Clustering and Data Mining Applications (D. Banks, L. House, F. R. McMorris, P. Arabie and W. Gaul, eds.) 209--232. Springer, Berlin. MR2113611 Hand, D. J. (2004). Academic obsessions and classification realities: Ignoring practicalities in supervised classification. In Classification, Clustering and Data Mining Applications (D. Banks, L. House, F. R. McMorris, P. Arabie and W. Gaul, eds.) 209--232. Springer, Berlin. MR2113611
Hand, D. J. (2005). Supervised classification and tunnel vision. Applied Stochastic Models in Business and Industry 21 97--109. MR2137544 10.1002/asmb.540 1089.62077 Hand, D. J. (2005). Supervised classification and tunnel vision. Applied Stochastic Models in Business and Industry 21 97--109. MR2137544 10.1002/asmb.540 1089.62077
Hand, D. J. and Henley, W. E. (1997). Statistical classification methods in consumer credit scoring: A review. J. Roy. Statist. Soc. Ser. A 160 523--541. Hand, D. J. and Henley, W. E. (1997). Statistical classification methods in consumer credit scoring: A review. J. Roy. Statist. Soc. Ser. A 160 523--541.
Hoadley, B. (2001). Comment on ``Statistical modeling: The two cultures,'' by L. Breiman. Statist. Sci. 16 220--224. MR1874152 10.1214/ss/1009213726 euclid.ss/1009213726
1059.62505 Hoadley, B. (2001). Comment on ``Statistical modeling: The two cultures,'' by L. Breiman. Statist. Sci. 16 220--224. MR1874152 10.1214/ss/1009213726 euclid.ss/1009213726
1059.62505
Jamain, A. and Hand, D. J. (2005). Mining supervised classification performance studies: A meta-analytic investigation. Technical report, Dept. Mathematics, Imperial College London. Jamain, A. and Hand, D. J. (2005). Mining supervised classification performance studies: A meta-analytic investigation. Technical report, Dept. Mathematics, Imperial College London.
Kelly, M. G., Hand, D. J. and Adams, N. M. (1998). Defining the goals to optimise data mining performance. In Proc. Fourth International Conference on Knowledge Discovery and Data Mining (R. Agrawal, P. Stolorz and G. Piatetsky-Shapiro, eds.) 234--238. AAAI Press, Menlo Park, CA. Kelly, M. G., Hand, D. J. and Adams, N. M. (1998). Defining the goals to optimise data mining performance. In Proc. Fourth International Conference on Knowledge Discovery and Data Mining (R. Agrawal, P. Stolorz and G. Piatetsky-Shapiro, eds.) 234--238. AAAI Press, Menlo Park, CA.
Kelly, M. G., Hand, D. J. and Adams, N. M. (1999). The impact of changing populations on classifier performance. In Proc. Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (S. Chaudhuri and D. Madigan, eds.) 367--371. ACM, New York. Kelly, M. G., Hand, D. J. and Adams, N. M. (1999). The impact of changing populations on classifier performance. In Proc. Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (S. Chaudhuri and D. Madigan, eds.) 367--371. ACM, New York.
Kelly, M. G., Hand, D. J. and Adams, N. M. (1999). Supervised classification problems: How to be both judge and jury. In Advances in Intelligent Data Analysis. Lecture Notes in Comput. Sci. 1642 235--244. Springer, Berlin. MR1723394 Kelly, M. G., Hand, D. J. and Adams, N. M. (1999). Supervised classification problems: How to be both judge and jury. In Advances in Intelligent Data Analysis. Lecture Notes in Comput. Sci. 1642 235--244. Springer, Berlin. MR1723394
Klinkenberg, R. and Thorsten, J. (2000). Detecting concept drift with support vector machines. In Proc. 17th International Conference on Machine Learning (P. Langley, ed.) 487--494. Morgan Kaufmann, San Francisco. Klinkenberg, R. and Thorsten, J. (2000). Detecting concept drift with support vector machines. In Proc. 17th International Conference on Machine Learning (P. Langley, ed.) 487--494. Morgan Kaufmann, San Francisco.
Lane, T. and Brodley, C. E. (1998). Approaches to online learning and concept drift for user identification in computer security. In Proc. Fourth International Conference on Knowledge Discovery and Data Mining (R. Agrawal, P. Stolorz and G. Piatetsky-Shapiro, eds.) 259--263. AAAI Press, Menlo Park, CA. Lane, T. and Brodley, C. E. (1998). Approaches to online learning and concept drift for user identification in computer security. In Proc. Fourth International Conference on Knowledge Discovery and Data Mining (R. Agrawal, P. Stolorz and G. Piatetsky-Shapiro, eds.) 259--263. AAAI Press, Menlo Park, CA.
Newman, D. J., Hettich, S., Blake, C. L. and Merz, C. J. (1998). UCI repository of machine learning databases. Dept. Information and Computer Sciences, Univ. California, Irvine. Available at www.ics.uci.edu/~mlearn/MLRepository.html. Newman, D. J., Hettich, S., Blake, C. L. and Merz, C. J. (1998). UCI repository of machine learning databases. Dept. Information and Computer Sciences, Univ. California, Irvine. Available at www.ics.uci.edu/~mlearn/MLRepository.html.
Rendell, A. L. and Seshu, R. (1990). Learning hard concepts through constructive induction: Framework and rationale. Computational Intelligence 6 247--270. Rendell, A. L. and Seshu, R. (1990). Learning hard concepts through constructive induction: Framework and rationale. Computational Intelligence 6 247--270.
Ripley, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge Univ. Press. MR1438788 0853.62046 Ripley, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge Univ. Press. MR1438788 0853.62046
Shavlik, J., Mooney, R. J. and Towell, G. (1991). Symbolic and neural learning algorithms: An experimental comparison. Machine Learning 6 111--143. 1141.68327 Shavlik, J., Mooney, R. J. and Towell, G. (1991). Symbolic and neural learning algorithms: An experimental comparison. Machine Learning 6 111--143. 1141.68327
Thomas, L. C. (2000). A survey of credit and behavioural scoring: Forecasting financial risk of lending to consumers. International J. Forecasting 16 149--172. Thomas, L. C. (2000). A survey of credit and behavioural scoring: Forecasting financial risk of lending to consumers. International J. Forecasting 16 149--172.
Weiss, S. M., Galen, R. S. and Tadepalli, P. V. (1990). Maximizing the predictive value of production rules. Artificial Intelligence 45 47--71. 0899.68070 Weiss, S. M., Galen, R. S. and Tadepalli, P. V. (1990). Maximizing the predictive value of production rules. Artificial Intelligence 45 47--71. 0899.68070
