Bayesian Analysis

A method for avoiding bias from feature selection with application to naive Bayes classification models

Longhai Li, Radford M. Neal, and Jianguo Zhang

Full-text: Access has been disabled (more information)

Abstract

For many classification and regression problems, a large number of features are available for possible use --- this is typical of DNA microarray data on gene expression, for example. Often, for computational or other reasons, only a small subset of these features are selected for use in a model, based on some simple measure such as correlation with the response variable. This procedure may introduce an optimistic bias, however, in which the response variable appears to be more predictable than it actually is, because the high correlation of the selected features with the response may be partly or wholly due to chance. We show how this bias can be avoided when using a Bayesian model for the joint distribution of features and response. The crucial insight is that even if we forget the exact values of the unselected features, we should retain, and condition on, the knowledge that their correlation with the response was too small for them to be selected. In this paper we describe how this idea can be implemented for "naive Bayes" models of binary data. Experiments with simulated data confirm that this method avoids bias due to feature selection. We also apply the naive Bayes model to subsets of data relating gene expression to colon cancer, and find that correcting for bias from feature selection does improve predictive performance.

Article information

Source
Bayesian Anal. Volume 3, Number 1 (2008), 171-196.

Dates
First available in Project Euclid: 22 June 2012

Permanent link to this document
http://projecteuclid.org/euclid.ba/1340370566

Digital Object Identifier
doi:10.1214/08-BA307

Mathematical Reviews number (MathSciNet)
MR2383256

Zentralblatt MATH identifier
1330.62252

Keywords
Feature selection Optimistic bias Naive Bayes models Gene expression data

Citation

Li, Longhai; Zhang, Jianguo; Neal, Radford M. A method for avoiding bias from feature selection with application to naive Bayes classification models. Bayesian Anal. 3 (2008), no. 1, 171--196. doi:10.1214/08-BA307. http://projecteuclid.org/euclid.ba/1340370566.


Export citation

References

  • Alon, U., Barkai, N., Notterman, D. A., Gish, K., Ybarra, S., Mack, D., and Levine, A. (1999). "Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays." Proceedings of the National Academy of Sciences (USA), 96: 6745–6750.
  • Ambroise, C. and McLachlan, G. J. (2002). "Selection Bias in Gene Extraction on the Basis of Microarray Gene-expression Data." PNAS, 99(10): 6562–6566. Available from http://www.pnas.org/cgi/content/abstract/99/10/6562.
    URL: Link to item
    Zentralblatt MATH: 1034.92013
  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
    Mathematical Reviews (MathSciNet): MR2247587
    Zentralblatt MATH: 1107.68072
  • Dawid, A. P. (1982). "The well-calibrated Bayesian." Journal of the American Statistical Association, 77(379): 605–610.
    Mathematical Reviews (MathSciNet): MR675887
    Zentralblatt MATH: 0495.62005
    Digital Object Identifier: doi:10.1080/01621459.1982.10477856
  • Guyon, I., Gunn, S., Nikravesh, M., and Zadeh, L. A. (2006). Feature Extraction: Foundations and Applications, volume 207 of Studies in Fuzziness and Soft Computing. Springer.
    Zentralblatt MATH: 1114.68059
  • Lecocke, M. L. and Hess, K. (2004). "An Empirical Study of Optimism and Selection Bias in Binary Classification with Microarray Data." UT MD Anderson Cancer Center Department of Biostatistics Working Paper Series, available from http://www.bepress.com/mdandersonbiostat/paper3/.
    URL: Link to item
  • Li, L. (2007). "Bayesian Classification and Regression with High Dimensional Features." Ph.D. thesis, University of Toronto. Available from http://math.usask.ca/$\sim$longhai.
    Mathematical Reviews (MathSciNet): MR2711841
  • Li, Y. H. and Jain, A. K. (1998). "Classification of Text Documents." The Computer Journal, 41(8): 537–546.
    Zentralblatt MATH: 0920.68048
  • Raudys, S., Baumgartner, R., and Somorjai, R. (2005). "On Understanding and Assessing Feature Selection Bias." Artificial Intelligence in Medicine, 468–472. Available from http://www.springerlink.com/content/8e41e3wncj7yqhx3.
    URL: Link to item
  • Singhi, S. K. and Liu, H. (2006). "Feature Subset Selection Bias for Classification Learning." Proceedings of the 23rd International Conference on Machine Learning. Available from http://portal.acm.org/citation.cfm?id=1143951.
    URL: Link to item

International Society for Bayesian Analysis

Bayesian Analysis

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more