Open Access
Translator Disclaimer
March 2008 A method for avoiding bias from feature selection with application to naive Bayes classification models
Longhai Li, Radford M. Neal, Jianguo Zhang
Bayesian Anal. 3(1): 171-196 (March 2008). DOI: 10.1214/08-BA307


For many classification and regression problems, a large number of features are available for possible use --- this is typical of DNA microarray data on gene expression, for example. Often, for computational or other reasons, only a small subset of these features are selected for use in a model, based on some simple measure such as correlation with the response variable. This procedure may introduce an optimistic bias, however, in which the response variable appears to be more predictable than it actually is, because the high correlation of the selected features with the response may be partly or wholly due to chance. We show how this bias can be avoided when using a Bayesian model for the joint distribution of features and response. The crucial insight is that even if we forget the exact values of the unselected features, we should retain, and condition on, the knowledge that their correlation with the response was too small for them to be selected. In this paper we describe how this idea can be implemented for "naive Bayes" models of binary data. Experiments with simulated data confirm that this method avoids bias due to feature selection. We also apply the naive Bayes model to subsets of data relating gene expression to colon cancer, and find that correcting for bias from feature selection does improve predictive performance.


Download Citation

Longhai Li. Radford M. Neal. Jianguo Zhang. "A method for avoiding bias from feature selection with application to naive Bayes classification models." Bayesian Anal. 3 (1) 171 - 196, March 2008.


Published: March 2008
First available in Project Euclid: 22 June 2012

zbMATH: 1330.62252
MathSciNet: MR2383256
Digital Object Identifier: 10.1214/08-BA307

Keywords: Feature selection , gene expression data , Naive Bayes models , Optimistic bias

Rights: Copyright © 2008 International Society for Bayesian Analysis


Vol.3 • No. 1 • March 2008
Back to Top