Bias and stability of single variable classifiers for feature ranking and selection

• Published in: Expert systems with applications Vol. 41; no. 15; pp. 6945 - 6958
• Main Authors: Fakhraei, Shobeir, Soltanian-Zadeh, Hamid, Fotouhi, Farshad
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier Ltd 01-11-2014; Elsevier
• Subjects: AdaBoost; Applied sciences; Artificial intelligence; Bias; Classification; Classifiers; Computation; Computer science; control theory; systems; Connectionism. Neural networks; Data processing. List processing. Character string processing; Decision theory. Utility theory; Dimension reduction; Exact sciences and technology; Feature ranking; Feature selection; K-Nearest Neighbors; Logistic Regression; Mathematical models; Memory organisation. Data processing; Multilayer Perceptron; Naïve Bayes; Operational research and scientific management; Operational research. Management science; Random Forests; Ranking; Reduction; Single variable classifier; Software; Stability; Support Vector Machines; K-Nearest Neighbors; Feature selection; Naïve Bayes; Stability; Logistic Regression; Bias; Random Forests; Support Vector Machines; Single variable classifier; Dimension reduction; AdaBoost; Multilayer Perceptron; Feature ranking; Performance evaluation; Aggregate model; Vector support machine; Selection criterion; Probability learning; Feedforward; Bayes estimation; Capability index; Nearest neighbour; Data analysis; Dimensionality; Residence time; Hierarchical classification; Decision making; Empirical method; Neural network; Random decision forests; Reduction method; Discrimination; Unbiased estimation; Multilayer perceptrons; Dimension Reduction; Feature Ranking; Single Variable Classifier; Feature Selection
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2014.05.007

•We show that SVC feature ranking is highly sensitive to the choice of classifiers.•Ranking and classification with the same classifier is not always the best approach.•NB and AB generate better results than KNN and RF when used in both roles.•Multiclassifier ranking ensembles perform above average but not at the overall best.•We should also account for classifier parameter setting in SVC feature ranking.
Feature rankings are often used for supervised dimension reduction especially when discriminating power of each feature is of interest, dimensionality of dataset is extremely high, or computational power is limited to perform more complicated methods. In practice, it is recommended to start dimension reduction via simple methods such as feature rankings before applying more complex approaches. Single variable classifier (SVC) ranking is a feature ranking based on the predictive performance of a classifier built using only a single feature. While benefiting from capabilities of classifiers, this ranking method is not as computationally intensive as wrappers. In this paper, we report the results of an extensive study on the bias and stability of such feature ranking method. We study whether the classifiers influence the SVC rankings or the discriminative power of features themselves has a dominant impact on the final rankings. We show the common intuition of using the same classifier for feature ranking and final classification does not always result in the best prediction performance. We then study if heterogeneous classifiers ensemble approaches provide more unbiased rankings and if they improve final classification performance. Furthermore, we calculate an empirical prediction performance loss for using the same classifier in SVC feature ranking and final classification from the optimal choices.