using recursive classification to discover predictive features
Finding most predictive features for statistical classification is a challenging problem and has important applications. Support Vector Machines (SVMs), for example, have been found successful with a recursive procedure in selecting most important genes for cancer prediction. It is not well understood, however, how much the success depends on the choice of the classifier, and how much on the recursive procedure. We answer this question by examining multiple classifiers (SVM, ridge regression and Rocchio) with feature selection in recursive and non-recursive settings, on a DNA microarray dataset (AMLALL) and a text categorization benchmark (Reuters-21578). We found recursive ridge regression most effective: its best classification performance (zero error) on the AMLALL dataset was obtained when using only 3 genes (selected from over 7000), which is more impressive than the best published result on the same benchmark - zero error of recursive SVM using 8 genes. On Reuters-21578, recursive ridge regression also achieves the best result ever published (the improvement was verified in a significance test). An in-depth analysis of the experimental results shows that the choice of classifier heavily influences the recursive feature selection process: the ridge regression classifier tends to penalize redundant features to a much larger extent than the SVM does.
feature selection text categorization machine learning