Correcting the KullbackLeibler distance for feature selection
A frequent practice in feature selection is to maximize the KullbackLeibler (KL) distance between target classes. In this note we show that this common custom is frequently suboptimal, since it fails to take into account the fact that classification occurs using a finite number of samples. In classification, the variance and higher order moments of the likelihood function should be taken into account to select feature subsets, and the KullbackLeibler distance only relates to the mean separation. We derive appropriate expressions and show that these can lead to major increases in performance.
kullbackleibler distance feature selection classification error receiver operating curve (roc) neymanpearson classification