Incorporating feature selection into a clas sification or regression method often carries a number of advantages. In this paper we formalize feature selection specifically from a discriminative perspective of improving clas sification/regression accuracy. The feature selection method is developed as an extension to the recently proposed maximum entropy discrimination (MED) framework. We de scribe MED as a flexible (Bayesian) regular ization approach that subsumes, e.g., support vector classification, regression and exponen tial family models. For brevity, we restrict ourselves primarily to feature selection in the context of linear classification/regression methods and demonstrate that the proposed approach indeed carries substantial improve ments in practice. Moreover, we discuss and develop various extensions of feature selec tion, including the problem of dealing with example specific but unobserved degrees of freedom -alignments or invariants. [4, 6, 14] . We provide an additional point of contact in the current paper. The focus of this paper is on feature selection. The fea ture selection problem may involve finding the struc ture of a graphical model (as in [12] ) or identifying a set of components of the input examples that are relevant for a classification task. More generally, fea ture selection can be viewed as a problem of setting discrete structural parameters associated with a spe cific classification or regression method. We subscribe here to the view that feature selection is not merely for reducing the computational load associated with a high dimensional classification or regression problem but can be tailored primarily to improve prediction ac curacy ( cf. [9] ). This perspective excludes a number of otherwise useful feature selection approaches such as any filtering method that operates independently from the classification task/method at hand. Linear classi fiers, for example, impose strict constraints about _ the type of features that are at all useful. Such constramts should be included in the objective function governing the feature selection process. The form of feature selection we develop in this paper results in a type of feature weighting. Each feature or structural parameter is associated with a probabil ity value. The feature selection process translates into estimating the most discriminative probability distri bution over the structural parameters. Irrelevant fea tures quickly receive low albeit non-zero probabilities of being selected. We emphasize that the feature selec tion is carried out jointly and discriminatively together with the estimation of the specifi c classification or re gression method. This type of feature selection is, per haps surprisingly, most beneficial when the number of training examples is relatively small compared to their dimensionality. The paper is organized as follows. We begin by mo tivating the discriminative maximum entropy frame work from the point of view of regularization theory. We then explicate how to solve classification and re gression problems in the context of maximum entropy
Under the maximum entropy discrimination models, feature selection was studied in REF by introducing a set of structural variables.
Feature Selection and Dualities in Maximum Entropy Discrimination
ArXiv
ArXiv
Computer Science Mathematics