In many real-world learning tasks it is expensive to acquire a su cient number of labeled examples for training. This paper investigates methods for reducing annotation cost by sample selection. In this approach, during training the learning program examines many unlabeled examples and selects for labeling only those that are most informative at each stage. This avoids redundantly labeling examples that contribute little new information. Our work follows on previous research on Query By Committee, and extends the committee-based paradigm to the context of probabilistic classi cation. We describe a family of empirical methods for committee-based sample selection in probabilistic classication models, which evaluate the informativeness of an example by measuring the degree of disagreement between several model variants. These variants (the committee) are drawn randomly from a probability distribution conditioned by the training set labeled so far. The method was applied to the real-world natural language processing task of stochastic part-of-speech tagging. We nd that all variants of the method achieve a signi cant reduction in annotation cost, although their computational e ciency di ers. In particular, the simplest variant, a two member committee with no parameters to tune, gives excellent results. We also show that sample selection yields a signi cant reduction in the size of the model used by the tagger.
Argamon-Engelson et al. REF extend the QBC paradigm to the context of probabilistic classification and apply it to natural language processing tasks.
Committee-Based Sample Selection for Probabilistic Classifiers
Journal Of Artificial Intelligence Research, Volume 11, pages 335-360, 1999
Computer Science