Seed Word Selection for Weakly-Supervised Text Classification with Unsupervised Error Estimation (2021)

Abstract: Weakly-supervised text classification aims to induce text classifiers from only a few user-provided seed words. The vast majority of previous work assumes high-quality seed words are given. However, the expert-annotated seed words are sometimes non-trivial to come up with. Furthermore, in the weakly-supervised learning setting, we do not have any labeled document to measure the seed words’ efficacy, making the seed word selection process “a walk in the dark”. In this work, we remove the need for expert-curated seed words by first mining (noisy) candidate seed words associated with the category names. We then train interim models with individual candidate seed words. Lastly, we estimate the interim models’ error rate in an unsupervised manner. The seed words that yield the lowest estimated error rates are added to the final seed word set. A comprehensive evaluation of six binary classification tasks on four popular datasets demonstrates that the proposed method outperforms a baseline using only category name seed words and obtained comparable performance as a counterpart using expert-annotated seed words.

## introduction weakly-supervised text classification eliminates the need for any labeled document and induces classifiers with only a handful of carefully chosen seed words. however, some researchers pointed out that the choice of seed words has a significant impact on the performance of weakly-supervised models @xcite . the vast majority of previous work assumed high-quality seed words are given. however, many seed words reported in previous work are not intuitive to come up with. for example, in @xcite , the seed words used for the category "soccer" are {cup, champions, united} instead of more intuitive keywords like "soccer" or "football". we conjecture the authors might have tried these more general keywords but avoided them because they do not perform well. while it is common to use labeled corpora to evaluate weakly-supervised text classifiers in the literature, we do not have access to any labeled document for new categories in the real-world setting. therefore, there is no way to measure the model's performance and select the seed words that yield the best accuracy. a similar concern on assessing active learning performance at runtime has been raised by @xcite . in this work, we device optimseed, a novel framework to automatically compose and select seed words for weakly-supervised text classification. we firstly mine (noisy) candidate seed words associated with the category names. we then train interim models with individual candidate seed words in an iterative manner. lastly, we use an unsupervised error estimation method to estimate the interim models' error rates. the keywords that yield the lowest estimated error rates are selected as the final seed word set. a comprehensive evaluation of six classification tasks on four popular datasets demonstrates the effectiveness of the proposed method. the proposed method outperforms a baseline using only the category name as seed word and obtained comparable performance as a counterpart using expert-annotated seed words. we use binary classification as a case study in this work, while the idea can be generalized to multi-class classification using one-vs.-rest strategy. the contributions of this work are three-fold: ## related work we review the literature in three related fields: (1) weakly-supervised text classification, (2) unsupervised error estimation, and (3) keyword mining. ## experimental setup we use six binary classification tasks from four datasets to evaluate our framework. we choose the evaluation tasks so that they cover different granularities and domains. the details are as follows: we report the performance of the following weakly-supervised models besides generalized expectation (ge): we also report the performance of lr, a supervised logistic regression model trained using all the documents in the training set 3 . in all experiments, we mine 16 candidate seed words with the highest pmi-f req score for each category. we select a candidate keyword for the final classifier if its estimated accuracy is among the top three or is higher than 0.9 foot_3 . for ge, we use a reference distribution of 0.9 (meaning a labeled keyword is expected to appear in its specified categories 90% of the time) following @xcite . ## classification performance while conwea claimed to resolve ambiguity through contextualized embeddings, we observed that it works well only when the input seed words are unambiguous ("ours" or "gold" column). on the business-sports classification task, its accuracy was only 39.1% while other baselines could achieve over 90%. we inspected the model and found the keywords expanded by conwea are much noisier than optimseed, which caused the poor performance. we can make similar observations on the performance of sentiment classification tasks ( ## case study to demonstrate the working of our proposed framework, we present a case study on the classification task "international business" vs. "economy" in stage:acc seed words for "economy" init: 70.7 economy keyword expansion: 79.3 purchases pace index borrowing unemployment economists economy stimulus rates recovery economist rate fed reserve inflation growth final: 81.7 economist economists rate recovery index ## conclusion weakly-supervised text classification can induce classifiers with a handful of carefully-chosen seed words instead of labeled documents. however, the choice of seed words has a significant impact on classification performance. we proposed optimseed, a novel framework to compose the seed words automatically. it first mines keywords associated with the category name and then estimates each seed word's impact directly using unsupervised error estimation. the framework outputs seed words yielding a comparable performance as expert-curated seed words, virtually eliminating human experts from the loop.