Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification (2023)

Abstract: Recent advances in weakly supervised text classification mostly focus on designing sophisticated methods to turn high-level human heuristics into quality pseudo-labels. In this paper, we revisit the seed matching-based method, which is arguably the simplest way to generate pseudo-labels, and show that its power was greatly underestimated. We show that the limited performance of seed matching is largely due to the label bias injected by the simple seed-match rule, which prevents the classifier from learning reliable confidence for selecting high-quality pseudo-labels. Interestingly, simply deleting the seed words present in the matched input texts can mitigate the label bias and help learn better confidence. Subsequently, the performance achieved by seed matching can be improved significantly, making it on par with or even better than the state-of-the-art. Furthermore, to handle the case when the seed words are not made known, we propose to simply delete the word tokens in the input text randomly with a high deletion ratio. Remarkably, seed matching equipped with this random deletion method can often achieve even better performance than that with seed deletion.

## introduction recently, weakly supervised text classification, because of its light requirement of human effort, has been extensively studied @xcite @xcite @xcite @xcite . specifically, it requires only high-level human guidance to label the text, such as a few rules provided by human experts that match the text with the labels. these labels, which are not necessarily correct and are thus often dubbed as pseudo-labels, are then employed to train the text classifier following a standard fully supervised or semi-supervised training framework. state-of-the-art methods mostly focus on designing sophisticated human guidance to obtain high-quality labels, through contextualized weak supervision @xcite , prompting language models @xcite , clustering for soft matching @xcite , and complicated interactions between seeds @xcite . in this paper, we revisit the seed matching-based weak supervision (denoted as vanilla) @xcite @xcite , which is arguably the simplest way to generate pseudo-labels, and show that its power was greatly underestimated. specifically, this simple method matches input text with a label if the user-provided seed words of this label are contained in the input text. for example, in sentiment analysis, a document will be labeled as "positive" if it contains the word "happy". a text classifier is then trained based on all these pseudo-labels. one can expect a non-trivial number of errors in the seed matching-based pseudo-labels. in an ideal case, if we can select only those correct pseudolabels for training, the accuracy of the text classifier can be significantly boosted. for example, on the 20 newsgroups dataset, ideally with only those correct pseudo-labels one can get an accuracy of 90.6%, compared to 80.1% obtained on all pseudolabels (see more in section 4). in practice, to select those correct labels, a common way is to use the confidence score of a classifier trained on pseudolabels @xcite . however, those highconfidence pseudo-labels may not be correct in the weakly-supervised setting, likely because the classifier may fail to learn reliable confidence on these noisy pseudo-labels @xcite . in this paper, we take a deep dive into this problem and find that, surprisingly, the high noise rate ## conclusion and future work in this paper, we revisit the simple seed matchingbased weakly supervised text classification method and show that if its pseudo-labels are properly debiased, it can achieve state-of-the-art accuracy on many popular datasets, outperforming more sophisticated types of weak supervision. specifically, our controlled experiments show that confidence-base selection of seed matching-based pseudo-labels is ineffective largely because of the label bias injected by the simple, yet erroneous seed-match rule. we propose two effective debiasing methods, seed deletion, and random deletion, which can mitigate the label bias and significantly improve seed matchingbased weakly supervised text classification. in future work, we plan to extend this debiasing methodology to broader problems and methods. for example, for weakly supervised text classification, we wish to explore a generalization of the debiasing strategy to more sophisticated types of weak supervision. it will also be interesting to develop a backdoor defense framework around the proposed methods, especially random deletion.