Extremely Weakly-supervised Text Classification with Wordsets Mining and Sync-Denoising (2024)

Abstract: Extremely weakly-supervised text classification aims to classify texts without any labeled data, but only relying on class names as supervision. Existing works include prompt-based and seed-based methods. Prompt-based methods prompt language model with instructions, while seed-based methods generate pseudo-labels with word matching. Both of them have significant flaws, including zero-shot instability and context-dependent ambiguities. This paper introduces SetSync, which follows a new paradigm, i.e. wordset-based, which can avoid the above problems. In SetSync, a class is represented with wordsets, and pseudo-labels are generated with wordsets matching. To facilitate this, we propose to use information bottleneck to identify class-relevant wordsets. Moreover, we regard the classifier training as a hybrid learning of semi-supervised and noisy-labels, and propose a new training strategy, termed sync-denoising. Extensive experiments on 11 datasets show that SetSync outperforms all existing prompt and seed methods, exceeding SOTA by an impressive average of 8 points.

## introduction as a fundamental task in nlp, text classification has a wide range of real-world applications. however, the substantial annotation costs present significant challenges and obstacles to its practical implementation. as a result, extremely weaklysupervised text classification (wtc) has garnered considerable attention @xcite , which requires no human-annotated datasets, but relies solely on the class names as supervision signals to perform the text classification task. generally, weakly-supervised text classification involves two main steps: pseudo-labels generation and text classifier training, as shown in fig. 1 . 1) firstly, pseudo-labels for unlabeled texts are generated according to class names, which can be roughly divided into two major mainstream, namely prompt methods @xcite and seed methods @xcite . the prompt methods generate pseudo-labels by prompting language models (e.g. gpt) with instructions, and the seed methods first expand the seed class names into a larger set of related words and then generate pseudo-labels with a matching strategy. fig. 1 shows a specific example. 2) in the subsequent text classifier training step, the text classifier will be trained on generated pseudo-labels. the training strategies of the classifier mainly include fullysupervised training and self-training, where the former directly trains the classifier on pseudo-labeled data, and the latter utilizes self-training to exploit texts that are not assigned pseudo-labels. although previous methods have achieved some success, we observe that they are still sub-optimal: 1) prompt methods tend to have a lower performance, as reported in a recent review @xcite . this is because language models are known to exhibit bias towards text sequences more common in their pre-training data, which can lead to instability in zero-shot settings. 2) seed methods are known to be context-dependent and prone to ambiguity. for example, in the case of fig. 1 , the text "i am not in a bad mood" is an example of positive emotion being mislabeled as negative due to the presence of "bad" seed keywords in the negative class. in this case, the combination of "bad" and "not" can express the emotion accurately. although there have been some methods to solve the context-dependency problem before @xcite , we believe that they are still not thorough and perfect enough. 3) additionally, existing methods do not consider the noise of pseudo-labels in the training of classifier. specifically, existing wtc works extensively utilize self-training to make full use of samples without any pseudo-labels @xcite . however, during the self-training, incorrect pseudo-labels will lead to the propagation and expansion of errors, ultimately contributing to a decrease in classifier performance. in this paper, we solve the above problem with a novel framework called setsync, where we embark on enhancing wtc from two primary perspectives: pseudo-labels generation based on wordsets and classifier training with sync-denoising. to begin with, considering the context-dependency ambiguities in seed methods, we believe that individual seed words alone cannot effectively represent a category. thus, we propose to utilize wordsets to represent a category and generate pseudo-labels, where a wordset is a collection that contains some keywords. for example, {not, bad} and {good} are two wordsets for class positive, while {never, good} is a wordset for class negative, as shown in fig. 1 . the previous seed word expansion can be seen as a special case of ours, where each wordset contains only one word. thus, with the help of information bottleneck (ib) @xcite theory, we propose wordsets information bottleneck (wib) to mine category-related wordsets. information bottleneck was originally proposed for signal processing, attempts to find a short code of the input signal but preserve maximum information. contrasting with the continuous nature of signal space, the space of words is inherently discrete. applying ib directly to wordset mining will face problems such as high time complexity and difficulty in optimization. therefore, we propose wib in discrete space, which uses high-frequency filtering combined with frequent itemset mining for optimization. moreover, observing pseudo-labels, we can find it is a hybrid setting of semi-supervised and noisylabel learning, i.e. only a portion of the texts have pseudo-labels and they contain errors. therefore, we propose a new training strategy, named sync-denoising to train the classifier, where noisy pseudo-labels and unlabeled data participate in training synchronously under a unified denoising framework. we assume that the noise levels of pseudo-labels and unlabeled data obey two independent gaussian distributions and dynamically sample weights from them for denoising learning. we conducted extensive experiments on 11 text benchmarks. results show that our method substantially outperforms all existing seed-based and prompt-based methods, improving the accuracy of all 11 datasets by about 8 points on average. ## experiments for experimental settings and hyperparameter, please refer to the appendix.c. ## conclusion in this paper, we introduce setsync, which utilize wordsets to represent categories and generate pseudo-labels. to achieve this, we propose a classrelevant wordsets mining algorithm with wordsets information bottleneck. moreover, we revisit the classifier training in wtc and propose a new training strategy, called sync-denoising, which jointly optimize unlabeled and noisy labeled samples with a unified denoising framework. extensive experiment results on 11 datasets shows that setsync substantially outperforms all existing methods.