Rank-Aware Negative Training for Semi-Supervised Text Classification (2023)

Abstract: Semi-supervised text classification-based paradigms (SSTC) typically employ the spirit of self-training. The key idea is to train a deep classifier on limited labeled texts and then iteratively predict the unlabeled texts as their pseudo-labels for further training. However, the performance is largely affected by the accuracy of pseudo-labels, which may not be significant in real-world scenarios. This paper presents a Rank-aware Negative Training (RNT) framework to address SSTC in learning with noisy label settings. To alleviate the noisy information, we adapt a reasoning with uncertainty-based approach to rank the unlabeled texts based on the evidential support received from the labeled texts. Moreover, we propose the use of negative training to train RNT based on the concept that “the input instance does not belong to the complementary label”. A complementary label is randomly selected from all labels except the label on-target. Intuitively, the probability of a true label serving as a complementary label is low and thus provides less noisy information during the training, resulting in better performance on the test data. Finally, we evaluate the proposed solution on various text classification benchmark datasets. Our extensive experiments show that it consistently overcomes the state-of-the-art alternatives in most scenarios and achieves competitive performance in the others. The code of RNT is publicly available on GitHub.

## introduction the text classification task aims to associate a piece of text with a corresponding class that could be a sentiment, topic, or category. with the rapid development of deep neural networks, text classification has experienced a considerable shift to- * corresponding author. earlier sstc-based approaches adopt various data augmentation techniques via back-translation. they employ consistency loss between the predictions of unlabeled texts and corresponding augmented texts by translating the text into a targeted language and then translating it back to the source language @xcite @xcite . however, the performance of these approaches requires an additional neural machine translation (nmt), which may not be accurate and bothersome in real-world scenarios. recently, sstc has experienced a shift toward self-training, and plm fine-tuning @xcite . the basic idea is to fine-tune plms on the labeled data and iteratively employ prediction on the unlabeled data as pseudo-labels for further training. however, the pseudo-labels are treated equally likely to the truth labels and thus may lead to error accumulation @xcite . in this paper, we propose a rank-aware negative training (rnt) framework to address sstc under learning with noisy label settings. to alleviate the domination of noisy information during training, we adopt reasoning with an uncertaintybased approach to rank the unlabeled texts by measuring their shared features, also known as evidential support, with the labeled texts. eventually, the shared features that serve as a medium to convey knowledge from labeled texts (i.e., evidence) to the unlabeled texts (i.e., inference) are regarded as belief functions to reason about the degree of noisiness. these belief functions are combined to reach a final belief about the text being mislabeled. in other words, we attempt to discard the texts whose pseudo-labels may introduce inaccurate information to the training process. moreover, we propose using negative training (nt) @xcite to robustly train with potential noisy pseudo-labels. unlike positive training, nt is an indirect learning method that trains the network based on the concept that ''the input sentence does not belong to the complementary label'', whereas a complementary label is randomly generated from the label space except the label of the sentence on-target. considering the ag news dataset, given a sentence annotated as sport, the complementary label is randomly selected from all labels except sport (e.g., business). intuitively, the probability of a true label serving as a complementary label is low and thus can reduce the noisy information during the training process. finally, we conduct extensive experiments on various text classification benchmark datasets with different ratios of labeled examples, resulting in better performance on the test data. experimental results suggest that rnt can mostly outperform the sstc-based alternatives. moreover, it has been empirically shown that rnt can perform better than plms fine-tuned on sufficient labeled examples. in brief, our main contributions are three-fold: our extensive experiments show that it consistently overcomes the state-of-the-art alternatives in most cases and achieves competitive performance in others. ## related work this section reviews the existing solutions of the sstc task and learning with noisy labels. ## ranked-aware negative training this section describes the proposed framework, namely, rank-aware negative training (rnt), for semi-supervised text classification. an example of rnt is depicted in figure 1 . suppose we have a training dataset d consisting of a limited labeled set d l and a large unlabeled set d u . we follow the pseudo-labels method introduced by @xcite to associate d u with pseudo-labels based on the concept of positive training. simply put, we fine-tune the pre-trained language models (e.g., bert) on the d l set. it is noteworthy that we use bert for a fair comparison, while other models can be used similarly. as the pseudo-labels are not manually annotated, we propose ranking the texts based on their potential for mislabeling to identify and discard the most risky mislabeled texts. specifically, we first capture the shared information (i.e., we refer to this as the evidential support) between the labeled and unlabeled instances. then, we measure the amount of support that an unlabeled instance receives from the labeled instances being correctly labeled. we denote the filtered set as d u in figure 1 . finally, we train on both d l and d u through the concept of the negative training. next, we describe the framework in detail. ## evaluation and results we describe the evaluation tasks and report the experimental results in this section. the evaluation criteria are: (i) is rnt able to rank the instances of being mislabeled?; (ii) can the filtered data enhance the performance of the clean test data? figure 5 : ranking evaluation on dev sets with n @xmath0. the ranked dev set is first split into 10 proportions equally-likely. then, each proportion is inferred (i.e., calculate its accuracy) independently. the accuracy gradually drops as the noisy texts increase. in our experiment, we choose the proportions whose instances meet λ section 4.3 for further nt training. ## conclusion and future work in this paper, we proposed a self-training semisupervised framework, namely, rnt, to address the text classification problem in learning with noisy label settings. rnt first discards the high risky mislabeled texts based on reasoning with uncertainty theory. then, it uses the negative training technique to reduce the noisy information during training. our extensive experiments have shown that rnt mostly outperformed sstc-based alternatives. despite the robustness of negative training, clean samples that have identical distributions with test data are subjected to complementary labels. consequently, both clean and potentially noisy samples contribute equally to the final performance. a combination of both positive and negative training strategies in a unified framework can remedy the abundance of noisy samples; however, this needs further investigation.