LOPS : Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification (2022)

Abstract: Weakly supervised text classification methods typically train a deep neural classifier based on pseudo-labels. The quality of pseudo-labels is crucial to final performance but they are inevitably noisy due to their heuristic nature, so selecting the correct ones has a huge potential for performance boost. One straightforward solution is to select samples based on the softmax probability scores in the neural classifier corresponding to their pseudo-labels. However, we show through our experiments that such solutions are ineffective and unstable due to the erroneously high-confidence predictions from poorly calibrated models. Recent studies on the memorization effects of deep neural models suggest that these models first memorize training samples with clean labels and then those with noisy labels. Inspired by this observation, we propose a novel pseudo-label selection method LOPS that takes learning order of samples into consideration. We hypothesize that the learning order reflects the probability of wrong annotation in terms of ranking, and therefore, propose to select the samples that are learnt earlier. LOPS can be viewed as a strong performance-boost plug-in to most existing weakly-supervised text classification methods, as confirmed in extensive experiments on four real-world datasets.

## introduction weakly supervised text classification methods @xcite @xcite @xcite @xcite typically start with generating pseudo-labels, and train a deep neural classifier to learn the mapping between documents and classes. there is no doubt that the quality of pseudo-labels plays a fundamental role in the final classification accuracy, however, they are inevitably noisy due to their heuristic * jingbo shang is the corresponding author. nature. pseudo-labels are typically generated by some heuristic, for example, through string matching between the documents and user-provided seed words @xcite . deep neural networks (dnns) trained on such noisy labels have a high risk of making erroneous predictions. more importantly, when self-training is employed, such error can be further amplified upon boostrapping. to address this problem, in this paper, we study the pseudo-label selection in weakly supervised text classification, aiming to select a high quality subset of the pseudo-labeled documents (in every iteration when using self-training) that can potentially achieve a higher classification accuracy. a straightforward solution is to first train a deep neural classifier based on the pseudo-labeled documents and then threshold the documents by the predicted probability scores corresponding to their pseudo-labels. however, dnns usually have a poor calibration and generate overconfident predicted probability scores @xcite . for example, on new york times (nyt) coarse-grained dataset, as shown in figure 1 (a), 60% of wrong instances in the pseudo-labeled documents have a predicted probability by bert greater than 0.9 for their wrong pseudo-labels. recent studies on the memorization effects of dnns show that they memorize easy and clean instances first, and gradually learn hard instances and eventually memorize the wrong annotations @xcite @xcite . we have confirmed this in our experiments for different classifiers. for example, as shown in figure 1 (b), bert classifier learns most of the clean instances in the first epoch and learns wrong instances across all epochs. although it also learns good number of wrong instances in the first epoch, it is significantly less than the probability-based selection in figure 1 @xcite . therefore, we define the learning order of a pseudo-labeled document as the epoch when it is learnt during training i.e. when the training model's prediction is the same as its given pseudo-label. since correct samples are learnt first, we hypothesize that learning order-based selection will be able to filter out wrongly labeled samples. inspired by our observation, we propose a novel learning order inspired pseudo-label selection method lops, as shown in figure 2 . specifically, lops involves training a probing classifier on pseudo-labeled data and tracking the learning order of samples. we define a sample is learnt if and only if the classifier trained on pseudo-labels gives the same argmax prediction as its pseudolabel at the end of an epoch. we stop the training when at least τ % of samples corresponding to each class are learnt and select all the learnt samples. then, we train a text classifier on these selected pseudo-labeled documents that is further used for inference. we empirically show that lops can boost the accuracy of various weakly supervised text classification methods and it is much more effective and stable than probability score-based selections. our contributions are summarized as follows: • we propose a novel pseudo-label selection method lops that takes learning order of samples into consideration. in @xcite , pseudo-labels are created from the predictions of a trained neural network. @xcite combines different weak signals to produce soft labels. ## problem and motivation weakly supervised classification refers to the problem with inputs (1) a set of unlabeled text documents @xmath0, where @xmath1. (2) and m target labels @xmath2, . . . , m }. our goal is to find a labeling function f ∶ @xmath3. here we denote y * as the unknown true label of a document x. to cold start the classification of unlabeled documents, a source of weak supervision has to be introduced, which can come from various sources such as label surface names @xcite , label-indicative seed words @xcite , or rules (karamanolakis et al., 2021a). given a "weak" labeling function w ∶ @xmath4, pseudo-labels are then generated on a subset of the unlabeled documents, which yields a labeled subset @xmath5, w(x))}. for convenience, we denote d[j] to be the set of all documents that are pseudo-labeled as class j in d, namely d[j] = {(x, w(x)) ∈ d|w(x) = j}. pseudo-labels are noisy due to their heuristic nature. for example, on the nyt fine-grained dataset, we generate pseudo-labels using five different strategies @xcite @xcite @xcite and compute their noise ratios. as expected, no strategy is perfect and all of them generate noisy labels, ranging from 15% to 50% (see when a classifier is trained on such noisy training data, it can make some high confident erroneous predictions. and, upon bootstrapping the classifier on unlabeled data, it has a snowball effect where such high confident erroneous predictions are added to the training data, and thus corrupting it more. as this process repeats for a few iterations, it adds more noise and significantly affects the final performance. therefore, identifying and selecting the correctly labeled samples is necessary and has a huge potential for a boost in performance. note that, if the labels are not selected carefully, it could instead hurt the performance. our pseudo-label selection problem. the weak supervision is likely to generate a noisy labeled set, which means w(x) ≠ y * for some documents x. we denote d ✓ as the set of correctly labeled documents and d × = d \ d ✓ as the set of wrongly labeled documents, where note that pseudo-label selection is conceptually related to failure prediction @xcite @xcite and outof-distribution detection @xcite @xcite . however, the major difference here is for pseudo-label selection we have to detect wrong annotations in the training phase instead of inference phase. ## our lops framework in this section, we explain our framework lops in detail. first, we give an overview of confidence function-based pseudo-label selection and discuss probability score as confidence function. then, we explain learning order as confidence function. finally, we show our algorithm that performs selection based on learning order. ## experiments we evaluate our label selection method based on end-to-end classification performance using different state-of-the-art classifiers and weakly supervised text classification frameworks. and also, we evaluate learning order as a confidence function and provide a comparison with probability score as confidence function. ## conclusion and future work in this paper, we proposed lops, a novel learning order inspired pseudo-label selection method. our method is inspired from recent studies on memorization effects that showed that clean samples are learnt first and then wrong samples are memorized. experimental results demonstrate that our method is effective, stable and can act as a performance boost plugin on many text classifiers and weakly supervised text classification methods. in the future, we are interested in automatically identifying the right granularity to measure learning order for a given dataset. moreover, we are also interested in analyzing the learning order in classification tasks in image and speech domains.