S elf M ix: Robust Learning against Textual Label Noise with Self-Mixup Training (2022)

Abstract: The conventional success of textual classification relies on annotated data, and the new paradigm of pre-trained language models (PLMs) still requires a few labeled data for downstream tasks. However, in real-world applications, label noise inevitably exists in training data, damaging the effectiveness, robustness, and generalization of the models constructed on such data. Recently, remarkable achievements have been made to mitigate this dilemma in visual data, while only a few explore textual data. To fill this gap, we present SelfMix, a simple yet effective method, to handle label noise in text classification tasks. SelfMix uses the Gaussian Mixture Model to separate samples and leverages semi-supervised learning. Unlike previous works requiring multiple models, our method utilizes the dropout mechanism on a single model to reduce the confirmation bias in self-training and introduces a textual level mixup training strategy. Experimental results on three text classification benchmarks with different types of text show that the performance of our proposed method outperforms these strong baselines designed for both textual and visual data under different noise ratios and noise types. Our anonymous code is available at https://github.com/noise-learning/SelfMix .

## introduction the excellent performance of deep neural networks (dnns) depends on data with high-quality annotations. however, data obtained from the real world is inevitably mixed with wrong labels @xcite @xcite . models trained on these noisy datasets would easily overfit the noisy labels @xcite , especially for pretrained large models @xcite , and the performance will be negatively affected. research on learning with noisy labels (lnl) has gained popularity. previous work has revealed that clean samples and noisy samples play different roles in the training process and behave differently in terms of loss values or convergence speeds etc. @xcite . different types of noise have different effects on the training. for instance, the impact of class-conditional noise (ccn) can simulate the confusion between similar classes, and the effect of instance-dependent noise (idn) can be more complex. most of the current methods perform experiments on visual data. label noise on visual data often goes against objective facts and is easy to distinguish. as for nlp, there may be disagreement even among expert annotators due to the complexity of semantic features and the subjectivity of language understanding. for example, suppose there is a piece of news about "the economic benefit of competitive sports to our cities". in that case, it is hard to tell whether it belongs to economic news or sports news without fully understanding the contextual information. although a few works pay attention to the natural language area, their methods are mostly based on the trained-from-scratch models like lstm and text-cnn @xcite . however, plms might be a better choice since the whole training process can be divided into two stages, and the wrong labels do not corrupt the pre-training process. in conclusion, it is vital to explore how to learn with noisy labels on textual data and use the robust plms as the base model. this paper proposes self-mix, i.e., a self-distillation robust training method based on the pre-trained models. section 2 introduces some related works and explains the motivation of our proposed method. our contributions can be concluded as follows: learning with noisy labels. a direct yet effective idea to handle label noise is to find the noisy samples and reduce their influence by resampling or reweighting @xcite . jiang et al. (2018) train another neural network to provide a curriculum to help studentnet focus on the samples whose labels is probably correct. han et al. (2018) jointly train two deep neural networks and feed each model the top r% samples with the lowest loss evaluated by the other model in each mini-batch. following han et al. (2018), yu et al. (2019) explore how disagreement can help the model. some researchers believe that there exists a transition from ground-truth label distribution to the noisy label distribution and estimate the noise transition matrix to absorb this transition (goldberger and ben-reuven, 2016). northcutt et al. (2021) directly estimate the joint distribution matrix between the noisy labels and real labels. garg et al. (2021) use a fully connected layer to capture the distribution transition. however, most of these methods either need model ensembling or require cross-validation, which is time-consuming and needs multiple parameters. some other works focus on designing a more robust training strategy. since dnns with cross-entropy loss tend to overfit noisy labels @xcite , some researchers redesign noise-robust loss functions @xcite @xcite . when trained on noisy data, dnns tend to learn from the clean data during an "early learning" phase before eventually memorizing the wrong data @xcite , based on which @xcite offer an easy regularization capitalizing on early learning. some other works like @xcite find that only partial parameters are essential for generalization, which offers us a new perspective to reconsider what difference exactly the noisy labels make to the model's learning. this kind of approach treats all samples indiscriminately thus the performance is sometimes unsatisfactory under a high noise ratio. some excellent work combines these two ideas @xcite . @xcite add an auxiliary noise model n m over the classifier to predict noisy labels and jointly train the classifier and the noise model through a de-noising loss function. @xcite progressively sieve out corrupted examples and then leverage semi-supervised learning. mixup training. mixup training @xcite is a widely used data-augmentation method to alleviate memorization and sensitivity to adversarial samples on visual data. it combines the inputs and targets of two random training samples to generate augmented samples. however, applying mixup on textual data is a great challenge since linear interpolations on discrete inputs damage the semantic structure. some literature has explored the textual mixup mechanism like: @xcite propose to mix the hidden vector in the last few encoder layers; @xcite find a new way to combine two texts which can also be treated as a data augmentation strategy. in this paper, we do not make comparisons for the following reasons: (1) our embmix is simpler in practical use and there is little difference in the final performance of various methods according to @xcite . proposed method. since simply redesigning a robust loss function tends to have poor performance under a high noise ratio, we combine sample selection with the robust training methods. unlike the previous work that needs model ensembling or uses cross-validation, we train a single network with dropout to reduce confirmation bias in self-training. we make following improvements regarding to the characteristics of the textual data: (1) the decision boundaries in image-classification tasks are more clear. however, the main idea of the same text can vary under different contexts and sometimes there is even no absolute correct label. so we iteratively use the gaussian mixture model (gmm) to fit the loss distribution and use the predicted soft label to replace the label of the fusing data rather than setting a threshold and arbitrarily discarding the undesired samples at the beginning. (2) unlike the pixel input of visual data, the input of text is discrete. so for the separated data, we leverage a manifold mixup training strategy based on the aggregated representation from the plms. our method first uses gmm to select the samples that are more likely to be wrong and erase their original labels. then we leverage semi-supervised learning to jointly train the labeled set x (contains mostly clean samples) and an unlabeled set u (contains mostly noisy samples). we also introduce a manifold mixup strategy based on the hidden representation of the [cls] token named embmix. ## analysis and discussion to make a more comprehensive analysis of our proposed strategies, we offer fine-grained experiments and visualization to answer the following research questions (rqs): (1) can gmm actually distinguish the noisy samples on textual data? (2) how well can seflmix help prevent the model from overfitting the noisy labels? (3) selfmix utilizes more than one component. does each of them contribute to the final performance? (4) can selfmix be applied to other pre-trained models except bert? (5) noise and outliers both might have higher loss in early stages. while examples with noisy labels are useless or detrimental while training, how do we make sure with gmms we don't filter out outliers in this approach? answer 1: we demonstrate the loss distributions of the clean samples and noisy samples on imdb under 40% asymmetric noise in figure 3 (a-c ) and idn in figure 3 (e-g ). consistent with @xcite , the model tends to learn clean data during an early learning phase, and the 2-component gmm almost perfectly fits the loss distribution to distinguish the clean and noisy samples. during training, the loss output by selfmix is getting more polarized while the base model has already overfitted the wrong labels. notably, the cross-entropy loss values of different classes vary greatly under instance-dependent noise. and our proposed classregularization loss can help gmm better isolate these distributions in each class. answer 2: we record the test accuracy for every few mini-batches and show the learning process on ag-news (120k samples) and imdb (45k samples) answer 3: we remove each sub-method used in selfmix respectively and check the test accuracy to see whether each component of our proposed method contributes to the task ( answer 4: to verify the effectiveness of our proposed selfmix on other plms, we perform experiments on roberta. answer 5: 1).outlier refers to a data point that is significantly dissimilar to other data points or a point that does not imitate the expected typical behavior of the other points @xcite , which has some similarities with the concept of noisy samples. most noisy sample filtration methods are constructed based on the consumption or phenomenon that noisy samples behave differently from other data points during training. with the overlapped concept and the similar consumption/phenomenon in distinguishing noisy samples and outliers from other data points, many outlier detection methods resemble the noisy filtration ones @xcite , i.e., they view a point as an outlier/noisy sample if it is far away from its nearby neighbors in the representation space. as one of the most representative strategies in both noisy sample filtration and outlier detection, the conventional gmms used in this paper is difficult to distinguish precisely the outliers and noisy samples. 2).actually, excluding outliers along with the filtration of noisy samples from clean data may not be harmful. as mentioned by @xcite , these selected outliers (i.e., unlabeled examples) have high uncertainty and cannot provide much help to learners. @xcite also show that excluding outliers from the noisy training data significantly improves the performance of the centroid-based classifier. moreover, @xcite have made a comprehensive study on what impact outliers exactly bring to deep neural networks. for the tasks of image classification, outliers/hard samples are only helpful when training on easy-to-learn data. in this paper, the mixed data is challenging enough that it may not benefit from keeping these filtrated outliers in training. from another perspective, outliers in textual data appear to be inherently misleading or ambiguous examples located on the clustering boundary. the mixup strategy of this work can generate adequate samples around the boundary. 6 conclusions this paper presents selfmix to handle label noise on textual data. it uses the gaussian mixture model for sample selection and applies embmix for semisupervised learning. unlike the mutual distillation methods requiring co-training or model assembling, the proposed framework needs only a single model with dropout mechanism and utilizes two specific regularizations. extensive experiments conducted on three representative text classification datasets under different noise settings indicate that self-mix achieves a significant improvement over strong baselines. however, the proposed framework does not explicitly distinguish outliers and label noise. the future work includes exploring the different roles the noisy data and outliers play and applying our method to other supervised natural language tasks like named entity recognition.