Adaptive Textual Label Noise Learning based on Pre-trained Models (2023)

Abstract: The label noise in real-world scenarios is unpredictable and can even be a mixture of different types of noise. To meet this challenge, we develop an adaptive textual label noise learning framework based on pre-trained models, which consists of an adaptive warm-up stage and a hybrid training stage. Specifically, an early stopping method, relying solely on the training set, is designed to dynamically terminate the warm-up process based on the model’s fit level to different noise scenarios. The hybrid training stage incorporates several generalization strategies to gradually correct mislabeled instances, thereby making better use of noisy data. Experiments on multiple datasets demonstrate that our approach performs comparably or even surpasses the state-of-the-art methods in various noise scenarios, including scenarios with the mixture of multiple types of noise.

## introduction in recent years, deep neural networks (dnns) have been successfully applied in many fields @xcite @xcite and the performance largely depends on well-labeled data. however, accessing large-scale datasets with expert annotation in the real world is difficult due to the significant time and labor costs involved. instead, the noisy data obtained directly from the real world is often utilized in practical scenarios, even though it inevitably contains some incorrect labels. thus, research on learning with noisy labels has gained attention in various fields such as natural language processing (nlp) @xcite @xcite . there are two main types of label noise in nlp: class-conditional noise (ccn) and instancedependent noise (idn). ccn assumes that label noise is dependent on the true class, which can simulate the confusion between similar classes like * corresponding author "game" and "entertainment". on the other hand, idn assumes that label noise is dependent on the instance, which simulates the confusion caused by the characteristics of the instance. for example, a piece of news containing the phrase "played on a pitch as slow as a bank queue" may be misclassified as economic news instead of sports news due to the specific wording. however, most studies focus on a particular type of noise. for example, @xcite introduces a non-linear processing layer to learn the noise transition matrix of ccn. @xcite designs the class-regularization loss according to the characteristic of idn. however, there is a premise for applying these methods, which is that the noise is known and of a single type. in the real-world, the noise scenarios are more complex and involve a mixture of multiple noises arising from various factors, such as data ambiguity, collection errors, or annotator inexperience. methods that specifically target one type of noise are less effective when dealing with other types of noise. this limitation hinders their applicability in real scenarios where the noise is unknown and variable. to address the challenges posed by real noise scenarios, we develop an adaptive textual label noise learning framework based on pre-trained models. this framework can handle various noise scenarios well, including different types of noise and mixed noise types. specifically, our approach begins with an adaptive warm-up stage, then divides the data into clean and noisy sets by the correctness statistic of samples, and utilizes different generalization strategies on them. in particular, there are three key designs in our approach. first, the warm-up stage is designed to automatically stop early based on the model's fit level to the noise scenario, which effectively prevents the model overfitting erroneous labels especially under idn scenarios or with a high ratio of noise. no-tably, the adaptive warm-up method relies solely on the raw training set, rather than a clean validation set, making it more suitable for practical scenarios. second, unlike previous works @xcite that fit gmm @xcite on the training losses to separate data, the data is separated according to the correctness statistic of each sample. the correctness statistic is accumulated by assessing the consistency between the model's predictions and the given labels during the warm-up stage. this prolonged observation provides more accurate grounds for data partitioning. third, a linear decay fusion strategy is designed to gradually correct the potential wrong labels to generate more accurate pseudo-labels by adjusting the fusion weights of the original labels and the model outputs. we conduct extensive experiments on four classification datasets, considering different types of noise: class-conditional noise, instance-dependent noise, and a mixture of multiple noises. to the best of our knowledge, previous methods have not explored such mixed noise. the experimental results demonstrate that our method surpasses existing general methods and approaches the performance of methods specifically designed for particular noise types in different noise settings. our contributions can be concluded as follows: universal label noise learning. label noise learning methods can be divided into two groups: loss correction and sample selection methods @xcite . loss correction tries to reduce the effect of noisy labels during training by adding regularization item in loss, designing robust network structure for noisy label and so on. for example, @xcite adds a reverse cross-entropy term to the traditional loss to reduce the disturbance brought by noise. elr @xcite adds a regularization term prevent the model from memorizing the noisy labels because it would not be fitted in the early training stage. sample selection divides the data into clean and noisy subsets, and uses different methods for different subsets. for example, co-teaching @xcite ) maintains two networks. during training, the two networks respectively pick out some small-loss samples as clean data for each other to learn. dividemix @xcite uses gaussian mixture model to separate clean and noisy samples. the noisy samples are treated as unlabeled data, whose pseudo-labels are generated by the model. finally, mixmatch @xcite ) method is adopted for mixed training on clean set and noisy set. ## analysis in this section, some experiments are designed to make a more comprehensive analysis of our proposed method. the main results are shown in ## conclusion based on the unknown and complex nature of noise, we propose a noise learning framework based on pre-trained models that can adapt to various textual noise scenarios. this framework automatically stops the warm-up process based on the magnitude and complexity of the noise to prevent the model from overfitting noisy labels. to further leverage mislabeled data, the linear decay fusion strategy is combined with mixup and r-drop to improve performance while maintaining stability. experimen-tal results demonstrate that our method achieves performance comparable to state-of-the-art in all settings within common noise range.