Noise Learning for Text Classification: A Benchmark (2022)

Abstract: Noise Learning is important in the task of text classification which depends on massive labeled data that could be error-prone. However, we find that noise learning in text classification is relatively underdeveloped: 1. many methods that have been proven effective in the image domain are not explored in text classification, 2. it is difficult to conduct a fair comparison between previous studies as they do experiments in different noise settings. In this work, we adapt four state-of-the-art methods of noise learning from the image domain to text classification. Moreover, we conduct comprehensive experiments on our benchmark of noise learning with seven commonly-used methods, four datasets, and five noise modes. Additionally, most previous works are based on an implicit hypothesis that the commonly-used datasets such as TREC, Ag-News and Chnsenticorp contain no errors. However, these datasets indeed contain 0.61% to 15.77% noise labels which we define as intrinsic noise that can cause inaccurate evaluation. Therefore, we build a new dataset Golden-Chnsenticorp( G-Chnsenticorp) without intrinsic noise to more accurately compare the effects of different noise learning methods. To the best of our knowledge, this is the first benchmark of noise learning for text classification.

## introduction the fast development of text classification cannot be achieved without massive labeled data resources, especially for supervised embedding-based methods. however, not all training data are correctly labeled in practice @xcite . these incorrectly labeled data are called noisy labels. to alleviate the interference caused by noisy labels, many noise learning methods have been proposed @xcite @xcite @xcite @xcite . although both cv and nlp domains have serious label noise problems, these work are mainly focused on the cv domain, only @xcite is dedicated to nlp domain. so in order to support the development of noise learning in nlp, we would like to propose a noise learning benchmark in the field of text classification. we find that the previous studies in noise learning for text classification tasks have two weaknesses. 1. the implicit hypothesis is unreasonable. 2. they lack horizontal comparison. ## related work plenty of previous studies have examined the factors that impact label noise learning models. @xcite prove that a model of sufficient complexity can over-fit any noise. @xcite analyze convergence and generalization in neural networks from the perspective of gaussian processes in the infinite-width limit. @xcite propose a model that is extremely adaptive to specific patterns of artificial noise. @xcite prove that gradient descent with early stopping is robust to label noise for overparameterized neural networks. from the perspective of noise mode, algan and ulusoy (2020) conduct a detailed analysis of the influence of label noise on model training and propose a generic framework to generate feature-dependent label noise. hataya and nakayama (2018) investigate the behavior of convolutional neural networks (cnns) under class-dependently simulated label noise. flatow and penner (2017) test the robustness of the model by randomly permuting the labels of the training set with increasing frequency.jiang et al. (2020) establish the first benchmark of controlled real-world label noise in the cv field. following the work of han et al. (2020) which divides noise learning methods into three categories optimization-based method, objective-based method, and data-based method for a more comprehensive comparative analysis, we select several commonly-used noise learning methods from each category to conduct comparison experiments. there exist a few benchmarks @xcite in the field of image classification, but there is no benchmark in the field of text classification. therefore, many works in text classification task @xcite only do comparative experiments with their own baseline. moreover, very few studies consider different noise settings. a robust benchmark is much needed in the development of the field of text classification. ## g-chnsenticorp database ambiguity level. we then define three categories of intrinsic noise based on the aforementioned annotator agreement thresholds: fatal noise [90%, 100%], inexact noise [60%, 90%), and ambiguous noise [0%, 60%). specifically, when more than 90% of the annotators agree upon a label different from the original one, we consider the sample as fatal noise. when 60% to 90% of annotators agree on a label different the original, we consider the sample as inexact noise. when less than 60% of annotators agree on the label, no matter what label it is we consider it as ambiguous. ratios of the three intrinsic noises in trec, ag-news and chnsenticorp are summarized in we investigate three common datasets for text classification illustrated as follows: with regard to the chnsenticorp dataset, we recruited a team of 10 experts in hotel management as annotators to label samples from chnsenticorp and found that there existed four categories in the dataset: positive, negative, irrelevant, and neutral. we remove irrelevant and neutral samples to construct a new binary dataset golden-chnsenticorp (g-chnsenticorp) since we believe those samples are ambiguous in the task of text classification. based on the three types of intrinsic noise, we reconstruct the g-chnsenticorp dataset as follows: we correct samples of fatal noise with the annotator majority label and remove samples of inexact or ambiguous noise. note that since g-chnsenticorp is a simple binary text sentiment classification dataset, we are confident to correct the fatal noise labels with our expert majority label and remove the other two types of noise. ambiguous noise samples such as "good breakfast but bad bed" are indeed noise to the binary classification dataset and challenging samples such as sarcastic reviews are not classified as ambiguous or inexact noise during annotating. the new dataset can ensure the robustness of the models trained on it. through our annotating and modification as mentioned above, we obtain the final g-chnsenticorp dataset which contains around 4,000 training samples. ## noise generation methods in single-label text classification tasks, we assign a corresponding label to each sentence. for all n samples with k different types of classes, we let @xmath0, y 1 ), (x 2 , y 2 ), . . . , (x n , y n )} as noise-free dataset, where x i denotes the ith sentence in the dataset, y @xmath1, . . . , k} denotes the class of the ith sentence. however, it is hard for us to find truly noise-free data in the dataset except for manual verification. therefore, we first assume all pre-given data are true. then, we use noise transfer matrix to automatically generate the corresponding noise-labeled dataset. here, we set here, i represents the identity matrix. a denotes an allones matrix with zeros along the diagonal. b represents the identity matrix where the last column is transferred to the first column. c represents the matrix with zeros along the diagonal, and except for the diagonal, the values are uniformly and independently distributed. d is a matrix independent of the k -1 dimensional unit simplex with zeros along the diagonal. generally speaking, noise labels can be categorized into four types according to different noise transfer matrices. as shown in ## method common noise learning methods can be divided into three categories: optimization-based method, objective-based method and data-based method @xcite . ## results and discussion in our experiments, we evaluate the performance of the models in ## conclusion in conclusion, we firstly construct a text classification dataset without intrinsic noise, then do experiments on these datasets using some sota noise learining methods, and finally draw some useful conclusions about noise learning: intrinsic noise is more difficult to be learned than other noise modes; many methods do not work well when the noise ratio is higher than 30%; a small amount of white noise can benefit classification methods. this is the first time a benchmark of noise learning for text classification has been established. we construct a dataset without intrinsic noise for more accurate evaluations in the noise learning. we present this benchmark to summarize and compare the contributions and weaknesses of previous work in noise learing, to make up for their lack in intrinsic noise analysis, and hopefully to provide a reference for future research in the field of noise learning. in future work, we will build more data without intrinsic noise and conduct more in-depth analysis of intrinsic noise in other settings. of course, we will also research a better noise learing method based on the experimental findings of this paper.