Model-agnostic Methods for Text Classification with Inherent Noise (2020)

Abstract: Text classification is a fundamental problem, and recently, deep neural networks (DNN) have shown promising results in many natural language tasks. However, their human-level performance relies on high-quality annotations, which are time-consuming and expensive to collect. As we move towards large inexpensive datasets, the inherent label noise degrades the generalization of DNN. While most machine learning literature focuses on building complex networks to handle noise, in this work, we evaluate model-agnostic methods to handle inherent noise in large scale text classification that can be easily incorporated into existing machine learning workflows with minimal interruption. Specifically, we conduct a point-by-point comparative study between several noise-robust methods on three datasets encompassing three popular classification models. To our knowledge, this is the first time such a comprehensive study in text classification encircling popular models and model-agnostic loss methods has been conducted. In this study, we describe our learning and demonstrate the application of our approach, which outperformed baselines by up to 10 % in classification accuracy while requiring no network modifications.

## introduction text classification is a fundamental problem in natural language processing, where the objective is to categorize text into a set of predefined classes. it has been shown to be valuable in many domains, such as social media @xcite , cognitive-biometric recognition @xcite ) and e-commerce @xcite . modern-day enterprises are heavily dependent on the performance of text classification models, where even a marginal improvement in the performance can accrue billions of dollars @xcite and substantially improve the customer experience. currently, dnn @xcite are the state of the art machine learning models widely deployed for text classification tasks in major enterprises @xcite @xcite . like any other supervised classifiers, the performance of these dnn trained using standard cross-entropy loss is strongly dependent on the quality and quantity of the data. however, collecting high-quality manual labels is time-consuming and expensive. at the same time, there are less expensive sources to collect labeled data, such as mechanical turk @xcite , search engine meta data, and social media tags. these inexpensive large datasets have a high level of noise, as multiple annotators generate the labels under different skill-set and biases. in e-commerce, an example of one such confusing case is when the same product title is labeled differently by agents into separate but related categories, as shown in fig. 1 . blindly trusting these large inexpensive datasets as gold-standard can decrease the performance of models. learning from noisy labels is an active area of research in computer vision, and several model cognizant approaches @xcite have been proposed. however, these approaches work on building complex network architecture to handle noise and require substantial back-ground knowledge and training to operate. for many enterprises, the performance of text classification models plays a crucial role in their revenue earnings, and the difficulty of implementing complex architecture becomes a bottleneck. conversely, there is minimal research studying the performance of the model-agnostic methods to handle inherent label noise, that can be easily incorporated into existing machine learning workflows with no network modifications for large scale text classification tasks. under model agnostic schemes, there are several different lines of work which include modeling noise-transition matrix @xcite @xcite , training auxiliary network @xcite , training with clean labels @xcite , label regularization @xcite , data augmentation @xcite , and noise-robust loss functions. in this work, we focus our attention on techniques that do not add any overhead computation. specifically, we evaluate label smoothing regularization, data augmentation technique, and state of the art noise-robust loss functions @xcite @xcite to examine its effect in mitigating inherent label noise for large scale text datasets. these methods are simpler and easy to implement than other lines-of-work in tackling noisy labels, which either gets very complex @xcite or has strong assumptions on the type of noise present @xcite . we conduct our study on large web-scale text data scraped from popular e-commerce platforms, which contains a significant number of classes leading to a higher inherent noise due to annotator confusion. in contrast with previous work @xcite , we do not introduce any external noise into our dataset. to the best of our knowledge, no previous study has been done to study model-agnostic methods in mitigating inherent label noise for large scale text classification. to summarize, the main contributions of our work are as follows: ## related work in this section, we provide a brief literature review for model agnostic methods popularly used in machine learning to handle noise. these include modeling label noise, training auxiliary network, data augmentation, noise-robust loss functions, and regularization schemes. existing literature in modeling label noise can be further subdivided into two groups: class-conditional and instance-conditional label noise. the first group assumes that the noise is independent of the instance and models the transition probability from true class to noisy class. @xcite assumed the class-conditional label noise for binary classification task and consequently use an em-based algorithm to learn the model parameters and the noise transition matrix. @xcite extended the multi-class counterpart of class-conditional noise and proposed a constrained linear layer at the top of the softmax layer, which under some strong assumptions can be interpreted as the noise transition matrix. a similar work by @xcite uses forward and backward methods to explicitly model the noise transition matrix and also provide a way to estimate the noise transition matrix. the second group assumes that the label noise is conditioned for each instance. @xcite developed a noise model, where noise is modeled on the instance and its class. similarly, @xcite model the noise through conditional random fields (crf), where the clean labels are modeled as latent variables during training. under training auxiliary network, @xcite proposed training two different networks which back-propagate the loss when the predictions of the two network disagree. mentor network @xcite , another popular method, learns a sample weighting scheme to supervise the training of a base network, termed studentnet, that learns under label noise contingencies. similarly, @xcite present an unsupervised approach to learn the curriculum based on the complexity of the instance in the feature space. supporting the loss function category, @xcite presented robust surrogate loss functions for handling noisy labels in a binary classification task. mean absolute error (mae) @xcite was shown to be inherently robust to label noise for the classification task. similarly bootstrapping loss function @xcite was proposed, which introduced a weighted combination of target labels and network predictions to compensate for noisy samples. while @xcite uses a fixed hyper-parameter as weights, d2l @xcite proposes to use the subspace complexity score of the model as weights which gets updated at every iteration. to overcome the limitation of mae, generalized cross entropy @xcite was proposed, which is a combination of mae and categorical cross entropy (cce) loss. symmetric cross-entropy @xcite augments the standard cce, similar to symmetric kl-divergence, with the noise robust reverse cross-entropy. data augmentation and regularization schemes are some other ways introduced to make the learning procedure robust to noisy labels. these include, mixup @xcite , that uses convex combinations of training data points and its corresponding labels, and label smoothing regularization (lsr) @xcite , where a smoothing parameter is used to modify the hard one-hot labels into soft labels to mitigate over-fitting to noisy labels. ## experiments and results in this section, we evaluate several model-agnostic approaches discussed above on three large scale datasets shown in in this section, we provide a brief discussion of the models used in our comparative study. ffnn: in this work, we use ffnn @xcite with average pooling operation @xcite on input feature with two hidden layers having 1024 and 512 units respectively. we employ a relu activation function for non linearity with 0.2 dropout followed by a output layer of k output values, where k is the number of classes. cnn: 1d cnn @xcite and fixed the maximum length of sentence to 10 and embedding dimension 128. in our network architecture, we use one convolutional layer having 128 filters with a convolution window/kernel size of 5 followed by max-pooling and finally a fully connected layer with 512 neurons. lstm: for lstm @xcite , we use the same sentence length and embedding size as cnn. in our network architecture, the first layer of lstm is the embedding layer followed by variational dropout. the next layer is the lstm layer, with 256 memory units, followed by the output layer of k output values. more recently, researchers have started to apply graph convolutional networks @xcite ) for text classification. preliminary results are encouraging; however, they bring in the additional complexity of the graph. in this work, we restrict ourselves to more popular techniques, i.e., ffnn, cnn, lstm. ## conclusion in this study, we demonstrate the effectiveness of model-agnostic methods in advancing the performance of machine learning models for large scale text classifications. while most of the machine learning literature focused on building complex networks to handle noise, very few works have studied the performance of simpler methods that can give a significant impact. to the best of our knowledge, this is the first attempt to apply model agnostic methods requiring no network modifications to handle inherent noise for text classification datasets. we fill the gap in existing literature, where applying these methods to large scale text classification tasks is not the norm. although we have shown improvements for data scraped from e-commerce platforms, the methods mentioned above can be applied to any large text classification task. the methods mentioned are easy to implement and can be easily integrated into any machine learning workflows without breaking the existing code-base. in contrast to previous works, we did not add noise and hypothesize that large dataset having thousands of samples and hundreds of unique classes can inadvertently introduce noise. moreover, this paper serves as a brief literature review of model-agnostic methods that can be applied to text classification and other related domains, requiring no network modifications and minimal computation overhead.