STGN : an Implicit Regularization Method for Learning with Noisy Labels in Natural Language Processing (2022)

Abstract: Noisy labels are ubiquitous in natural language processing (NLP) tasks. Existing work, namely learning with noisy labels in NLP, is often limited to dedicated tasks or specific training procedures, making it hard to be widely used. To address this issue, SGD noise has been explored to provide a more general way to alleviate the effect of noisy labels by involving benign noise in the process of stochastic gradient descent. However, previous studies exert identical perturbation for all samples, which may cause overfitting on incorrect ones or optimizing correct ones inadequately. To facilitate this, we propose a novel stochastic tailor-made gradient noise (STGN), mitigating the effect of inherent label noise by introducing tailor-made benign noise for each sample. Specifically, we investigate multiple principles to precisely and stably discriminate correct samples from incorrect ones and thus apply different intensities of perturbation to them. A detailed theoretical analysis shows that STGN has good properties, beneficial for model generalization. Experiments on three different NLP tasks demonstrate the effectiveness and versatility of STGN. Also, STGN can boost existing robust training methods.

## introduction deep neural networks (dnns) have notably succeeded on various natural language processing (nlp) tasks. however, noisy labels induced by labeling errors are prevalent in a wide range of corpora @xcite . dnns can gradually fit correct data (i.e., noise-free data) and eventually memorize all data, including incorrect data (i.e., data with label noise) @xcite , which would affect the generalization of deep models and result in the performance degradation. hence, it is pressing and necessary to develop i.e., training accuracy from incorrect ones (%), smaller is better) and learning correct samples (right, i.e., training accuracy from correct ones (%), larger is better) during training on sst-5 with 40% label noise. two ratios together form the training accuracy (%). our method stgn gains more training accuracy on correct samples than existing method sln but memorizes much less label noise. to prevent overfitting on noisy labels, previous work mainly focuses on specific tasks @xcite or relies on additional data annotations @xcite , limiting its wide applications. for example, @xcite study noisy labels in entity linking, which depends on the definition of multiinstance learning. however, the framework is dedicated and hard to transfer to other tasks like text classification. as a general method, regularization can be applied to enhance generalization of the base model without excessive limitations. specifically, implicit regularization (e.g., learning rate, dropout), arising regularization effect from optimization, can guarantee dnns' generalization effectively even non-use of any explicit regularization @xcite (e.g., l1 norm regularization). among those sources of implicit regularization, the noise in stochastic gradient descent (sgd noise) is deemed as a crucial one. previous studies @xcite @xcite inject gaussian noise into different positions of the gradient in the backpropagation of dnns. however, they do not distinguish between method ∇θ ℓ(f, y) sln ∇ θ ℓ(f, y + σygy) σy > 0, @xmath0, gy ∼ n (0, ic×c) to address this issue, we investigate a benign noise called stochastic tailor-made gradient noise (stgn) to resist inherent label noise, where tailormade denotes devising diverse disturbing magnitude for correct and incorrect samples. specifically, we combine multiple guidelines to separate correct and incorrect samples precisely, followed by updating the perturbation of each sample iteratively in a self-supervised manner. in this way, stgn exerts large benign noise on incorrect samples to relieve overfitting and learns correct ones without perturbation to promote sufficient optimization. in fig. 1 , stgn gains more training accuracy on correct samples than sln (σ @xmath1.5) but memorizes much less label noise. besides, we further provide a detailed theoretical analysis, revealing the mechanism of stgn facilitating the model generalization. experimental results on sentiment analysis, named entity recognition (ner), and event relation reasoning demonstrate consistent gain, which verifies the effectiveness and generality of stgn. moreover, benign tailor-made noise can enhance the generalization of the model efficiently. ## experiments we aim to build robust nlp learning methods on noisy training sets to achieve well generalization performance on clean test sets. to this end, we experiment on three tasks (i.e., sentiment analysis, ner, and event relation reasoning) to test the effectiveness and generality of stgn. after that, we examine the theoretical analysis experimentally. we further explore the influence of hyperparameters and guidelines in stgn through an ablation study. due to the limited space, complete experimental settings are in appendix a.1. ## conclusion in this paper, stgn exerts different benign noise intensities on correct and incorrect samples, ensuring the goals of optimization and generalization are always the same to resist inherent label noise. we empirically validate the universality and effectiveness of stgn under synthetic label noise and real-world label noise, demonstrating that stgn can be applied to many nlp tasks without dedicated framework or excessive clean corpus.