Noisy Label Regularisation for Textual Regression (2022)

Abstract: Training with noisy labelled data is known to be detrimental to model performance, especially for high-capacity neural network models in low-resource domains. Our experiments suggest that standard regularisation strategies, such as weight decay and dropout, are ineffective in the face of noisy labels. We propose a simple noisy label detection method that prevents error propagation from the input layer. The approach is based on the observation that the projection of noisy labels is learned through memorisation at advanced stages of learning, and that the Pearson correlation is sensitive to outliers. Extensive experiments over real-world human-disagreement annotations as well as randomly-corrupted and data-augmented labels, across various tasks and domains, demonstrate that our method is effective, regularising noisy labels and improving generalisation performance.

## introduction modern deep neural networks (dnns) have millions or billions of trainable parameters, far more than the number of examples they are trained on. to avoid over-fitting, they are heavily reliant on large-scale training, including data derived through methods such as self supervision, data augmentation, and self labelling @xcite @xcite . however, such methods inevitably introduce noise, through biased data, unnatural inputs, or incorrect labels. weak perturbations applied to inputs can improve model performance by forcing dnns to learn noise-invariant latent representations @xcite . but training with noisy labels has been shown to be detrimental to generalisation performance across tasks including image classification @xcite , dialogue generation @xcite , and entity-relation extraction @xcite . dnns fit noisy labels by "memorising" each ex-ample -over-fitting corrupted training sets, and yielding poor generalisation @xcite . given this background, our focus in this paper is on how to alleviate memorisation and improve generalisation when training with noisy labels. previous related work has proposed three directions: (1) regularisation techniques @xcite ; (2) augmenting the loss function with an explicit representation of the distribution of noise @xcite ; and (3) explicit detection of noisy labels @xcite @xcite . however, the vast majority of this work has focused on classification tasks, and there has been very little work in the context of regression tasks and low-resource domains. in this work, we fill the gap by targeting noisy label regularisation for text regression, in the form of semantic text similarity (sts), sentiment analysis, and machine translation quality assessment. our work makes three contributions: (1) empirical clarification of the role of explicit regularisation in noisy label training in a regression setting; (2) proposal of an effective noisy label detection method for continuous labels; and ## task and datasets in this paper, we investigate text regression across three separate tasks, and a total of 10 datasets. ## case study we first examine the susceptibility of dnns to overfit random labels @xcite , based on the clinical n2c2-sts data set using bert @xcite . then we conduct ablation experiments using various regularisation techniques, to observe whether they can reduce the degradation caused by noisy labels. corrupted training set: to generate partiallynoisy training data, we corrupt training set d by randomly selecting m instances and replacing their labels with @xmath0, 5] sampled from a uniform distribution, forming noisy subset d noisy , leaving the clean partition d clean . thus the corrupted training set is d ′ = d noisy ∪ d clean (where experimental setup: we randomly split the 1,642 instances in the n2c2-sts training set into 1,242 and 400 instances, as training set d and a validation set. @xmath1.2, 0.4, 1.0} to generate three corrupted training sets, denoted corrupt2, corrupt4, and corrupt10, respectively, with corresponding clean partitions denoted as clean2, clean4 and ⊘ below. note that the same validation set is used for all experiments in this section and section 6.1. the model is trained on bert-base-uncased, optimising with a linear scheduler with warmup @xmath2.1, train batch @xmath3, learning @xmath4, and training @xmath5. ## noisy label detection we propose a two-step method to identify noisy labels from training data based on iterative predictions, followed by three different strategies for training with noisy examples, namely: (1) dis: discard noisy examples; (2) rep: repair noisy labels with pseudo labels; and (3) res: resample the same number of instances from the "clean" set, to make up for discarded noisy examples. ## experiments we first evaluate the noise detection method on two synthetically-corrupted versions of n2c2-sts ("n2c2") (corrupt2 and corrupt4), where we have perfect knowledge of the noisy and clean subset, and then apply our methods to real-world sts, sa, and da datasets, where noisy labels are unknown. ## conclusion regularisation strategies improve model generalisation performance in a range of contexts, but are not able to effectively address generalisation degradation caused by training with noisy labels. in this paper, we have proposed a noisy label training method for text regression tasks, based on identifying noise through iterative prediction and targeted evaluation criteria, followed by discarding or repairing of noisy labels. extensive experiments on three rating tasks demonstrate the effectiveness of our approach.