Learning to Detect Noisy Labels Using Model-Based Features (2022)

Abstract: Label noise is ubiquitous in various machine learning scenarios such as self-labeling with model predictions and erroneous data annotation. Many existing approaches are based on heuristics such as sample losses, which might not be flexible enough to achieve optimal solutions. Meta learning based methods address this issue by learning a data selection function, but can be hard to optimize. In light of these pros and cons, we propose SENT (Selection-Enhanced Noisy label Training) that does not rely on meta learning while having the flexibility of being data-driven. SENT transfers the noise distribution to a clean set and trains a model to distinguish noisy labels from clean ones using model-based features. Empirically, on a wide range of tasks including text classification and speech recognition, SENT improves performance over strong baselines under the settings of self-training and label corruption.

## introduction state-of-the-art deep neural networks require large amounts of annotated training data. though the success of large pre-training models @xcite alleviates such requirements, high-quality labeled data are still crucial to obtain the best performance on downstream tasks. however, it is extremely expensive to acquire large-scale annotated data for every new task. to overcome the challenge of rigorous data requirements, recent works utilize weak labels for supervision, including heuristic rules @xcite @xcite , large-scale datasets with cheap but noisy labels @xcite ;, and self-training @xcite . in selftraining, one trains a model on a labeled dataset and then predicts on a large amount of unlabeled data. then the clean labeled data and pseudo-labeled data are combined to further train the model. the aforementioned sources of supervision share two important characteristics: first, they are learning from noisy labels. second, the noise is dependent on data features. thus, they could be unified into the framework of noisy label learning, for which numerous approaches have been proposed to reduce the negative impact of noise. however, there are three problems in prior work on noisy label learning: first, many existing approaches are based on heuristics such as sample losses which are not flexible enough @xcite @xcite ; second, many previous works require prior knowledge of the noise distribution of the dataset to adjust the hyperparameters, which is often not available in real-world applications @xcite . third, meta learning based methods avoid previous problems but suffer from optimization difficulties @xcite such as longer training time, heavy hyper-parameter tuning and an unstable convergence process. to address the above problems, we propose a simple data-driven approach which does not rely on meta learning while being flexible. our contributions are summarized as follows: ## method now we present our approach, sent (selection enhanced noisy label training), that learns to select a subset from a noisy dataset and only uses the selected subset for training to reduce label noise. the core idea is to transfer the noise distribution so that both clean and noisy labels are available on a data subset. a selection model is trained to distinguish clean labels from noisy ones and then applied to selecting a clean subset from a noisy dataset. formally, given a noisy training dataset d @xmath0, y i )|1 ≤ @xmath1, our approach is to learn a selection strategy f to select a clean subset d ′ @xmath2, @xmath3, 2, ..., n } for training. here (x i , y i ) is a training sample, y i is the noisy label, y * i is the corresponding unknown true label, and n is the size of training set. let model m be our main model which is trained on the noisy dataset d train and performs certain tasks such as text classification, speech recognition and others. we will also have a selection model s trained on a small clean development dataset d select . the task of s is to learn the selection strategy f . there are two main stages in our approach: noise transfer and selection learning. ## conclusions in this paper, we propose sent to address the problem of label noises. compared with meta learning based models, our selection model is trained with full supervision using cross entropy loss which facilitates the convergence process. in the meantime, we model idn noise without the prior knowledge of noise distribution. we also unify the setting of self-training and label corruption in the framework of noisy label learning and conduct extensive experiments on both settings. besides, the selection model in our framework is feature-based. these features are very informative but might be limited in terms of expressivity. this reserves the space for further improvement to learn more data-driven features under our framework.