Learning from Noisy Labels for Entity-Centric Information Extraction (2021)

Abstract: Recent information extraction approaches have relied on training deep neural models. However, such models can easily overfit noisy labels and suffer from performance degradation. While it is very costly to filter noisy labels in large learning resources, recent studies show that such labels take more training steps to be memorized and are more frequently forgotten than clean labels, therefore are identifiable in training. Motivated by such properties, we propose a simple co-regularization framework for entity-centric information extraction, which consists of several neural models with identical structures but different parameter initialization. These models are jointly optimized with the task-specific losses and are regularized to generate similar predictions based on an agreement loss, which prevents overfitting on noisy labels. Extensive experiments on two widely used but noisy benchmarks for information extraction, TACRED and CoNLL03, demonstrate the effectiveness of our framework. We release our code to the community for future research.

## introduction deep neural models have achieved significant success on various information extraction (ie) tasks. however, when training labels contain noise, deep neural models can easily overfit the noisy labels, leading to severe performance degradation @xcite . unfortunately, labeling on large corpora, regardless of using human annotation @xcite or automated heuristics @xcite , inevitably suffers from labeling errors. this problem has even drastically affected widely used benchmarks, such as conll03 @xcite and ta-cred @xcite , where a notable portion of incorrect labels have been caused in annotation and largely hindered the performance of sota 1 our code is publically available at https://github. com/wzhouad/nll-@xmath0 figure 1 : illustration of our co-regularization framework. the base models are jointly optimized with the task-specific loss from label y and an agreement loss, which regularizes the models to generate similar predictions to the aggregated soft target probability q. so far, few research efforts have been made to developing noise-robust ie models, and existing work mainly focuses on the weakly supervised or distantly supervised setting @xcite @xcite . most of such methods typically depend on multi-instance learning that relies on bags of instances provided by distant supervision @xcite @xcite or require an additional clean and sufficiently large reference dataset to develop a noise filtering model @xcite . accordingly, those methods may not be generally adapted to supervised training settings, where the aforementioned auxiliary learning resources are not always available. particularly, crossweigh @xcite ) is a representative work that denoises a natural language dataset without using extra learning resources. this method trains multiple independent models on different partitions of training data and downweighs instances on which the models disagree. though effective, a method of this kind requires training tens of redundant neural models, leading to excessive computational overhead for large models. as far as we know, the problem of noisy labels in supervised learning for ie tasks has not been well investigated. in this paper, we aim to develop a general denoising framework that can easily incorporate existing supervised learning models for entity-centric ie tasks. our method is motivated by studies @xcite showing that noisy labels often have delayed learning curves, as incorrectly labeled instances are more likely to contradict the inductive bias captured by the model. hence, noisy label instances take a longer time to be picked up by neural models and are frequently forgotten in later epochs. therefore, predictions by more than one model tend to disagree on such instances. accordingly, we propose a simple yet effective co-regularization framework to handle noisy training labels, as illustrated in fig. 1 . our framework consists of two or more neural classifiers with identical structures but different initialization. in training, all classifiers are optimized on the training data with the task-specific loss and jointly regularized with regard to an agreement loss that is defined as the kullback-leibler (kl) divergence among predicted probability distributions. then for instances where a classifier's predictions disagree with labels, the agreement loss encourages the classifier to give similar predictions to the other classifier(s) instead of the actual (possibly noisy) labels. in this way, the framework prevents the incorporated classifiers from overfitting noisy labels. we apply the framework to two important entitycentric ie tasks, named entity recognition (ner) and relation extraction (re). we conduct extensive experiments on two prevalent but noisy benchmarks, conll03 for ner and tacred for re, and apply the proposed learning frameworks to train various models from prior studies for these two tasks. the results demonstrate the effectiveness of our method in noise-robust training, leading to promising and consistent performance improvement. we present contributions as follows: ## method in this paper, we focus on developing a noiserobust learning framework that improves supervised models for entity-centric ie tasks. in such tasks, (noisy) labels can be assigned to either individual tokens (ner) or pairs of entities (re) in natural language text. specifically, our objective is to learn a noise-robust model f with the presence of such noisily labeled instances from d without using external resources such as a clean development dataset @xcite . ## tasks we evaluate our framework on two fundamental entity-centric ie tasks, namely re and ner. our framework can incorporate any kind of neural model that is dedicated to either task. particularly, in this paper, we adopt off-the-shelf sota models that are mainly based on transformers. this section introduces the two attempted tasks and the design of task-specific models. relation extraction. re aims at identifying the relations between a pair of entities in a piece of text from the given vocabulary of relations. specifically, given a sentence x and two entities e s and e o , identified as the subject and object entities respectively, the goal is to predict the relation between e s and e o . following @xcite , we formulate this task as a sentence classification problem. accordingly, we first apply the entity masking technique @xcite to the input sentence and replace the subject and object entities with their named entity types. for example, a short sentence "bill gates founded microsoft" will become "[subject-person] founded [object-organization]" after entity masking. we then feed the sentence to the pre-trained language model and use a softmax classifier on the representation of the [cls] token to predict the relation. ## experiment in this section, we evaluate the proposed learning framework based on two (noisy) benchmark datasets for the two entity-centric ie tasks ( §4.1- §4.4). in addition, a noise filtering analysis is presented to show how our framework prevents an incorporated neural model from overfitting noisy training data ( §4.5), along with a detailed ablation study about configurations with varied model copies, alternative noise filtering strategies, target functions, and different noise rates ( §4.6). ## related work we discuss two lines of related work. each has a large body of work which we can only provide as a highly selected summary. ## conclusion this paper presents a co-regularization framework for learning supervised ie models from noisy data. this framework consists of two or more identically structured models with different initialization, which are encouraged to give similar predictions on the same inputs by optimizing an agreement loss. on noisy examples where model predictions usually differ from the labels, the agreement loss prevents the model from overfitting noisy labels. experiments on ner and re benchmarks show that our framework yields promising improvements on various ie models. for future work, we plan to extend the use of the proposed framework to other tasks such as event-centric ie @xcite and co-reference resolution @xcite .