Improving Distantly Supervised Relation Extraction with Self-Ensemble Noise Filtering (2021)

Abstract: Distantly supervised models are very popular for relation extraction since we can obtain a large amount of training data using the distant supervision method without human annotation. In distant supervision, a sentence is considered as a source of a tuple if the sentence contains both entities of the tuple. However, this condition is too permissive and does not guarantee the presence of relevant relation-specific information in the sentence. As such, distantly supervised training data contains much noise which adversely affects the performance of the models. In this paper, we propose a self-ensemble filtering mechanism to filter out the noisy samples during the training process. We evaluate our proposed framework on the New York Times dataset which is obtained via distant supervision. Our experiments with multiple state-of-the-art neural relation extraction models show that our proposed filtering mechanism improves the robustness of the models and increases their F1 scores.

## introduction the task of relation extraction is about finding relation or no relation between two entities. this is an important task to fill the gaps of existing knowledge bases (kb). open information extraction (openie) @xcite is one way of extracting relations from text. they consider the verb in a sentence as the relation and then find the noun phrases located to the left and right of that verb as the entities. but this process has two serious problems: first, the same relation can appear in the text with many verb forms and openie treats them as different relations. this leads to the duplication of relations in kb. second, openie treats any verbs in a sentence as a relation which can generate a large number of insignificant tuples which cannot be added to a kb. supervised relation extraction models, on the other hand, do not have these problems. but they require a large amount of annotated data which is difficult to get. @xcite , @xcite , @xcite used the idea of distant supervision to automatically obtain the training data to overcome this problem. the idea of distant supervision is that if a sentence contains both the entities of a tuple, it is chosen as a source sentence of this tuple. although this process can generate some noisy training instances, it can give a significant amount of training data which can be used to build supervised models for this task. they map the tuples from existing kbs such as freebase @xcite to the text corpus such as wikipedia articles @xcite or new york times articles @xcite . based on distantly supervised training data, researchers have proposed many state-of-the-art models for relation extraction. @xcite , @xcite , @xcite proposed feature-based learning models and used entity tokens and their nearby tokens, their part-ofspeech tags, and other linguistic features to train their models. recently, many neural network-based models have been proposed to avoid feature engineering. @xcite and @xcite used convolutional neural networks (cnn) with max-pooling to find the relation between two given entities. @xcite , @xcite , @xcite used attention framework in their neural models for this task. but the distantly supervised data may contain many noisy samples. sometimes sentences may contain the two entities of a positive tuple, but they may not contain the relation specific information. these kinds of sentences and entity pairs are considered as positive noisy samples. another set of noisy samples comes from the way samples for we include examples of clean and noisy samples generated using distant supervision in the presence of the noisy samples in the distantly supervised data adversely affects the performance of the models. our goal is to remove the noisy samples from the training process to make the models more robust for this task. we propose a selfensemble based noisy samples filtering method for this purpose. our framework identifies the noisy samples during the training and removes them from training data in the following iterations. this framework can be used with any supervised relation extraction model. we run experiments with several state-of-the-art neural models, namely convolutional neural network (cnn) @xcite , piecewise convolutional neural network (pcnn) @xcite , entity attention (ea) @xcite , and bi-gru word attention (bgwa) @xcite with the distantly supervised new york times dataset @xcite . our framework improves the f1 score of these models by 2.1%, 1.1%, 2.1%, and 2.3% respectively foot_0 . ## task description sentence-level relation extraction is defined as follows: given a sentence s and two entities {e 1 , e 2 } marked in the sentence, find the relation r(e 1 , e 2 ) between these two entities in s from a pre-defined set of relations r ∪ {none}. r is the set of positive relations and none indicates that none of the relations in r holds between the two marked entities in the sentence. the relation between the entities is argument orderspecific, i.e., r(e 1 , e 2 ) and r(e 2 , e 1 ) are not the same. the input to the system is a sentence s and two entities e 1 and e 2 , and output is the relation r(e 1 , e 2 ) ∈ r ∪ {none}. distant supervised datasets are used for training relation extraction models. but the presence of noisy samples negatively affects their performance. in this work, we try to identify these noisy samples during training and filter them out from the subsequent training process to improve the performance of the models. ## self-ensemble filtering framework figure 1 shows our proposed self-ensemble filtering framework. this framework is inspired from the work by @xcite . we start with clean and noisy samples and assume that all samples are clean. at the end of each iteration, we predict the labels of the entire training samples. based on the predicted label and the label assigned by distant supervision, we decide to filter out a sample in the next iteration. after each iteration, we consider the entire training samples for the filtering process. the individual models at each iteration can be very sensitive to wrong labels, so in our training process, we maintain a self-ensemble version of the models which is a moving average of the models of previous iterations. we hypothesize that the predictions of the ensemble model are more stable than the individual models. so the predictions from the ensemble model are used to identify the noisy samples. these noisy samples are removed from the training samples of the next iteration. we consider the entire distantly supervised training data for prediction and filtering so that if a sample is filtered out wrongly in an iteration, it can be included again in the training data in the subsequent iteration. ## student models we have used the following state-of-the-art neural relation extraction models as the student model in our filtering framework. these models use three types of embedding vectors: (1) word embedding vector @xmath0. the sentences are represented using a sequence of vectors {x 1 , x 2 , ....., x n } where x @xmath1. represents the concatenation of vectors and n is the sentence length. these token vectors x t are given as input to all the following models. ## related work there are two approaches for relation extraction @xcite : (i) pipeline approaches @xcite @xcite (ii) joint extraction approaches @xcite . most of these models work with distantly supervised noisy datasets. thus noise mitigation is an important dimension in this area of research. multi-instance relation extraction is one of the popular methods for noise mitigation. riedel et al. (2010 their goal was to reduce the effect of noisy samples using this multi-instance setting. they used different types of sentence selection mechanisms to give importance to the sentences that contain relation specific keywords and ignore the noisy sentences. but this idea may not be effective if there is only one sentence for an entity pair. @xcite and @xcite used the multi-task learning approach for mitigating the influence of the noisy samples. they used fine-grained entity typing as an additional task in their model. @xcite used an adversarial training approach for the same purpose. they add noise to the word embeddings to make the model more robust for distantly supervised training. @xcite used the generative adversarial network (gan) to address this issue of the noisy samples in relation extraction. they used a separate binary classifier as a generator in their model for each positive relation class to identify the true positives for that relation and filter out the noisy ones. @xcite used reinforcement learning to identify the noisy samples for the positive relation classes. @xcite used reinforcement learning to identify the noisy samples for the positive relations and then use the identified noisy samples as unlabelled data in their model. @xcite used a clustering approach to identify the noisy samples. they assign the correct relation label to these noisy samples and use them as additional training data in their model. different from these approaches, we propose a student-teacher framework that can work with any supervised neural network models to address the issue of noisy samples in distantlysupervised datasets. ## conclusion in this work, we propose a self-ensemble based noisy samples filtering framework for distantly supervised relation extraction. our framework identifies the noisy samples during training and removes them from the training data in the following iterations. this framework can be used with any supervised relation extraction models. we run experiments using several state-of-the-art neural models with this proposed filtering framework on the distantly supervised new york times dataset. the results show that our proposed framework improves the robustness of these models and increases their f1 score on the relation extraction task.