SENT : S entence-level Distant Relation Extraction via Negative Training (2021)

Abstract: Distant supervision for relation extraction provides uniform bag labels for each sentence inside the bag, while accurate sentence labels are important for downstream applications that need the exact relation type. Directly using bag labels for sentence-level training will introduce much noise, thus severely degrading performance. In this work, we propose the use of negative training (NT), in which a model is trained using complementary labels regarding that “the instance does not belong to these complementary labels”. Since the probability of selecting a true label as a complementary label is low, NT provides less noisy information. Furthermore, the model trained with NT is able to separate the noisy data from the training data. Based on NT, we propose a sentence-level framework, SENT, for distant relation extraction. SENT not only filters the noisy data to construct a cleaner dataset, but also performs a re-labeling process to transform the noisy data into useful training data, thus further benefiting the model’s performance. Experimental results show the significant improvement of the proposed method over previous methods on sentence-level evaluation and de-noise effect.

## introduction relation extraction (re), which aims to extract the relation between entity pairs from unstructured text, is a fundamental task in natural language processing. the extracted relation facts can benefit various downstream applications, e.g., knowledge graph completion @xcite , information extraction @xcite and question answering @xcite . a significant challenge for relation extraction is the lack of large-scale labeled data. thus, distant obama was born in the united states. ## methodology in order to achieve sentence-level relation classification using bag-level labels in distant re, we propose a framework, sent, which contains three main steps (as shown in fig. 2 ): (1) separating the noisy data from the training data with negative training (sec.3.1); (2) filtering the noisy data as well as re-labeling a part of confident instances (sec.3.2); (3) leveraging an effective training algorithm based on (1) and ( 2 ) to further boost the performance (sec.3.3). specifically, we denote the input data in this task as s * = {(s 1 , y * 1 ), . . . , (s n , y * n )}, where y * @xmath0, . . . , c} is the bag-level label of the i th input sentence s i . obviously, this is a noisy dataset drawn from a noisy distribution d * because these bag-level labels y * come from the distant label of each entity bag. for each s i containing a pair of @xmath1, e 2 >, y * i is one of the relation facts foot_0 @xmath2, e 2 > participates in in the database. such annotation method indicates that y * i is a potential noisy label for s i . here, we denote d as the real data distribution without noise, and the clean dataset drawn from d as @xmath3, y 1 ), . . . , (s n , y n )}. the ambition of this work is to find the best estimated parameters θ of the real mapping f : @xmath4, (x, y) ∈ d based on the noisy data s * . we design three steps for achieving this goal: (1) recognizing the set of noisy data s * n from s * using negative training, where ## experiments the experiments in this work are divided into two parts, respectively conducted on two datasets: the nyt-10 dataset @xcite and the tacred dataset @xcite . the first part is the effectiveness study on sentence-level evaluation for distant re. different from bag-level evaluation, a sentence-level evaluation compute precision (prec.), recall (rec.) and f1 metric directly on all of the individual instances in the dataset. in this part, we adopt the nyt-10 data set for sentence-level training, following the setting of @xcite , who publishes a manually labeled sentence-level test set. 4 besides, they also publish a test set for evaluating noisefiltering ability. details of the adopted dataset are shown in we construct the second part of experiments (sec.4.4) to better understand sent's behaviors. since no labeled training data are available in the distant supervision setting, we construct a noisy dataset with 30% noise from a labeled dataset, tacred @xcite 5 . we regard this constructed dataset as noisy-tacred. the reason datasets nyt-10 noisy-tacred #label num. 24 41 train #instances 371461 68124 #positive 110518 26575 #noise unknown 20586 dev #instances 2379 22631 #positive 337 5436 test #instances 2164 15509 #positive 323 3325 when constructing noisy-tacred, the noisy instances are uniformly selected with 30% noise ratio. then, each noisy label is created by sampling a label from a complementary class with a weight of class frequency (in order to maintain the data distribution). note that the original dataset consists of 80% "no relation" data, which means 80% of the noisy instances are "false-positive" instances, corresponding to the large amount of "false-positive" noise in nyt-10. details of the noisy-tacred are also shown in ## conclusion sent, a novel sentencelevel framework based on negative training (nt) for sentence-level training on distant re data. nt not only prevent the model from overfitting noisy data, but also separate the noisy data from the training data. by iteratively performing noisefiltering and re-labeling based on nt, sent helps re-fine the noisy distant data and achieves remarkable performance. experimental results verify the improvement of sent over previous methods on sentence-level relation extraction and noise-filtering effect.