Revisiting the Negative Data of Distantly Supervised Relation Extraction (2021)

Abstract: Distantly supervision automatically generates plenty of training samples for relation extraction. However, it also incurs two major problems: noisy labels and imbalanced training data. Previous works focus more on reducing wrongly labeled relations (false positives) while few explore the missing relations that are caused by incompleteness of knowledge base (false negatives). Furthermore, the quantity of negative labels overwhelmingly surpasses the positive ones in previous problem formulations. In this paper, we first provide a thorough analysis of the above challenges caused by negative data. Next, we formulate the problem of relation extraction into as a positive unlabeled learning task to alleviate false negative problem. Thirdly, we propose a pipeline approach, dubbed ReRe, that first performs sentence classification with relational labels and then extracts the subjects/objects. Experimental results show that the proposed method consistently outperforms existing approaches and remains excellent performance even learned with a large quantity of false positive samples. Source code is available online at https://github.com/redreamality/RERE-relation-extraction .

## introduction relational extraction is a crucial step towards knowledge graph construction. it aims at identifying relational triples from a given sentence in the form of subject, relation, object , in short, s, r, o . for example, given s1 in figure 1 , we hope to extract william shakespeare, birthplace, stratford-upon-avon . this task is usually modeled as a supervised learning problem and distant supervision @xcite is utilized to acquire large-scale training data. the core idea is to obtain training data is through automatically labeling a sentence with existing relational triples from a knowledge base (kb). for example, given a triple s, r, o and a sentence, if the sentence contains both s and o, distant supervision methods regard s, r, o as a valid sample for the sentence. if no relational triples are applicable, the sentence is labeled as "na". despite the abundant training data obtained with distant supervision, nonnegligible errors also occur in the labels. there are two types of errors. in the first type, the labeled relation does not conform with the original meaning of sentence, and this type of error is referred to as false positive (fp). for example, in s2, the sentence "shakespeare spent the last few years of his life in stratford-upon-avon." does not express the relation birthplace, thus being a fp. in the second type, large amounts of relations in sentences are missing due to the incompleteness of kb, which is referred to as false negative (fn). for instance, in s3, "buffett was born in 1930 in omaha, nebraska." is wrongly labeled as na since there is no relation (e.g., birthplace) between buffett and omaha, nebraska in the kb. many efforts have been devoted to solving the fp problem, including pattern-based methods @xcite , multi-instance learning methods @xcite and reinforcement learning methods @xcite . significant improvements have been made. however, fn problem receives much less attention @xcite @xcite . to the best of our knowledge, none existing work with deep neural networks to solve this problem. we argue that this problem is fatal in practice since there are massive fn cases in datasets. for example, there exist at least 33% and 35% fns in nyt and ske datasets, respectively. we will deeply analyze the problem in section 2.1 another huge problem in relation extraction is the overwhelming negative labels. as is widely acknowledged, information extraction tasks are highly imbalanced in class labels @xcite @xcite . in particular, the negative labels account for most of the labels in relation extraction under almost any problem formulation, which makes relation extraction a hard machine learning problem. we systematically analyze this in section 2.2. in this paper, we address these challenges caused by negative data. our main contribution can be summarized as follows. ## problem analysis and pilot experiments we use (c i , t i ) to denote a training instance, where c i is a sentence consisting of n tokens c @xmath0, ..., c in ] labeled by a set of triples t @xmath1, r, o } from the training set d. for rigorous definition, [c i1 , ..., c in ] can be viewed as an ordered set {(c i1 , 1), ..., (c in , n )} so that set operations can be applied. we assume @xmath2, where r is a finite set of all relations in d. other model/taskspecific notations are defined after each problem formulation. we now clarify some terms used in the introduction and title without formal definition. a negative sample refers to a triple t / ∈ t i . negative label refers to the negative class label (e.g., usually "0" for binary classification), used for supervision with respect to task-specific models. under different task formulation, the negative labels can be different. negative data is a general term that includes both negative labels and negative samples. there are two kinds of false negatives. relation-level false negative (s3 in figure 1 ) refers to the situation where there exists @xmath3, r , o / ∈ t i , but r is actually expressed by c i , and does not appear in other @xmath4. similarly, entity-level false negative (s4 and s5 in figure 1 ) means r appears in other @xmath5. imbalanced class distribution means that the quantity of negative labels is much larger than that of positive ones. ## our model the relational triple extraction task decomposed in eq. ( 2 ) inspires us to design a two-staged pipeline, in which we first detect relation at sentence level and then extract subjects/objects for each relation. the overall architecture of rere is shown in figure 2. ## conclusion in this paper, we revisit the negative data in relation extraction task. we first show that the false negative rate is largely underestimated by previous researches. we then systematically compare three commonly adopted paradigms and prove that our paradigm suffers less from the overwhelming negative labels. based on this advantage, we propose rere, a pipelined framework that first detect relations at sentence level and then extract entities for each specific relation and provide a multi-label pu learning loss to recover false negatives. empirical results show that rere consistently outperforms the existing state-of-the-arts by a considerable gap, even when learned with large false negative rates.