Two Training Strategies for Improving Relation Extraction over Universal Graph (2021)

Abstract: This paper explores how the Distantly Supervised Relation Extraction (DS-RE) can benefit from the use of a Universal Graph (UG), the combination of a Knowledge Graph (KG) and a large-scale text collection. A straightforward extension of a current state-of-the-art neural model for DS-RE with a UG may lead to degradation in performance. We first report that this degradation is associated with the difficulty in learning a UG and then propose two training strategies: (1) Path Type Adaptive Pretraining, which sequentially trains the model with different types of UG paths so as to prevent the reliance on a single type of UG path; and (2) Complexity Ranking Guided Attention mechanism, which restricts the attention span according to the complexity of a UG path so as to force the model to extract features not only from simple UG paths but also from complex ones. Experimental results on both biomedical and NYT10 datasets prove the robustness of our methods and achieve a new state-of-the-art result on the NYT10 dataset. The code and datasets used in this paper are available at https://github.com/baodaiqin/UGDSRE .

## introduction relation extraction (re) is an important task in natural language processing (nlp). re aims to turn unstructured texts into structured knowledge graph (kg), which is typically stored as (e 1 , r, e 2 ) triplets, where e 1 is a head entity, r is a relation and e 2 is a tail entity, such as (aspirin, may treat, pain) and (guy maddin, place lived, winnpeg). re can be formulated as a classification task to predict a predefined relation r from entity pair (e 1 , e 2 ) annotated evidences. one obstacle that is encountered when building a re system is the generation of a large amount of manually annotated training instances, which is ex-pensive and time-consuming. for coping with this difficulty, @xcite propose distant supervision (ds) to automatically generate training samples via linking kgs to texts. they assume that if (e 1 , r, e 2 ) is in a kg, then all sentences that contain (e 1 , e 2 ) (hereafter, sentence evidences) express the relation r. it is well known that the ds assumption is too strong and inevitably accompanies the wrong labeling problem, such as the sentence evidences (1b and 2) below, which fail to express may treat and place lived relation respectively. recently, neural network models with attention mechanism have been proposed to alleviate the wrong labeling problem and attend informative sentence evidences such as (1a) @xcite @xcite @xcite . however, there can be a large portion of entity pairs that lack such informative sentence evidences that explicitly express their relation. this makes distantly supervised relation extraction (ds-re) further challenging @xcite . for compensating the lack of informative sentence evidences, @xcite utilize multihop paths connecting a target entity pair (hereafter, path) over a kg as extra evidences for ds-re. an example of such multi-hop kg path can be seen in figure 1 , where p 1 depicts a multi-hop kg path t r 1 m ay _t re at t r 2 ug text relation kg relation hop 1 : " colesevelam hcl component of colesevelam ... hop 2 : colesevelam ... may treat type 2 diabetes" hop 1 : " colesevelam hcl provides an effective alternative to ... colestipol while offering the potential for fewer adverse effects ... hop 2 : colestipol may treat hyperglyceridemia hop 3 : epidemiolgical studies revealed strong and international reproducible links between type 2 diabetes and hyperglyceridemia." "to evaluate initial combination therapy with metformin plus colesevelam hcl e1 in drug-naive hispanic patients with type 2 diabetes e2 ." ... ... multi-hop path evidence sentence evidence p 1 p 2 p 3 s 1 s 2 path evidence encoder sentence evidence encoder p 2 s 1 s 2 p 3 concat rc attention mechanism attention mechanism a' 1 a 1 a' 3 a 2 p 1 s all p all p 1 p 2 a' 2 a' i : attention value for p i s/p all : vector representation for all sentences/paths that contain/connect a target entity pair. p i : vector representation for p i s i : vector representation for s i a i : attention value for s i concat: concatenate rc: relation classification figure 1: overview of our ug-based framework, where colesevelam hcl and type 2 diabetes are the target entities, colesevelam ..., colestipol and hyperglyceridemia are intermediate entities, each ug path consists of multiple hops and each hop represents a kg relation (such as "colestipol may treat hyperglyceridemia") or text (or textual) relation (such as tr1 and tr2), which is the sentence containing two (target or intermediate) entities. the model of @xcite uses such multi-hop paths as additional features for predicting the relation between a given target entity pair (e 1 , e 2 ), which is reported effective for performance improvement. however, kgs are often highly incomplete @xcite and may be too sparse to provide enough informative paths in practice, which may hamper the effectiveness of multi-hop paths. given this background, in this study, we take one step further, aiming for inducing maximal signals of distant supervision from both a kg and a large text collection (hereafter, text). for this purpose, we consider using multi-hop paths over a universal graph (ug) as extra features for ds-re. here, we define a ug as a joint graph representation of both kg and text, where each node represents an entity from kg or text, and each edge indicates a kg relation or textual relation, as shown in figure 1 . the path p 2 in the figure is an example of ug path, comprising a textual edge tr1, a kg edge may treat, and another textual edge tr2. by augmenting the original kg with textual edges, one can expect far more chances to find informative path evidences between any given target entity pairs, because the number of such textual edges is likely to be much larger than the number of kg edges (note that one can collect as many textual edges as needed from a raw text corpus with an entity linker). extending a kg to a ug, therefore, may allow a ds-re model to learn richer distant supervision signals. the idea of using multi-hop paths over a ug is not necessarily new on its own. for example, @xcite propose to use a ug for knowledge graph completion, and @xcite propose a model trained to reason over a ug for question answering. however, there is no prior study that has explored the effective way to use a ug for the task of ds-re from text. in fact, finding an effective way of using a ug for ds-re is not as simple as it may seem. as we report in this paper, a straightforward extension of the @xcite model to the ug setting may result in performance degradation. motivated by this, in this paper, we address how one can make effective use of ug for ds-re. we first report our observation that a straightforward extension of the @xcite model to the ug setting tends to allocate the majority of attention to only a limited set of ug paths such as short kg paths and miss out the learning from a wide range of ug paths ( §4.1), which hinders performance gain. in order to alleviate the negative effect of the attention bias and realize the potential of ug paths, we propose two training (or debiasing) strategies: (1) path type adaptive pretraining ( §4.2), which aims to improve the adaptability of the model to various ug paths; and (2) complexity ranking guided attention mechanism ( §4.3), which enables the model to learn from both simple and complex ug paths. experimental results on both biomedical and nyt10 ( riedel et al., 2010) datasets prove that: (1 ) ug paths have the potential to bring performance gain for ds-re as compared with kg paths; (2) the proposed training methods are effective to fully exploit the potential of ug paths for ds-re because the proposed methods significantly and consistently outperform several baselines on both datasets and especially achieve a new state-of-the-art result on the nyt10 dataset. ## base model we select the ds-re model proposed by @xcite 2019 ) as our base model and extend it into our ug setting. given a target entity pair (e 1 , e 2 ), a bag of corresponding sentence evidences s @xmath0, ..., s n } and a bag of ug paths p @xmath1, ..., p m }, the base model aims to measure the probability of (e 1 , e 2 ) having a predefined relation r (including the empty relation na). the base model consists of four main modules: kg encoder, sentence evidence encoder, path evidence encoder and relation classification layer, as shown in figure 2 . ## conclusion and future work we have introduced ug paths as extra evidences for the task of ds-re from text. in order to fully take advantage of the rich ug paths, we have proposed two training (or debiasing) strategies: path type adaptive pretraining and complexity ranking guided attention mechanism. we have conducted experiments on both biomedical and nyt10 datasets. the results show that the two proposed methods are effective for exploiting the potential of ug paths for improving the performance of ds-re. in the future, we plan to carry out the following steps: (1) we further investigate how the proposed training methods influence the performance via manual analysis so as to better the efficiency; and