Entity and Evidence Guided Document-Level Relation Extraction (2021)

Abstract: Document-level relation extraction is a challenging task, requiring reasoning over multiple sentences to predict a set of relations in a document. In this paper, we propose a novel framework E2GRE (Entity and Evidence Guided Relation Extraction) that jointly extracts relations and the underlying evidence sentences by using large pretrained language model (LM) as input encoder. First, we propose to guide the pretrained LM’s attention mechanism to focus on relevant context by using attention probabilities as additional features for evidence prediction. Furthermore, instead of feeding the whole document into pretrained LMs to obtain entity representation, we concatenate document text with head entities to help LMs concentrate on parts of the document that are more related to the head entity. Our E2GRE jointly learns relation extraction and evidence prediction effectively, showing large gains on both these tasks, which we find are highly correlated.

## introduction relation extraction (re), the problem of predicting relations between pairs of entities from text, has received increasing research attention in recent years @xcite @xcite . this problem has important downstream applications to numerous tasks, such as automatic knowledge acquisition from web documents for knowledge graph construction @xcite , question answering @xcite and dialogue systems @xcite . while most document: [0] the legend of zelda : the minish cap ( ) is an action -adventure game and the twelfth entry in the legend of zelda series. aside from extracting entity relations from a document, it is often useful to also highlight the evidence that a system uses to predict them, so that a human or second system can verify them for consistency. what is more, evidence prediction can potentially supplement re performance by restricting the model's focus on the correct context. in preliminary experiments, we find that current models are able to achieve around 87% re f1 on docred by only keeping the gold evidence sentences when trained and evaluated only on the gold evidence sentences, which is a significant im-provement on current leaderboard docred re f1 numbers (∼ 63% re f1). however, evidence prediction is a challenging task, and most existing relation extraction (re) approaches ignore the task of evidence prediction entirely. most recent approaches for relation extraction fine-tune large pretrained language models (lms) (e.g.,bert @xcite , roberta @xcite ) as input encoder. however, naively adapting pretrained lms for document-level re faces an issue which limits its performance. due to the length of a given document, many more entities and relations exist in document-level re than in intra-sentence re. a pretrained lm has to simultaneously encode information regarding all pairs of entities for relation extraction, making the task more difficult, and limiting the pretrained lm's effectiveness. in this paper we propose a new framework: entity and evidence guided relation extraction (e2gre), which jointly solves relation extraction and evidence prediction. for evidence prediction, we take a pretrained lm as input encoder and use its internal attention probabilities as additional features to predict evidence sentences. as a result, we use supporting evidence sentences to provide direct supervision on which tokens the lm should attend to during finetuning, which in turn helps improve relation extraction in a joint training framework. to further help lms focus on a smaller set of relevant word context from a long document, we also introduce entity-guided input sequences as the input to these models, by appending each head entity to the document text, one at a time. this allows the lm encoder to explicitly model relations involving a specific head entity while ignoring all other entity pairs, thus simplifying the task for the lm encoder. the joint training framework helps the model locate the correct semantics that are required for each relation prediction. to the best of our knowledge 1 , we are the first to present an effective joint training framework for relation extraction and evidence prediction. each of these ideas gives a significant boost in performance, and by combining them, we are able to achieve highly competitive results on the docred leaderboard. we obtain 62.5 relation extraction f1 and 50.5 evidence prediction f1 from our e2gre trained roberta large model, which is the current state-of-the-art performance on evi-1 based on published papers on docred. our main contributions are summarized as follows: ## related work early work attempted to solve re with statistical methods with different feature engineering @xcite . later on, neural models have shown better performance at capturing semantic relationships between entities. these methods include cnn-based approaches @xcite and lstm-based approaches @xcite . on top of using cnns/lstm encoders, previous models added additional layers to model semantic interactions. for example, @xcite introduced using hierarchical attentions in order to generate relational information from coarse-tofine semantic ideas; @xcite applied gcns over pruned dependency trees, and @xcite introduced attention guided graph convolutional networks (ag-gcns) over dependency trees. these models have shown good performance on intra-sentence relation extraction, but are not easily adapted for document-level re. many approaches for document-level re are graph-based neural network methods. @xcite first introduced a document graph being used for document-level re; in @xcite , an entity-centric, multi-scale representation learning on entity/sentence/document-level lstm model was proposed for document-level n-ary re task. @xcite recently proposed a novel edge-oriented graph model that deviates from existing graph models. @xcite proposed an induced latent graph and @xcite used an explicit heterogeneous graph for docred. these graph models generally focus on constructing unique nodes and edges, and have the advantage of connecting and aggregating different granularities of information. @xcite pointed out multi-entity and multi-label issues for documentlevel re, and proposed two techniques: adaptive thresholding and localized context pooling, to address these problems. pretrained language models @xcite @xcite are powerful nlp tools trained with enormous amounts of unlabelled data. in order to take advantage of the large amounts of text that these models have seen, finetuning on large pretrained lms has been shown to be effective on relation extraction @xcite . generally, large pretrained lms are used to encode a sequence and then generate the representation of a head/tail entity pair to learn a classification @xcite . baldini soares et al. [2019] introduced a new concept similar to bert called "matchingthe-black" and pretrained a transformer-like model for relation learning. the models were finetuned on semeval-2010 task 8 and tacred achieved stateof-the-art results. our framework aims to improve the effectiveness of pretrained lms for documentlevel relation extraction, with our entity and evidence guided approaches. ## method in this section, we introduce our e2gre framework. first, we describe how to generate entityguided inputs. then we present how to jointly train re with evidence prediction, and finally show how to combine this with our evidence-guided attentions. we use bert as our pretrained lm when describing our framework.