Graph Enhanced Dual Attention Network for Document-Level Relation Extraction (2020)

Abstract: Document-level relation extraction requires inter-sentence reasoning capabilities to capture local and global contextual information for multiple relational facts. To improve inter-sentence reasoning, we propose to characterize the complex interaction between sentences and potential relation instances via a Graph Enhanced Dual Attention network (GEDA). In GEDA, sentence representation generated by the sentence-to-relation (S2R) attention is refined and synthesized by a Heterogeneous Graph Convolutional Network before being fed into the relation-to-sentence (R2S) attention . We further design a simple yet effective regularizer based on the natural duality of the S2R and R2S attention, whose weights are also supervised by the supporting evidence of relation instances during training. An extensive set of experiments on an existing large-scale dataset show that our model achieve competitive performance, especially for the inter-sentence relation extraction, while the neural predictions can also be interpretable and easily observed.

## introduction relation extraction (re) is an important research topic in natural language processing with plenty of applications. the task of re is to detect the relationship from a given context and target entities. depending on the given context, re can be divided into two types: sentence-level re @xcite @xcite @xcite @xcite and document-level re @xcite @xcite . compared with widely-studied sentence-level re, document-level re is a more challenging task that requires more investigations. one of main challenges in document-level re is how to perform inter-sentence reasoning over a long document to synthesize local and global information for potential relation facts. according to the statistics of docred @xcite , a large-scale human-annotated document-level re dataset, there are 40.7% relational facts can only be extracted from multiple sentences in this dataset. that means a desirable neural model for document-level re requires sophisticated inter-sentence reasoning capabilities. in this paper, we propose to improve inter-sentence reasoning from a perspective of better characterizing the complex interaction between multiple sentences and multiple potential relation instances in a document. since one relation instance may be expressed by multiple sentences and one sentence may reveal multiple relational facts (or parts of relational facts), it is natural and straightforward to use attention between sentences and potential relation instances to capture the complex many-to-many interaction. to this end, we introduce a bi-directional attention mechanism consisting of the attention paid by a sentence to relation instances (sentence-to-relation, s2r) and the attention paid by a relation instance to sentences (relation-to-sentence, r2s). though bi-directional attention has been widely used in other nlp tasks (e.g., machine comprehension @xcite and sentiment analysis @xcite , what makes our architecture unique and innovative is the following three designs based on the classic bi-directional attention. to evaluate our approach, we carried out extensive experiments on the docred dataset @xcite . from the experiment result, we found that characterizing the interaction between sentences and relation instances by our graph-enhanced dual attention network could significantly improve the performance of document-level re. our main contributions are: ## proposed approach in this section, we present the geda in detail. the overall architecture is shown in fig. 1 . given a document d contains n words, m sentences, and k different entities, there will be k • (k -1) potential relation instances, our goal is to extract all the potential relation instances in a parallel way. geda mainly consists of 5 components: 1) the encoding layer, which generates the preliminary representations of sentences, entities and relation instances; 2) the graph-enhanced bi-directional attention layer, which synthesizes the intra-sentence and inter-sentence information to generate a refined representation of relation instances; 3) the constraint of attention duality as a regularizer; 4) the evidence-supervision loss; 5) and the final classification layer, which projects the representation of a potential relation instance to the probabilities for each relation type. ## related work for document-level re, @xcite used graph-lstm networks to cross-sentence n-ary re, @xcite models inter-sentential dependency through a similar graph. (nguyen and verspoor, 2018) applied cnn to cross-sentence re with improved character encoding. @xcite used a modified transformer @xcite with cnn, followed by a bi-affine pairwise scoring prediction, and applying distant supervision and multi-task learning. @xcite addressed document-level n-ary re with the design of multi-scale representation learning, which aims to learn the representation of entity tuples at both mention-level and entity-level. @xcite used gcn as encoder and a multi-instance based classifier. @xcite applied gcn with a graph consisted of different types of nodes and edges, aiming to infer those representations from other edges. unlike previous work, we focus on inter-sentence reasoning and model the interaction between relation instances and sentences. ## conclusion we have introduced our neural architecture geda for document-level relation extraction. the intuition behind geda is to characterize the interaction between sentences and relation instances better to improve inter-sentence reasoning over the whole document. the novelty of geda mainly lies in the graph-based refinement of sentence representation and two simple yet effective regularizers based on attention duality and supporting evidence respectively. experiments verified the superiority of the graph-enhanced dual attention mechanism, especially for the inter-sentence relation extraction. in the future, we will investigate more sophisticated reasoning techniques targeting more specific scenarios of inter-sentence relation extraction, e.g., involving common-sense reasoning.