Document-level Relation Extraction with Dual-tier Heterogeneous Graph (2020)

Abstract: Document-level relation extraction (RE) poses new challenges over its sentence-level counterpart since it requires an adequate comprehension of the whole document and the multi-hop reasoning ability across multiple sentences to reach the final result. In this paper, we propose a novel graph-based model with Dual-tier Heterogeneous Graph (DHG) for document-level RE. In particular, DHG is composed of a structure modeling layer followed by a relation reasoning layer. The major advantage is that it is capable of not only capturing both the sequential and structural information of documents but also mixing them together to benefit for multi-hop reasoning and final decision-making. Furthermore, we employ Graph Neural Networks (GNNs) based message propagation strategy to accumulate information on DHG. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on two widely used datasets, and further analyses suggest that all the modules in our model are indispensable for document-level RE.

## introduction relation extraction (re), which aims to identify relational facts between entities from plain text, is one of the most fundamental tasks in information extraction (ie) and natural language processing (nlp). existing methods usually focus on extracting relations from a single sentence (i.e., sentence-level re) @xcite @xcite . however, sentence-level re suffers from a nonignorable limitation in practice: a large number of relational facts are expressed in multiple sentences. lately, the attention of identifying inter-sentence relations heightens the interest of moving re forward from sentence-level to document-level. the most straightforward way to perform document-level re is to treat documents as long sequences and then employ sequential models, adapted from sentence-level re, to extract the relation between given entities @xcite @xcite . however, these methods inevitably face several challenges in modeling long-term dependencies and enabling multi-hop reasoning, yet identifying long-term and inter-sentence relations is precisely the key and difficult point of document-level re. recently, a number of exquisite graph-based models for document-level nlp tasks show their sparkles in capturing multi-hop relations @xcite @xcite . nevertheless, these approaches often fail to adequately capture the inherent structure of documents and discard masses of valuable structural information when transforming documents into graphs. as a matter of fact, the document structure, especially the positional, syntactical, and hierarchical structure, has proven to be very effective for many document-level tasks @xcite . look at a concrete example shown in figure 1 , in order to predict the relation between bradycardia and ramipril, one has to first grasp the key point that bradycardia is caused by hyperpotassemia in sentence 1, then identify the fact that the cause of hyperkalemia is ramipril from sentence 3, and finally infer from these observations that ramipril is also the cause of bradycardia with a background coreference knowledge that hyperpotassemia and hyperkalemia refer to the exact same entity. it demonstrates that @xcite a 76 year-old woman was transferred to the emergency room with loss of consciousness due to marked bradycardia caused by hyperpotassemia. @xcite the concentration of serum potassium was high, and normal sinus rhythm was restored after correction of the serum potassium level. the decision-making process in document-level re requires understanding first and reasoning later over multiple sentences, which is obviously beyond the reach of traditional sequential approaches. furthermore, there are amounts of useful structural information to help detect the relation between bradycardia and ramipril, such as the keyword caused by is close to bradycardia in the first sentence and these two mentions of hyperkalemia belong to two different sentences, which provides an in-depth understanding of the whole text and should be fully exploited in the document modeling process. in this paper, we propose a novel document-level re model that builds a dual-tier heterogeneous graph (dhg) to successively model document structure and enable relational reasoning. specifically, the first-tier of dhg is a structure modeling layer (sml) that responsible for comprehensively encoding the inherent structure of document from three aspects of sequence, syntax, and hierarchy, thus each document is transformed into a graph in which nodes are words and sentences whereas edges are relationships between them. in the second-tier, based on the semantically-rich representations induced from the previous layer, a relation reasoning layer (rrl) is introduced to propagate relational information among various entities and enable multi-hop relational reasoning. with graph neural networks (gnns), we assume that the desired signal for identifying relational facts could be captured by propagating node information along edges in our dhg. by this means, taking figure 1 as an example again, the messages of keywords caused by and cause will be propagated to word nodes through sentence nodes in sml, and gradually accumulated to the entity nodes in rrl for the final decision. to the best of our knowledge, it is the first attempt to separate document modeling from multi-hop reasoning in document-level nlp tasks via a dual-tier heterogeneous structure. we conduct extensive experiments on two public widely used document-level re datasets. results suggest that the proposed model achieves state-of-the-art performance. through detailed ablation studies, we further show that all the components in our approach are indispensable for document-level re. moreover, we also demonstrate that incorporating pre-trained language model (e.g., bert @xcite ) with dhg can bring further improvements. ## preliminaries in this section, we first briefly recall some basic concept of document-level re, and then describe the proposed message propagation strategy over our dhg. ## methodology in this section, we introduce the proposed dual-tier heterogeneous graph (dhg) and document-level re model. figure 3 shows the overall system diagram. specifically, the re model can be categorized into five layers: (1) input layer is responsible for transforming input words into dense vectorized representations; (2) context encoding layer could be any common sequence encoder to generate a contextualized representation for each word; (3) structure modeling layer is the first-tier heterogeneous graph of dhg, aiming to model the inherent structural information of plain text, including adjacency, affiliation, and syntactic dependency relations; (4) relation reasoning layer focuses on capturing multi-hop relations between entity pairs in a document, which is the second-tier heterogeneous graph of dhg; (5) output layer treats relation prediction as a multi-label classification problem and predicts possible relations for each entity pair. ## experiments this section first introduces datasets and experimental settings, and then presents performance comparison results with baselines as well as imperative analysis in order to validate the effectiveness of dhg. ## related work the study presented in this paper is directly related to existing researches on document-level relation extraction @xcite , which is recently introduced as a branch of relation extraction. the goal of this task is to identify all relations between each entity pair within a given document. some early work first designs an extensive set of features and then trains a classifier based on these feature vectors @xcite @xcite . later, many neural network-based methods are introduced to solve the problem. still, most of these approaches use sequential models, which can be regarded as simple extensions of sentence-level re models @xcite . beyond that, the approach presented in this paper is related to recent studies using graph neural networks for document-level relation extraction @xcite . different from the previous work, our innovation lies in that we creatively propose to utilize a dual-tier heterogeneous graph to model the inherent structure of a document in the lower layer, and then enable multi-hop relational reasoning in the upper layer, instead of handling these complex requirements with one messy or imperfect graph. meanwhile, the weighted mechanism in our message propagation strategy is inspired by the success of wgcn @xcite , which shows the effectiveness of using learnable weights to determine the amount of information from neighbors used in local aggregation. besides these studies, our work is also relevant to the following research directions: sentence-level re: sentence-level is the most classical and simple setting of re. @xcite and @xcite demonstrate that the positional and structural information is quite effective for sentence-level re. there are also some works employ graph-based models for cross-sentence re @xcite @xcite , but they ideally restrict all relation candidates in one continuous sentence-span with a fixed length, while the problem we want to solve in this paper requires extracting relations from the whole document, which is more difficult but practical. gnns for nlp: recently, there is a considerable amount of interest in applying gnns to nlp tasks. for example, in neural machine translation, gnns have been employed to integrate syntactic and semantic information into encoders @xcite ; de @xcite and @xcite employ gnns over a heterogeneous graph to do multi-hop machine reading comprehension, which inspire our idea of the relation reasoning layer of dhg. ## conclusion in this paper, we present a novel gnns-based approach for document-level re with dhg, a dual-tier heterogeneous graph, to achieve document modeling and multi-hop reasoning in proper order. experimental results on two widely used document-level re datasets suggest that the proposed model achieves state-of-the-art performance. we believe our approach is robust enough and can be readily adapted for other document-level nlp tasks without much manual efforts for domain adaptation. in the future, we would like to investigate explainable gnns for document-level re and integrate pre-training techniques with the proposed dual-tier heterogeneous structure in document-level nlp tasks.