Reasoning with Latent Structure Refinement for Document-Level Relation Extraction (2020)

Abstract: Document-level relation extraction requires integrating information within and across multiple sentences of a document and capturing complex interactions between inter-sentence entities. However, effective aggregation of relevant information in the document remains a challenging research question. Existing approaches construct static document-level graphs based on syntactic trees, co-references or heuristics from the unstructured text to model the dependencies. Unlike previous methods that may not be able to capture rich non-local interactions for inference, we propose a novel model that empowers the relational reasoning across sentences by automatically inducing the latent document-level graph. We further develop a refinement strategy, which enables the model to incrementally aggregate relevant information for multi-hop reasoning. Specifically, our model achieves an F1 score of 59.05 on a large-scale document-level dataset (DocRED), significantly improving over the previous results, and also yields new state-of-the-art results on the CDR and GDA dataset. Furthermore, extensive analyses show that the model is able to discover more accurate inter-sentence relations.

## introduction relation extraction aims to detect relations among entities in the text and plays a significant role in a variety of natural language processing applications. early research efforts focus on predicting relations between entities within the sentence @xcite . however, valuable relational information between entities, such as biomedical findings, is expressed by multiple mentions across sentence boundaries in real-world scenarios @xcite . therefore, the scope of extraction in biomedical domain has recently been expanded to cross-sentence level @xcite @xcite . a more challenging, yet practical extension, is the document-level relation extraction, where a system needs to comprehend multiple sentences to infer the relations among entities by synthesizing relevant information from the entire document @xcite . figure 1 shows an example adapted from the recently proposed document-level dataset docred @xcite . in order to infer the inter-sentence relation (i.e., country of citizenship) between yulia tymoshenko and ukrainian, one first has to identify the fact that lutsenko works with yulia tymoshenko. next we identify that lutsenko manages internal affairs, which is a ukrainian authority. after incrementally connecting the evidence in the document and performing the step-by-step reasoning, we are able to infer that yulia tymoshenko is also a ukrainian. prior efforts show that interactions between mentions of entities facilitate the reasoning process in the document-level relation extraction. thus, @xcite and @xcite leverage multi-instance learning @xcite . on the other hand, structural information has been used to perform better reasoning since it models the non-local dependencies that are obscure from the surface form alone. @xcite construct dependency graph to capture interactions among n-ary entities for cross-sentence extraction. @xcite extend this approach by using co-reference links to connect dependency trees of sentences to construct the document-level graph. instead, @xcite construct a heterogeneous graph based on a set of heuristics, and then apply an edge-oriented model @xcite to perform inference. unlike previous methods, where a documentlevel structure is constructed by co-references and rules, our proposed model treats the graph structure as a latent variable and induces it in an end-to-end fashion. our model is built based on the structured attention @xcite . using a variant of matrix-tree theorem @xcite , our model is able to generate task-specific dependency structures for capturing non-local interactions between entities. we further develop an iterative refinement strategy, which enables our model to dynamically build the latent structure based on the last iteration, allowing the model to incrementally capture the complex interactions for better multi-hop reasoning @xcite . experiments show that our model significantly outperforms the existing approaches on docred, a large-scale document-level relation extraction dataset with a large number of entities and relations, and also yields new state-of-the-art results on two popular document-level relation extraction datasets in the biomedical domain. the code and pretrained model are available at https: //github.com/nanguoshun/lsr 1 . our contributions are summarized as follows: ## model in this section, we present our proposed latent structure refinement (lsr) model for the document-level relation extraction task. our lsr model consists of three components: node constructor, dynamic reasoner, and classifier. the node constructor first encodes each sentence of an input document and outputs contextual representations. representations that correspond to mentions and tokens on the shortest dependency path in a sentence are extracted as nodes. the dynamic reasoner is then applied to induce a document-level structure based on the extracted nodes. representations of nodes are updated based on information propagation on the latent structure, which is iteratively refined. final representations of nodes are used to calculate classification scores by the classifier. ## conclusion we introduce a novel latent structure refinement (lsr) model for better reasoning in the documentlevel relation extraction task. unlike previous approaches that rely on syntactic trees, co-references or heuristics, lsr dynamically learns a documentlevel structure and makes predictions in an end-toend fashion. there are multiple avenues for future work. one possible direction is to extend the scope of structure induction for constructions of nodes without relying on an external parser.