Global-to-Local Neural Networks for Document-Level Relation Extraction (2020)

Abstract: Relation extraction (RE) aims to identify the semantic relations between named entities in text. Recent years have witnessed it raised to the document level, which requires complex reasoning with entities and mentions throughout an entire document. In this paper, we propose a novel model to document-level RE, by encoding the document information in terms of entity global and local representations as well as context relation representations. Entity global representations model the semantic information of all entities in the document, entity local representations aggregate the contextual information of multiple mentions of specific entities, and context relation representations encode the topic information of other relations. Experimental results demonstrate that our model achieves superior performance on two public datasets for document-level RE. It is particularly effective in extracting relations between entities of long distance and having multiple mentions.

## introduction relation extraction (re) aims to identify the semantic relations between named entities in text. while previous work @xcite @xcite focuses on extracting relations within a sentence, a.k.a. sentence-level re, recent studies @xcite @xcite have escalated it to the document level, since a large amount of relations between entities usually span across multiple sentences in the real world. according to an analysis on wikipedia corpus @xcite , at least 40.7% of relations can only be extracted on the document level. compared with sentence-level re, documentlevel re requires more complex reasoning, such * corresponding author @xcite . arrows denote intra/inter-sentential relations. as logical reasoning, coreference reasoning and common-sense reasoning. a document often contains many entities, and some entities have multiple mentions under the same phrase of alias. to identify the relations between entities appearing in different sentences, document-level re models must be capable of modeling the complex interactions between multiple entities and synthesizing the context information of multiple mentions. figure 1 shows an example of document-level re. assume that one wants to extract the relation between "surfers riverwalk" in s11 and "queensland" in s1. one has to find that "surfers riverwalk" contains "pacific fair" (from s11), and "pacific fair" (coreference) is located in "queensland" (from s1). this chain of interactions helps infer the inter-sentential relation "located in" between "surfers riverwalk" and "queensland". ## related work re has been intensively studied in a long history. in this section, we review closely-related work. sentence-level re. conventional work addressed sentence-level re by using carefully-designed patterns @xcite , features @xcite and kernels @xcite . recently, deep learning-based work has advanced the state-of-the-art without heavy feature engineering. various neural networks have been exploited, e.g., cnn @xcite , rnn @xcite and gnn @xcite . furthermore, to cope with the wrong labeling problem caused by distant supervision, zeng et al. (2015) adopted piecewise cnn (pcnn), lin et al. (2016); zhang et al. (2017) employed attention mechanisms, and zhang et al. (2019); qu et al. (2019) leveraged knowledge graphs as external resources. all these models are limited to extracting intra-sentential relations. they also ignore the interactions of entities outside a target entity pair. document-level re. as documents often provide richer information than sentences, there has been an increasing interest in document-level re. gu et al. (2017); nguyen and verspoor (2018); gupta et al. (2019); wang et al. (2019) extended the sentencelevel re models to the document level. ye et al. (2020) explicitly incorporated coreference information into language representation models (e.g., bert). zheng et al. (2018); tang et al. (2020) proposed hierarchical networks to aggregate information from the word, sentence and document levels. quirk and poon (2017) proposed the notion of document-level graphs, where nodes denote words and edges incorporate both syntactic dependencies and discourse relations. following this, peng et al. (2017) first splitted a document-level graph into two directed acyclic graphs (dags), then used a graph lstm for each dag to learn the contextual representation of each word, which was concate-context relation rep. … … … … bert document r-gcn s s s e e e m m m m m m s : sentence m : mention e : entity entity global rep. … mention rep. … entity local rep. … … … … … … multiheadattn 0 multiheadattn 1 … … … … entity rep. … relation rep. … … contextattn … … concatenate classifier (a) encoding layer (b) global rep. layer (c) local rep. layer (d) classifier layer weights figure 2: architecture of the proposed model. nated and finally fed to the relation classifier. differently, song et al. (2018) kept the original graph structure and directly modeled the whole documentlevel graph using graph-state lstm. these models only predict the relation of a single mention pair in a document at a time, and ignore multiple mentions of a target entity pair as well as other entities. several models predict the relation of a target entity pair by aggregating the scores of all mention pairs with multi-instance learning. verga et al. (2018) proposed a transformer-based model. later, sahu et al. (2019) switched transformer to gcn. the two models only consider one target entity pair per document, and construct the documentlevel graphs relying on external syntactic analysis tools. christopoulou et al. (2019) built a document graph with heterogeneous types of nodes and edges, and proposed an edge-oriented model to obtain global representations for relation classification. our model differs in further learning entity local representations to reduce the influence of irrelevant information and considering other relations in the document to refine the prediction. recently, nan et al. ( 2020 ) defined a document graph as a latent variable and induced it based on the structured attention. unlike our work, it improves the performance of document-level re models by optimizing the structure of the document graph. besides, a few models @xcite borrowed the reading comprehension techniques to document-level re. however, they require domain knowledge to design question templates, and may perform poorly in zero-answer and multi-answers scenarios @xcite , which are very common for re. ## proposed model we model document-level re as a classification problem. given a document annotated with enti-ties and their corresponding textual mentions, the objective of document-level re is to identify the relations of all entity pairs in the document. figure 2 depicts the architecture of our model, named glre. it receives an entire document with annotations as input. first, in (a) encoding layer, it uses a pre-trained language model such as bert @xcite to encode the document. then, in (b) global representation layer, it constructs a global heterogeneous graph with different types of nodes and edges, and encodes the graph using a stacked r-gcn @xcite to capture entity global representations. next, in (c) local representation layer, it aggregates multiple mentions of specific entities using multi-head attention @xcite to obtain entity local representations. finally, in (d) classifier layer, it combines the context relation representations obtained with self-attention @xcite to make final relation prediction. please see the rest of this section for technical details. ## experiments and results we implemented our glre with pytorch 1.5. the source code and datasets are available online. 1 in this section, we report our experimental results. ## conclusion in this paper, we proposed glre, a global-to-local neural network for document-level re. entity global representations model the semantic information of an entire document with r-gcn, and entity local representations aggregate the contextual information of mentions selectively using multi-head attention. moreover, context relation representations encode the topic information of other relations using self-attention. our experiments demonstrated the superiority of glre over many comparative models, especially the big leads in extracting relations between entities of long distance and with multiple mentions. in future work, we plan to integrate knowledge graphs and explore other document graph modeling ways (e.g., hierarchical graphs) to improve the performance.