Three Sentences Are All You Need: Local Path Enhanced Document Relation Extraction (2021)

Abstract: Document-level Relation Extraction (RE) is a more challenging task than sentence RE as it often requires reasoning over multiple sentences. Yet, human annotators usually use a small number of sentences to identify the relationship between a given entity pair. In this paper, we present an embarrassingly simple but effective method to heuristically select evidence sentences for document-level RE, which can be easily combined with BiLSTM to achieve good performance on benchmark datasets, even better than fancy graph neural network based methods. We have released our code at https://github.com/AndrewZhe/Three-Sentences-Are-All-You-Need .

## introduction the task of relation extraction @xcite focuses on extracting relations between entity pairs in texts, and has played an important role in information extraction. while earlier works focus on extracting relations within a sentence @xcite , recent studies begin to explore re at document level @xcite @xcite , which is more challenging as it often requires reasoning across multiple sentences. compared with sentence level extraction, documents are significantly longer with useful information scattered in a larger scale. however, given a pair of entities, one may only need a few sentences, not the entire document, to infer their relationship; reading the whole document may not be necessary, since it may introduce unrelated information inevitably. as we can see in figure 1 , s[1] is sufficient to recognize finland as the country of espoo, and recognizing the rest two instances requires just 2 sentences as supporting evidence as while the document has 6 sentences, only 1 or 2 sentences form the evidence for each relation instance. our preliminary finding suggests that, instead of taking the entire document as context, a casespecific selection may be more useful to help a model focus on the most relevant and informative evidence. previous studies apply graph neural networks (gnns) for this filtering process @xcite . here, gnns are used to collect relevant information from the entire context through an aggregation scheme @xcite and achieve great performance, but the selection of crucial evidence from documents is still implicit and lacks interpretability. if, as indicated by our pilot study, most entity relationships can be decided with just 1 ∼ 3 evidence sentences, is there a simpler method that can filter the document explicitly while maintaining the crucial information? we take a closer look at how entity pairs are contextually related in the annotated supporting evidence, and find that annotators tend to select sentences that can connect the two entities. we therefore design three heuristic rules to extract a small set of paths from the document, which can be seen as an approximation of the supporting evidence. specifically, the consecutive paths consider the scenario where the head and tail entities are close in the context: if they are within 3 consecutive sentences, we regard these sentences as one path. the multi-hop paths correspond to the entity pairs in distant sentences, which can be bridged via other entities that co-occur with the head entity and tail entity in different sentences. as the third relation in figure 1 shows, finland co-occurs with the espoo cathedral in s[1] and with the ec parish in s[6], which makes it a bridge to connect the espoo cathedral and the ec parish. in this case, s[1] and s[6] compose a multi-hop path. when neither of the above rules applies, we collect all the pairs of sentences where one contains head entity and the other contains tail entity as default paths. by comparing our path set with humanannotated supporting evidence, we find that up to 87.5% of the supporting evidence can be fully covered by our heuristically selected paths. in other words, our straightforward and interpretable rules serve as an effective proxy to select supporting evidence from documents. we further feed our selected paths to a simple neural network model and obtain surprisingly good performance on docred, showing that our selected evidence can retain sufficient information from the entire document to support document-level relation extraction. for document re, the major challenge is that the subject and object involved in a relationship may appear in different sentences. thus, more than one sentence is required to capture the relations. nonetheless, how many sentences from the entire document are required to identify the relationship between an entity pair? to address this question, we analyze the supporting evidence presented in docred. the supporting evidence for a relation instance refers to all the sentences that can be used to decide whether this relation holds between the entity pair, labeled by human annotators @xcite . we further extend our study to two widely used document re datasets, cdr @xcite and gda @xcite , where cdr is manually constructed and gda is distantly supervised. in order to find the minimal number of sentences required, we ask annotators to label a minimal set of sentences that are exactly sufficient to identify an entity relation instance, instead of including all relation-associated sentences as the original do-cred pattern. we randomly select 100 instances respectively from cdr and gda for this further annotation, and the results are shown at the bottom of now our question is how to select the supporting sentences that are sufficient to identify an entity relation instance. intuitively, the supporting evidence should be the sentences that build up the connection between a pair of entities. thus, we aim to extract sentence paths from the head entity to the tail entity to describe how they are connected. as for the simplest case, if there exists one sentence that contains both the head and tail entities, the sentence itself can be seen as a path (the intra-sentence case). for more complex situations where the head and tail entities do not co-occur in one sentence, we define the following 3 types of paths which indicate how the head and tail entities can be possibly related in the context. figure 2 provides a visualization of the three types of paths. consecutive paths previous studies have shown that the majority of inter-sentence relations are often in nearby text @xcite . we thus select the consecutive sentences to form a path when the head and tail entities are in nearby sentences. formally, if one mention of the head entity appears in sentence s i and one mention of the tail entity is in sentence s j , these two sentences along with the sentence in between, i.e., sentence s i+1 , . . . , s j-1 (or s j+1 , . . . , s i-1 when @xmath0. given that no more than 3 sentences would suffice for inference, we limit the length of these consecutive paths to be at most 3, which means |j -i| ≤ 2. note that this definition can be naturally extended to the intra-sentence case where @xmath1. we thus consider the intra-sentence case as a type of the consecutive path. a pair of entities can correspond to multiple consecutive paths since they can be mentioned more than once. multi-hop paths another typical case for intersentence relation instances is the multi-hop relation @xcite . in such cases, the head and tail entities are far from each other in the document but can be connected through bridge entities, just like the entity the espoo cathedral in figure 1 bridges the ec parish and finland in sentence 1 and 6. for these cases, we start from the head entity, go through all the bridge entities, arrive at the tail entity, and select all the corresponding sentences in this route as a path. formally, for the head entity e h and the tail entity e t , the multi-hop relation indicates that there exist a list of bridge entities e b 1 , . . . , e b k such that (e h , e b 1 ), (e b 1 , e b 2 ), . . . , (e b k , e t ) form k + 1 intra-sentence relations respectively in sentence s p 1 , . . . , s p k+1 . following this route, we choose these k +1 sentences as the multi-hop path. given the discovery in §2 that most instances only needs 3 sentences, we restrict k to be at most 2, i.e., with only 1 or 2 bridge entities. it is possible to have several multi-hop paths for a certain pair with different lists of bridge entities. default paths if neither of the aforementioned rules applies, we consider a rough estimate for the evidence with the most relevant sentences. we collect all pairs of sentences where one contains the head entity and the other contains the tail entity as default paths. formally, let {s h 1 , . . . , s hp } and {s t 1 , . . . , s tq } denote the sets of sentences that contain the head entity e h and the tail entity e t , respectively. for this entity pair, we will have p × q default paths {s h 1 , s t 1 }, . . . , {s hp , s tq }. note that this type of paths is extracted only when no paths are found with the previous two patterns. ## comparing with annotated evidence to demonstrate the effectiveness of our heuristic rules, we check the size of our path set on docred and their consistency with the gold supporting evidence. as mentioned in §2, the gold annotation acts as a collection of all related evidence, while each of our extracted paths represents one possible and minimal sentence set. ideally, if the path set is sufficient, all connecting sentences between the entity pair should be successfully captured. in other words, they would be presented via various paths in our path set. therefore, the union of paths is expected to be a superset of the supporting evidence. we use the coverage of the supporting evidence to measure the sufficiency of our path set, which stands for the percentage of instances whose supporting evidence is fully covered by the union of our paths. meanwhile, the total number of paths (#p ath) and union size of the paths (#sent) should also remain at a low standard, so as to avoid redundancy. meanwhile, notice that the union of the three types contains only 2.69 different sentences on average, which means that our methods can filter out up to 2/3 of the original text. also, our method is computationally efficient since only 2.27 paths need to be modeled on average. this demonstrates that our methods form a sufficient and nonredundant estimate for the gold supporting evidence, drastically alleviating the negative impact of irrelevant information. ## experiments to further validate the sufficiency of our selected paths, we perform evaluation on docred by feeding the paths to an re model. while previous works take entire documents as input, we replace the document with our selected paths regarding a given entity pair. intuitively, if the paths can cover all crucial information in the document, we would expect comparable or better performance with identical model architecture, as our paths contain little irrelevant information and may help focus on a few key sentences. setup given a pair of entities, all paths are first extracted as described in §3. since each path corresponds to one possible connection of the head and tail entities, we predict the relations with each path independently and aggregate the results afterwards. for every single path c, we concatenate all sentences in it as one segment [w c 1 , ..., w c m ], where the order of sentences is the same as in the original document. the segment is fed to a bilstm to obtain the contextual embeddings after obtaining the prediction of every path between a given entity pair, we aggregate the predicted results by selecting the most likely predictions: p ij (r) = max c p c ij (r). we use the glove-100 @xcite embedding for the bilstm encoder with hidden size 256. following previous works @xcite , we report the f1 for intra-and intersentence entity pairs along with the overall f1 score as evaluation metrics. ## discussion so far we have shown from experiments the limited number of sentences required to deduce a relation instance. while the interesting results seem unconventional for document re, which features complex inter-sentence relations, it is worth mentioning that possible explanations exist in current works in related fields. the interdisciplinary outlooks may provide helpful insights for community members to understand the causes of the three-sentences phenomenon and revisit the problem of documentlevel relation extraction. linguistic perspective one likely cause of the discussed phenomenon is that the seemingly distant relations are not so difficult given their linguistic form. @xcite mentions that a majority of inter-sentence relation instances are in fact due to co-references (anaphoric expressions or alternative descriptions). in these cases, relations could be considered to be described entirely within one sen-tence but with head or tail entities being referred to indirectly. considering anaphoric expressions are likely to appear in surrounding sentences for the candidate mentions (chowdhury and zweigenbaum, 2013), these findings are directly in line with our observation that consecutive paths could support more than 70% relation instances, and provide evidence for three-sentences phenomenon. cognitive perspective another possible explanation is that the re task is naturally defined within a limited amount of entities and context, given the nature of the human brain. it is widely believed that working memory (wm) @xcite plays a vital role to store and manipulate information in inference tasks @xcite , but the capacity of separate information chunks in wm are often limited to 4 @xcite . as we need to memorize all the separate entities in the inference chain along with their relations, it is natural that we tend to describe a relation within a limited number of sentences, since rendering a relationship with more sentences may cause our wm to exceed its capacity. @xcite show that the success rate of completing a reading task drastically drops if too much information, exceeding the subject's wm capacity, is required for the task. therefore, as the datasets are constructed from natural language, the three-sentences phenomenon in the data may be a common pattern that we (unconsciously) follow for mutual understanding. ## conclusion in this paper, we perform an analysis over 3 document re benchmark datasets, and find that human annotators often use a small number of sentences to extract entity relations in document level. this motivates us to think over which sentences are critical for document re. we carefully design heuristic rules to select informative path sets from entire documents, which can be further combined with a simple bilstm to achieve competitive performance on a benchmark dataset, even better than complex graph-based methods.