Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion (2022)

Abstract: Document-level relation extraction (DocRE) aims to extract semantic relations among entity pairs in a document. Typical DocRE methods blindly take the full document as input, while a subset of the sentences in the document, noted as the evidence, are often sufficient for humans to predict the relation of an entity pair. In this paper, we propose an evidence-enhanced framework, Eider, that empowers DocRE by efficiently extracting evidence and effectively fusing the extracted evidence in inference. We first jointly train an RE model with a lightweight evidence extraction model, which is efficient in both memory and runtime. Empirically, even training the evidence model on silver labels constructed by our heuristic rules can lead to better RE performance. We further design a simple yet effective inference process that makes RE predictions on both extracted evidence and the full document, then fuses the predictions through a blending layer. This allows Eider to focus on important sentences while still having access to the complete information in the document. Extensive experiments show that Eider outperforms state-of-the-art methods on three benchmark datasets (e.g., by 1.37/1.26 Ign F1/F1 on DocRED).

## introduction relation extraction (re) is the task of extracting semantic relations among entities within a given text, which has abundant applications such as knowledge graph construction, question answering, and biomedical text analysis @xcite @xcite . prior studies mostly focus on predicting the relation between two entity mentions in a single sentence. however, in reality, an entity may have multiple mentions throughout a document. it is also common that a relation can only be inferred given multiple sentences as the figure 1 : a test sample in the docred dataset @xcite , where the i th sentence in the document is marked with [i] at the start. our model correctly predicts @xcite as evidence, and if we only use the extracted evidence as input, the model can predict the relation "country of origin" correctly. unlike typical docre models that blindly take the whole document as input, a human may only need a few sentences to infer the relation of an entity pair. for each entity pair, we define the minimal set of sentences required by human annotators to infer their relation as their evidence sentences. as shown in figure 1 , to predict the relation between "hero of the day" and "the united states", it is sufficient to know that load (the album) was released in the united states from the 1 st sentence, and "hero of the day" is a single of load from the 10 th sentence. in other words, the 1 st and 10 th sentences serve as the evidence to infer this relation. although the 9 th sentence also mentions "the united states", it is irrelevant to this specific relation. including such irrelevant sentences in input might sometimes introduce noise to the model and be more detrimental than beneficial. despite the usefulness of evidence, few prior studies leverage it in a proper way @xcite . in particular, @xcite extracts the evidence sentences together with re but does not utilize them after extraction. besides, it requires human-annotated evidence for training, and also suffers from massive memory usage and training time. another work @xcite trains an re model solely on evidence sentences, which misses important information in the original document and fails to show improvements when paired up with pre-trained language models. in this paper, we propose an evidence-enhanced docre framework eider, which efficiently extracts evidence and effectively leverages the extracted evidence to improve docre. during training, we enhance docre by jointly extracting relations and evidence using multi-task learning, which allows the two tasks to benefit from providing additional training signals for each other. there are two major challenges regarding evidence extraction. the first challenge is the memory and runtime overhead due to training an additional task. for example, a prior multi-task method @xcite ) needs over 14h and three consumer gpus to train, while the individual re model only takes around 90min on one gpu. in comparison, eider uses a simpler evidence extraction model, which can fit into a single gpu and only requires 95min runtime. the second challenge is that human-annotated evidence sentences are costly and heavily relying on them limits model applicability. therefore, we design several heuristic rules to construct silver labels in case the evidence annotation is unavailable. we observe that eider still improves re performance when trained with our silver labels, and sometimes even performs on par with using gold labels. with the evidence extracted, either by our rules or evidence extraction model, we propose to further enhance docre by utilizing the evidence in inference. in the extreme case, if there is only one sentence related to the relation, one can make predictions solely based on this sentence and reduce the problem to sentence-level relation extraction. one naive approach is thus to directly replace the original document with the extracted evidence @xcite . however, since no systems can extract evidence perfectly, solely relying on extracted sentences may miss important information and harm model performance in certain cases (see we conduct extensive experiments on three widely-adopted docre benchmarks: docred @xcite , cdr @xcite and gda @xcite . experiment results show that eider achieves state-of-the-art performance on all the datasets. performance analysis further shows that the improvement of eider is most significant on inter-sentence entity pairs, suggesting that leveraging evidence is especially effective in reasoning over multiple sentences. in particular, eider significantly improves the performance on entity pairs that require co-reference/multi-hop reasoning by 1.98/2.08 f1 on docred, respectively. contributions. (1 ) we propose an efficient joint relation and evidence extraction model that allows the two tasks to mutually enhance each other without heavily relying on evidence annotation. (2) we design a simple and effective docre inference process enhanced by the extracted evidence, enabling more focus on the important sentences with no information loss. (3) we demonstrate that our evidence-enhanced framework outperforms stateof-the-art methods on three docre datasets. ## methodology an illustration of the framework of eider is shown in figure 2 . in training, we jointly extract relation and evidence using multi-task learning, where the two tasks have their own classifier and share the base encoder (sec. 3.1). in inference, we fuse the predictions on the original document and the extracted evidence using a blending layer (sec. 3.2). in case the evidence annotation is not available, we also provide several heuristic rules to construct silver evidence labels as an alternative (sec. 3.3).