Denoising Relation Extraction from Document-level Distant Supervision (2020)

Abstract: Distant supervision (DS) has been widely adopted to generate auto-labeled data for sentence-level relation extraction (RE) and achieved great results. However, the existing success of DS cannot be directly transferred to more challenging document-level relation extraction (DocRE), as the inevitable noise caused by DS may be even multiplied in documents and significantly harm the performance of RE. To alleviate this issue, we propose a novel pre-trained model for DocRE, which de-emphasize noisy DS data via multiple pre-training tasks. The experimental results on the large-scale DocRE benchmark show that our model can capture useful information from noisy data and achieve promising results.

## introduction relation extraction (re) aims to identify relational facts between entities from texts. recently, neural relation extraction (nre) models have been verified in sentence-level re @xcite . distant supervision (ds) @xcite provides large-scale distantly-supervised data that multiplies instances and enables sufficient model training. sentence-level re focuses on extracting intrasentence relations between entities in a sentence. however, it is extremely restricted with generality and coverage in practice, since there are plenty of inter-sentence relational facts hidden across multiple sentences. statistics on a large-scale re dataset constructed from wikipedia documents show that at least 40.7% relational facts can only be inferred from multiple sentences @xcite . therefore, document-level relation extraction (docre) is proposed to jointly extract both inter-and intra-sentence relations @xcite . fig. 1 gives a brief illustration of docre. most docre models heavily rely on high-quality human-annotated training data, which is timeconsuming and labor-intensive. however, it is extremely challenging to extend the sentence-level ds to the document level. the challenges of conducting document-level ds mainly come from: (1) each entity contains multiple mentions, and mentions without relational context bring noise to entity representations; (2) the inherent noise of ds will be even multiplied at the document level. statistics in @xcite show that 61.8% inter-sentence relation instances generated by document-level ds are actually noise; (3) it is challenging to capture useful relational semantics from long documents, since most contents in the documents may be irrelevant to the given entities and relations. in sentencelevel re, several efforts @xcite have been devoted to denoise the ds corpus by jointly considering multiple instances. however, these denoising methods can not be directly adapted to docre, since they are specially designed for bag-level re evaluations. in this work, we attempt to introduce documentlevel ds to docre after denoising. to alleviate the noise, we propose a pre-trained model with three specially designed tasks to denoise the document-level ds corpus and leverage useful information. the three pre-training tasks include: (1) mention-entity matching, which aims to capture useful information from multiple mentions to produce informative representations for entities. it consists of intra-document and inter-document sub-tasks. the intra-document sub-task aims to match masked mentions and entities within a document to grasp the coreference information. the inter-document sub-task aims to match entities between two documents to grasp the entity association across documents. (2) relation detection, which focuses on denoising "not-a-relation (na)" and incorrectly labeled instances by detecting the entity pairs with relations, i.e., positive instances. it is specially designed as the document-level denoising task. we also conduct a pre-denoising module trained with this task to filter out na instances before pre-training. (3) relational fact alignment, which requires the model to produce similar representations for the same entity pair from diverse expressions. this allows the model to focus more on diverse relational expressions and denoising irrelevant information from the document. in experiments, we evaluate our model on an open docre benchmark and achieve significant improvement over competitive baselines. we also conduct detailed analysis and ablation test, which further highlight the significance of ds data and verify the effectiveness of our pre-trained model for docre. to the best of our knowledge, we are the first to denoise document-level ds with pre-trained models. we will release our codes in the future. ## methodology in this section, we present our proposed model in detail. fig. 2 gives an illustration of our framework. we first apply the pre-denoising module to screen out some na instances from all documents. then we pre-train the document encoder with three pre-training tasks on the document-level distantly supervised dataset. finally, we fine-tune the model on the human-annotated dataset. ## conclusion in this work, we propose to denoise distantly supervised data in docre by multiple pre-training tasks. experiment results verify the effectiveness of our model. in the future, we will explore how to improve the efficiency of our pre-training.