C od RED : A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild (2021)

Abstract: Existing relation extraction (RE) methods typically focus on extracting relational facts between entity pairs within single sentences or documents. However, a large quantity of relational facts in knowledge bases can only be inferred across documents in practice. In this work, we present the problem of cross-document RE, making an initial step towards knowledge acquisition in the wild. To facilitate the research, we construct the first human-annotated cross-document RE dataset CodRED. Compared to existing RE datasets, CodRED presents two key challenges: Given two entities, (1) it requires finding the relevant documents that can provide clues for identifying their relations; (2) it requires reasoning over multiple documents to extract the relational facts. We conduct comprehensive experiments to show that CodRED is challenging to existing RE methods including strong BERT-based models.

## introduction relation extraction (re), which aims to extract relations between entities from plain text, serves as an essential resource in populating knowledge bases (kbs) from large-scale corpora automatically. existing re systems typically focus on either sentence-level re @xcite @xcite @xcite or document-level re @xcite @xcite , and have achieved promising results on several public benchmarks. however, these works can only extract relational facts from single sentences or documents containing both two target entities, which inevitably limits the coverage of knowledge acquisition. according to our statistics on wikipedia documents, for over 57.6% of the relational facts in wikidata @xcite , the head and tail entities do not co-occur in a single document. this inspires that it is crucial to break through the limitations of document boundaries to acquire knowledge in the wild. in this work, we make an initial step in this direction, presenting the problem of cross-document re (cross-doc re), which requires a re system to infer the relation between two entities by retrieving and reasoning over multiple documents. compared to conventional sentence/document-level re, cross-doc re presents new challenges in two levels of granularity: (1) at the coarse-grained level, given an entity pair, re systems are required to find multiple informative documents for each entity, instead of restricted to the sentence/document containing both entities; (2) on the fine-grained level, re systems are required to perform both intra-and cross-document reasoning in multiple documents and then predict the relations by aggregating information. the challenges come from not only the non-trivial nature of each phase, but also the intrinsic inter-dependence among the phases. fig. 1 shows an example for cross-doc re, in which amun-her-khepeshef and merneptah do not co-appear in a single document. to identify their relation, we need to first retrieve the relevant documents for each entity and then recognize two reasoning text paths in these documents. the first reasoning text path (the documents titled "nefertari" and "memeptah") shows that both target entities are the son of ramesses ii, and the second one indicates that they also share a common sister meritamen. the information of these two reasoning text paths is complementary to each other and suggests the relation between amun-her-khepeshef and merneptah is sibling. although several datasets have been proposed for investigating cross-document reasoning @xcite , there is still no dataset designed for cross-doc re. to facilitate the research, we construct the first humanannotated cross-document relation extraction dataset named as codred, aiming to test the re systems' ability of knowledge acquisition in the wild. codred has the following features: (1) it requires natural language understanding in different granularity, including coarse-grained document retrieval, as well as fine-grained cross-document multi-hop reasoning; to assess codred, we propose two representative solutions based on the strong bert-based re architecture, including (1) a pipeline model that first extracts a relational graph for each document, and then reasons over these graphs to extract the relation; and (2) an end-to-end model that jointly considers text across different documents in text paths to predict the relation. we conduct comprehensive experiments under both closed and open settings on codred. experimental results show that codred is very challenging to the strong bert-based solutions, indicating ample room for further research. ## data collection in this section, we introduce the data collection process of the cross-doc re dataset. given an entity pair (h, t), cross-doc re consists of two stages: (1) document retrieval, which finds multiple relevant documents of given entity pairs h and t from a large-scale corpus d, which could provide clues for identifying their relationship; (2) cross-document reasoning, which reasons over the retrieved documents to predict the relation. to focus on the problem of cross-doc re, we only annotate the relational facts where the composing entities do not co-occur in a single document. as illustrated in fig. 1 , a relational fact can be better inferred through multiple complementary reasoning text paths (i.e., two documents that contain the head and tail entity respectively, and are connected by bridging entities) in the wild. hence, we want to construct a cross-doc re dataset in which each instance contains a relational fact with multiple reasoning text paths as well as strong supervision about supporting evidence. however, it is infeasible for human annotators to label multiple reasoning text paths over documents for a relational fact from scratch. we thus carefully design a principled data collection pipeline for cross-doc re. specifically, we construct co-dred from the english wikipedia and wikidata through three stages: (1) generating distantly supervised annotations from wikipedia documents, which serves as relation recommendations for further human annotations; (2) annotating relations and the corresponding supporting evidence by multiple independent crowdworkers; (3) generating adversarial hard na instances (i.e., entity pairs and text paths that do not express positive relations) to alleviate the reasoning shortcuts in re. here we introduce main procedure of data collection, and we refer readers to the appendix for more details. ## post-processing and benchmarks we first introduce the data split process, including the split of positive and na entity pairs. (1) positive entity pair split. we split the positive entity pairs into training, development and test set, such that there is no overlap in entity names under the same positive relations, to prevent the correlation between relations and entity names. since codred requires natural language understanding in different granularity, we design two benchmark settings to fully evaluate each required capability including (1) document retrieval, and (2) cross-document reasoning. closed setting. in this setting, we test model capabilities in cross-document reasoning. given an entity pair, re models need to extract relations based on the given positive text paths and na text paths. the first challenge comes from intra-and cross-document multi-hop reasoning in each text path. re models need to first resolve complex interactions between entities within long documents, which may require logical, coreference and commonsense reasoning @xcite . then re models have to overcome the semantic gap between documents, and perform cross-document multi-hop reasoning through multiple potential bridging entities (4.7 on average) to establish the relation in each reasoning text path. the second challenge is that re models need to synthesize all information in multiple text paths to obtain the final relation. open setting. this setting fully tests the ability of re in the wild. given a target entity pair, models need to first retrieve relevant documents for the entity pair from full english wikipedia corpus (5, 882, 234 documents in total, 3, 646 reasoning text path candidates for each entity pair on average), then perform cross-document reasoning with the retrieved documents to predict the relation. compared with natural language queries in open domain question answering @xcite , the sparse query information in entity pairs presents unique challenges to document retrieval ability. the second challenge comes from both the quadratic number of potential paths (efficiency), and the finegrained influence of document retrieval on the extraction of relations (effectiveness). ## data analysis in this section, we present data analysis of co-dred, including data statistics, required abilities in our dataset, and cross-document relation instances. data statistics. codred enjoys diversity in open dataset dr cdr idr isr tacred fewrel knowledgenet bc5cdr dialogre docred codred the documents cover a variety of topics, including geography (28.7%), entertainment (19.6%), and society (8.5%), etc. the average length of documents is 2, 416 words, presenting challenges for modeling long text in both efficiency and effectiveness. we refer readers to the appendix for more details. ## baselines in this section, we design baseline models to assess the challenge of codred. in the closed setting, we design two representative baselines that perform cross-document reasoning based on strong architectures, including: (1) a pipeline model that first extracts a relational graph (i.e., graph containing entities and their relations) for each document, and then reasons over these graphs to extract the relation; and (2) an end-to-end model that jointly considers text across different documents in text paths to predict the relation. in the open setting, we first retrieve relevant documents and connect them into text paths, and then perform cross-document reasoning to predict the relation. we refer readers to the appendix for implementation details. ## experiments in this section, we assess the challenges of co-dred in both closed and open benchmark settings. ## related work a variety of re datasets have been constructed to promote the development of re systems in recent years, which can be categorized in two main categories: (1) sentence-level re datasets focus on extracting relations on sentence-level, where the composing entities of a relational fact must co-appear in single sentences, with the relations labeled by human annotators @xcite @xcite 2010; model ent. ctx. auc f1 p@500 entity only 10.46 21.19 21.70 pipeline 12.72 25.46 25.40 17.45 30.54 30.60 end-to-end 41.76 47.33 58.60 47.94 51.26 62.80 cross-document natural language understanding has received increasing interest in recent years. several datasets have been constructed including cross-document question answering @xcite and cross-document summarization @xcite @xcite . in comparison with existing datasets, our dataset is tailored for the task of re with fine-grained path and evidence annotations, and investigates the more open and challenging scenario of knowledge acquisition. ## conclusion in this work, we study the problem of cross-doc re. to facilitate the research for the problem, we present the first human-annotated dataset codred, and propose two representative solutions. experimental results show that codred is challenging for strong re models, indicating ample room for improvement. in this work, we focus on acquiring knowledge from text paths consisting of two documents. in the future, we plan to further explore longer text paths to better facilitate knowledge acquisition in the wild.