Harvesting Events from Multiple Sources: Towards a Cross-Document Event Extraction Paradigm (2024)

Abstract: Document-level event extraction aims to extract structured event information from unstructured text. However, a single document often contains limited event information and the roles of different event arguments may be biased due to the influence of the information source.This paper addresses the limitations of traditional document-level event extraction by proposing the task of cross-document event extraction (CDEE) to integrate event information from multiple documents and provide a comprehensive perspective on events. We construct a novel cross-document event extraction dataset, namely CLES, which contains 20,059 documents and 37,688 mention-level events, where over 70% of them are cross-document. To address the task, we propose a CDEE pipeline that includes 5 steps, namely event extraction, coreference resolution, entity normalization, role normalization and entity-role resolution. Our CDEE pipeline achieves about 72% F1 in end-to-end cross-document event extraction, suggesting the challenge of this task and setting up a benchmark for future research. Our work builds a new line of information extraction research and will attract new research attention.

## introduction in the realm of natural language processing, document-level event extraction (dee) has been a focal area of research, striving to distill structured information from unstructured text. this process typically involves identifying and categorizing events, along with their associated entities and relations, within a single document @xcite . this approach has demonstrated its effectiveness in numerous applications, such as information retrieval @xcite , content summarization @xcite and knowledge graph construction @xcite ). * corresponding author. type sport_event trigger clinched victory date july 11, 1982 mention1 italy role winner mention2 germany role loser ① in an unexpected turn of events during the 1982 fifa world cup, italy clinched victory against germany with a final score of 3-1. the match, held on july 11, became one of the most memorable finals in world cup history. m m type sport_event trigger scoring date 1982 mention1 paolo rossi role player mention2 italy role team ② paolo rossi, italy's forward, played a pivotal role in the 1982 world cup final, scoring the opening goal that set the tone for italy's triumph. rossi's performance was celebrated as a key factor in italy's victory. type sport_event trigger hosted location santiago bernabéu stadium, madrid mention1 italy role winner mention2 germany role loser ③ the 1982 fifa world cup final was hosted in madrid's santiago bernabéu stadium, attracting a crowd of over 90,000 spectators who witnessed italy's historic win over germany. event 1982 fifa world cup type sport_event location santiago bernabéu stadium, madrid date july 11, 1982 document collections mention1 italy role winner mention2 germany role loser mention3 paolo rossi role player figure 1 : an example of cross-document event extraction, where a comprehensive event is obtained from three event mentions in three documents. although significant advancements have been made @xcite @xcite , dee often encounters limitations in terms of the scope and depth of information that it can provide. different documents may present varying perspectives or emphasize different aspects of the same event, leading to a fragmented and sometimes biased understanding when viewed in isolation. specifically, event information may be distributed across multiple documents. as shown in figure 1 , three documents contain different event mentions referring to the same event, where the bottom-left document includes the "date" argument while the top-right document includes the "location" argument. recognizing these limitations, we propose the task of cross-document event extraction (cdee), which categorizes events into mention-level and concept-level. a mention-level event refers to the event defined within single document, while a concept-level event refers to a complete event obtained by integrating information from multiple documents. compared with dee, the key issue that cdee aims to address is the problem of completeness, which means obtaining a complete representation of an event by aggregating event information from multiple documents. after integrating the extraction results of events from multiple documents, merging duplicate information and resolving conflicting information, the whole event can be built. to foster research in this unexplored field, we have constructed a new cross-document event extraction dataset, called cles (crosslinkeventscope). leveraging wikipedia as the information source, we utilized the hyperlinks inside to identify the documents relevant to events and aggregate them into collections. these collections not only encompass multiple perspectives of a single event but also include detailed background information and various viewpoints related to the event. afterwards, we employed a dee tool @xcite to mine event mentions within documents and manually merge event mentions into a complete event. the results in both processes were manually checked to guarantee the annotation quality. ultimately, the cles dataset comprises 9 event types, over 37,688 mention-level events and 3,633 concept-level events, where over 70% are cross-document. besides the dataset, we also contribute a cdee pipeline comprising 5 steps: (1) dee: event mentions as well as related arguments are extracted from individual document. (2) event coreference resolution: event mentions within an event collection are grouped by coreference relations. (3) entity normalization: a third-party entity linking library @xcite is utilized to align entities and then their attributes are standardized. (4) role normalization: the same type of roles across different documents are normalized by a role mapping table. (5) entity-role resolution: extracted results from different documents are aligned and integrated to eliminate repetitive and conflicted content. our experimental results show that the cdee pipeline is able to achieve about 72% f1 for endto-end cross-document event extraction, revealing its effectiveness but also the challenge of this task. in the end, we highlight the contributions of this paper as: ## our pipeline framework for cdee based on our dataset, we propose a new crossdocument event extraction framework, which mainly consists of the following components: event extraction, coreference resolution, entity normalization, role normalization and entity-role resolution. the framework architecture is illustrated in figure 3 . first, the documents in a collection are input into the event extraction module and output independent event extraction results for each document. then, the coreference resolution module eliminates irrelevant events and clusters the event mentions of the same event. subsequently, event arguments are normalized by the entity and role normalization modules respectively. finally, the entity-role resolution module performs deduplication and conflict resolution for the cross-document event extraction results, and yields complete representations of the events in the document collection. ## experiments given that our proposed model follows a pipeline architecture, we designed three experiments to test the performance of each module: document-level event extraction experiment, event coreference resolution experiment, and cross-document event extraction experiment. these three experiments precisely reflect the three most important aspects of our framework. through the document-level event extraction experiment, we can validate the effectiveness of incorporating rst. through the event coreference resolution experiment, we can evaluate the accuracy of our coreference resolution module in removing irrelevant events. through the cross-document event extraction experiment, we are able to show the effectiveness of our pipeline framework. ## conclusion in this paper, we introduced a novel task of crossdocument event extraction. a large-scale dataset, cles, is proposed based on wikipedia and a benchmark pipeline is built for the comparison of follow-up work. experiments show the feasibility and challenges of our task and dataset. our work paves the way for a more complex and comprehensive understanding of events, highlighting the importance of multi-document analysis in capturing real-world events. our work extends the scope of information extraction and will lead a new line of nlp research. zhiling zhang. 2023. harvesttext: a toolkit for text mining and preprocessing. https://github.com/ blmoistawinde/harvesttext . shun zheng, wei cao, wei xu, and jiang bian. 2019. doc2edag: an end-to-end document-level framework for chinese financial event extraction. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 337-346, hong kong, china. association for computational linguistics. tong zhu, xiaoye qu, wenliang chen, zhefeng wang, baoxing huai, nicholas jing yuan, and min zhang. 2022. efficient document-level event extraction via pseudo-trigger-aware pruned complete graph.