MAVEN - ERE : A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction (2022)

Abstract: The diverse relationships among real-world events, including coreference, temporal, causal, and subevent relations, are fundamental to understanding natural languages. However, two drawbacks of existing datasets limit event relation extraction (ERE) tasks: (1) Small scale. Due to the annotation complexity, the data scale of existing datasets is limited, which cannot well train and evaluate data-hungry models. (2) Absence of unified annotation. Different types of event relations naturally interact with each other, but existing datasets only cover limited relation types at once, which prevents models from taking full advantage of relation interactions. To address these issues, we construct a unified large-scale human-annotated ERE dataset MAVEN-ERE with improved annotation schemes. It contains 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations, which is larger than existing datasets of all the ERE tasks by at least an order of magnitude. Experiments show that ERE on MAVEN-ERE is quite challenging, and considering relation interactions with joint learning can improve performances. The dataset and source codes can be obtained from https://github.com/THU-KEG/MAVEN-ERE.

## introduction the battle of sultanabad occurred on feb. 13, 1812. … the persians won the battle by moving faster than the russians and attacking them near their camp. … in the end, however, the persians lost the invasion due to the russian maneuvering around the aras river. … tics and is insufficient for training sophisticated neural models @xcite . moreover, event relations in these datasets are often incomprehensive. for instance, tb-dense @xcite and matres @xcite only annotate event temporal relations for event pairs within adjacent sentences. (2) absence of unified annotation. naturally, various types of event relations have rich interactions with each other. for example, the cause events must start temporally before the effect events, and the superevents must temporally contain the subevents. the coreference relation is the foundation, and all the other relations are shared among coreferent event mentions. however, as shown in in this paper, we construct maven-ere, the first unified large-scale event relation dataset, based on the previous maven @xcite ) dataset, which is a massive general-domain event detection dataset covering 4, 480 english wikipedia documents and 168 fine-grained event types. as the example in figure 1 , maven-ere makes up the absence of unified annotation by annotating 4 kinds of event relations in the same documents. maven-ere has 103, 193 event coreference chains, 1, 216, 217 temporal relations, 57, 992 causal relations, and 15, 841 subevent relations. to our knowledge, maven-ere achieves the first million-scale human-annotated ere dataset. as shown in as shown in figure 1 , event relations are dense and complex. hence constructing maven-ere requires thorough and laborious crowd-sourcing annotation. to ensure affordable time and resource costs, we further develop a new annotation methodology based on @xcite , which is the only existing annotation scheme supporting all the relation types. specifically, we decompose the overall annotation task into multiple sequential stages, which reduces competence requirements for annotators. the overhead of later stages can also be reduced with the results of previous stages. first, we annotate coreference relations so that the later-stage annotations only need to consider one of all the coreferent event mentions. for temporal relation annotation, we develop a new timeline annotation scheme, which avoids laboriously identifying temporal relations for every event pair like previous works @xcite . this new scheme brings much denser annotation results. for every 100 words, maven-ere has more than 6 times the number of temporal relations as the previous most widely-used dataset ma-tres @xcite . for causal and subevent relation annotation, we set annotation constraints with temporal relations and the relation transitivity to reduce annotation scopes. we develop strong baselines for maven-ere based on a widely-used sophisticated pre-trained language model @xcite . experiments show that: (1) ere tasks are quite challenging and achieved performances are far from promising; (2) our large-scale data sufficiently trains the models and brings performance benefits; (3) considering the relation interactions with straightforwardly joint training improves the performances, which encourages more explorations. we also provide some empirical analyses to inspire future works. ## dataset construction based on the event triggers in maven @xcite , we annotate data for four ere tasks: extracting event coreference, temporal, causal, and subevent relations. for each task, we introduce its definition, the annotation process, and basic statistics of maven-ere compared with its typical existing datasets. for the overall statistic comparisons, please refer to appendix a. ## experiments and analyses to demonstrate the challenges of maven-ere and analyze the potential future directions for ere, we conduct a series of experiments. ## related work since the fundamental role of understanding event relations in nlp, various ere datasets have been constructed. event coreference relations are often covered in event extraction datasets like muc @xcite , ace @xcite and tac kbp @xcite @xcite . besides, some datasets focus on unrestricted coreference resolution and ignore event semantic types, like ontonotes @xcite and ecb datasets @xcite @xcite . following the timeml specification @xcite , established temporal relation datasets like timebank @xcite and tempeval @xcite @xcite have been constructed. however, these works exhibit low annotation agreements and efficiency issues. @xcite develop a multi-axis annotation annotation scheme based on the dense scheme of @xcite to alleviate them, and @xcite propose to anchor the event starting and ending points to specific time. our timeline annotation scheme is inspired by them. based on the temporal understanding, causal relation datasets @xcite @xcite @xcite are developed. to organize events into hierarchies, subevent relation datasets @xcite are collected. however, the scale of these datasets is limited, and different types of relations are rarely integrated into one dataset. some datasets @xcite @xcite @xcite annotate two or three kinds of relations. @xcite and @xcite provide unified annotation schemes for withindocument and cross-document event relations, respectively, but their constructed datasets are also small. we construct maven-ere referring to the guidelines of @xcite . ## conclusion and future work we present maven-ere, a unified large-scale dataset for event coreference, temporal, causal, and subevent relations, which significantly alleviates the small scale and absence of unified annotation issues of previous datasets. experiments show that real-world event relation extraction is quite challenging and may be improved by jointly considering multiple relation types and better modeling long-range dependency. in the future, we will extend the dataset to more scenarios like covering more event-related information and languages.