MAVEN - ARG : Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation (2024)

Abstract: Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a large-scale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-Arg, which augments MAVEN datasets with event argument annotations, making the first all-in-one dataset supporting event detection, event argument extraction (EAE), and event relation extraction. As an EAE benchmark, MAVEN-Arg offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation; (3) the exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments in document level. Experiments indicate that MAVEN-Arg is quite challenging for both fine-tuned EAE models and proprietary large language models (LLMs). Furthermore, to demonstrate the benefits of an all-in-one dataset, we preliminarily explore a potential application, future event prediction, with LLMs. MAVEN-Arg and codes can be obtained from https://github.com/THU-KEG/MAVEN-Argument.

## introduction conveying information about events is a core function of human languages @xcite @xcite , which highlights event understanding as a major objective for natural language understanding and a foundation for various downstream applications (ding et al., 2015; li et al., 2018a; goldfarb-tarrant et al., 2019; * corresponding author. the persians lost the invasion due to the russian maneuvering around the aras river. huang @xcite . as illustrated in figure 1 , event understanding is typically organized as three information extraction tasks @xcite : event detection (ed), which detects event occurrences by identifying event triggers and classifying event types; event argument extraction (eae), which extracts event arguments and classifies their argument roles; event relation extraction (ere), which analyzes the coreference, temporal, causal, and hierarchical relationships among events. despite the importance of event understanding, a large-scale dataset covering all the event understanding tasks has long been absent. established sentence-level event extraction (ed and eae) datasets like ace 2005 @xcite and tac kbp @xcite @xcite do not involve event relation types besides the basic coreferences. rams @xcite and wikievents @xcite extend eae to the document level but do not involve event relations. ere datasets are mostly developed independently for coreference @xcite , temporal @xcite , causal @xcite @xcite , and subevent @xcite relationships and do not cover event arguments. given annotation challenges from task complexity, these datasets often cover only thousands of events. due to the inconsistent event schemata and data, these datasets cannot be unified. this status quo hinders the development of endto-end event understanding methods and limits the potential for event-based downstream applications. maven @xcite is the largest humanannotated ed dataset, with a high-coverage event schema for general-domain events. based on it, @xcite further annotates the first unified ere dataset maven-ere, which covers all four types of event relationships and has a massive scale with more than one million event relations. building on the sustained efforts of these works over years, we complete the puzzle of an all-in-one event understanding dataset in this work. we construct maven-arg, which provides exhaustive event argument annotations based on maven. beyond finishing an all-in-one event understanding dataset, three main advantages of maven-arg make it a valuable eae benchmark. (1) comprehensive event schema. the original maven schema only defines event types but without argument roles. we engage experts to enhance maven schema with argument roles and to write detailed definitions for them, which help annotators and can also serve as task instructions for prompting large language models. the resulting event schema contains 162 event types, 612 argument roles, and 14, 655 words of definitions, which well cover general-domain events. (2) large data scale. maven-arg comprises 107, 507 event mentions, 290, 613 event arguments, and 129, 126 entity mentions, all of which are human annotated. to our knowledge, this makes it the largest eae dataset currently available. (3) exhaustive annotation. the development of eae has seen many variations in task settings, including annotating only the topic event @xcite of a document or all fine-grained events @xcite , annotating event arguments at the sentence level @xcite or document level @xcite , and limiting event arguments to entities @xcite or including non-entity arguments @xcite . maven-arg adopts the most exhaustive annotation. we annotate event arguments for all finegrained events at the document level, covering both entity and non-entity arguments. this enhances the dataset's utility for benchmarking and developing a wide range of eae methods. in the experiments, we reproduce several recent state-of-the-art eae models as baselines and also evaluate large language models with in-context learning. experimental results show that they can only achieve at most 40% f1 scores, which is far from promising. it indicates that maven-arg is quite challenging and more research efforts are needed to develop practical eae methods. furthermore, to demonstrate the advantage of an all-in-one event understanding dataset for enabling sophisticated event-based applications, we conduct a preliminary exploration of future event prediction. we sample causally related event chains from maven-arg and prompt llms to predict future events, including their types and arguments. experiments show that while most of the predictions are reasonable, they seldom align with the actual future. we encourage future work to further explore this application and hope maven-arg can help improve eae and develop diverse event-based applications. ## dataset construction in this section, we introduce the dataset construction methodology of maven-arg, which involves designing event schema ( § 2.1), annotating entities ( § 2.2) and event arguments ( § 2.3). ## data analysis to provide intuitive descriptions for maven-arg, we conduct data analyses in this section. ## related work event argument extraction datasets since the early muc datasets @xcite , event argument extraction (eae) as a part of event extraction has received widespread attention. to reduce task complexity and provide standardized extraction results, the ace datasets @xcite are designed with a schema covering 33 event types, limiting event argument annotation to entities within the same sentence as the trigger. ace 2005 @xcite has been the most widely used dataset for a long time, and the practice of ace has been broadly adopted. rich ere @xcite expands ace schema to 38 event types and constructs the tac kbp datasets @xcite @xcite . @xcite follows the ace schema to build a multilingual dataset. with the advancement of nlp methods, some works break some of the constraints of ace task definition to construct more practical datasets. rams @xcite , wikievents @xcite , and docee @xcite extends the annotation scope to the whole documents. however, rams and docee only annotate one topic event per document, ignoring fine-grained events within documents. maven @xcite and geneva @xcite both construct high-coverage general event schemata with over 100 event types. maven supports only event detection. geneva extends event arguments to cover non-entity spans but focuses on testing the generalizability rather than developing practical eae methods. its data are repurposed from framenet @xcite examples, which are individual sentences without document structure. maven-arg meticulously designs 612 unique argument roles for maven schema and conducts large-scale exhaustive annotation, which annotates both entity and non-entity arguments for finegrained events at the document level. ## conclusion and future work we introduce maven-arg, an event argument extraction dataset with comprehensive schema, large data scale, and exhaustive annotation. experiments indicate that maven-arg is quite challenging for both fine-tuned eae models and proprietary large language models. together with maven and maven-ere, maven-arg completes an all-inone dataset covering the entire process of event understanding. an application case of future event prediction demonstrates how an all-in-one dataset can enable broad event-based applications. in the future, we will explore constructing multilingual resources under this framework and developing practical eae methods with maven-arg.