GENEVA : Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles (2023)

Abstract: Recent works in Event Argument Extraction (EAE) have focused on improving model generalizability to cater to new events and domains. However, standard benchmarking datasets like ACE and ERE cover less than 40 event types and 25 entity-centric argument roles. Limited diversity and coverage hinder these datasets from adequately evaluating the generalizability of EAE models. In this paper, we first contribute by creating a large and diverse EAE ontology. This ontology is created by transforming FrameNet, a comprehensive semantic role labeling (SRL) dataset for EAE, by exploiting the similarity between these two tasks. Then, exhaustive human expert annotations are collected to build the ontology, concluding with 115 events and 220 argument roles, with a significant portion of roles not being entities. We utilize this ontology to further introduce GENEVA, a diverse generalizability benchmarking dataset comprising four test suites aimed at evaluating models’ ability to handle limited data and unseen event type generalization. We benchmark six EAE models from various families. The results show that owing to non-entity argument roles, even the best-performing model can only achieve 39% F1 score, indicating how GENEVA provides new challenges for generalization in EAE. Overall, our large and diverse EAE ontology can aid in creating more comprehensive future resources, while GENEVA is a challenging benchmarking dataset encouraging further research for improving generalizability in EAE. The code and data can be found at https://github.com/PlusLabNLP/GENEVA .

## introduction event argument extraction (eae) aims at extracting structured information of event-specific arguments and their roles for events from a pre-defined taxonomy. eae is a classic topic @xcite and elemental for a wide range of applications like building knowledge graphs @xcite , figure 1 : distribution of event types into various abstract event types foot_0 for geneva, ace, ere, rams, and wikievents datasets. we observe that geneva is relatively more diverse than the other datasets. to build a strong comprehensive benchmarking dataset, we first create a large and diverse ontology. creating such an ontology from scratch is timeconsuming and requires expert knowledge. to reduce human effort, we exploit the shared properties between semantic role labeling (srl) and eae @xcite and leverage a diverse and exhaustive srl dataset, framenet @xcite , to build the ontology. through extensive human expert annotations, we design mappings that transform the framenet schema to a large and diverse eae ontology, spanning 115 event types from five different abstract types. our ontology is also comprehensive, comprising 220 argument roles with a significant 37% of roles as non-entities. utilizing this ontology, we create geneva -a generalizability benchmarking dataset for event argument extraction. we exploit the humancurated ontology mappings to transfer framenet data for eae to build geneva. we further perform several human validation assessments to ensure high annotation quality. geneva comprises four test suites to assess the models' ability to learn from limited training data and generalize to unseen event types. these test suites are distinctly different based on the training and test data creation -(1) low resource, (2) few-shot, (3) zero-shot, and (4) cross-type transfer settings. we use these test suites to benchmark various classes of eae models -traditional classificationbased models @xcite @xcite , question-answeringbased models @xcite , and generative approaches @xcite . we also introduce new automated refinements in the low resource state-of-the-art model degree @xcite to generalize and scale up its manual input prompts. experiments reveal that degree performs the best and exhibits the best generalizability. however, owing to nonentity arguments in geneva, degree achieves an f1 score of only 39% on the zero-shot suite. under a similar setup on ace, degree achieves 53%, indicating how geneva poses additional challenges for generalizability benchmarking. to summarize, we make the following contributions. we construct a diverse and comprehensive eae ontology introducing non-entity argument roles. this ontology can be utilized further to develop more comprehensive datasets for eae. in addition, we propose a generalizability evaluation dataset geneva and benchmark various recent eae models. finally, we show how geneva is a challenging dataset, thus, encouraging future research for generalization in eae. event argument extraction models: traditionally, eae has been formulated as a classification problem @xcite . previous classification-based approaches have utilized pipelined approaches @xcite as well as incorporating global features for joint inference @xcite @xcite . however, these approaches exhibit poor generalizability in the low-data setting @xcite . to improve generalizability, some works have explored better usage of label semantics by formulating eae as a question-answering task @xcite @xcite . recent approaches have explored the use of natural language generative models for structured prediction to boost generalizability @xcite @xcite . another set of works transfers knowledge from similar tasks like abstract meaning representation and semantic role labeling @xcite @xcite . degree @xcite is a recently introduced state-of-theart generative model which has shown the best performance in the limited data regime. in our work, we benchmark the generalizability of various classes of old and new models on our dataset. ## ontology creation event annotations start with ontology creation, which defines the scope of the events and their corresponding argument roles of interests. towards this end, we aim to construct a large ontology of diverse event types with an exhaustive set of event argument roles. however, it is a challenging and tedious task that requires extensive expert supervision if building from scratch. to reduce human effort while maintaining high quality, we leverage the shared properties of srl and eae and utilize a diverse and comprehensive srl dataset -framenet to design our ontology. we first re-iterate the eae terminologies we follow ( § 3.1) and then describe how framenet aids our ontology design ( § 3.2). finally, we present our steps for creating the final ontology in § 3.3 and ontology statistics in § 3.4. ## geneva dataset previous eae datasets for evaluating generalizability like ace and ere have limited event diversity and are restricted to entity-centric arguments. to overcome these issues, we utilize our ontology to construct a new generalizability benchmarking dataset geneva comprising four specialized test suites. we describe our data creation process in § 4.1, provide data statistics in § 4.2 and discuss out test suites in § 4.3. ## experimental setup we evaluate the generalizability of various eae models on geneva. we describe these models in § 5.1 and the evaluation metrics in § 5.2. ## results following § 4.3, we organize the main experimental results into limited training data and unseen event data settings. when trained on complete training data, we observe that oneie and query&extract models achieve poor micro f1 scores of just 30.03 and 40.41 while all other models achieve f1 scores above 55. this can be attributed to the inability of their model designs to effectively handle overlapping arguments. 10 due to their inferior performance, we do not include oneie and query&extract in the benchmarking results. we present the full results in appendix h. ## analysis in this section, we provide analyses highlighting the various new challenges introduced by geneva. we discuss the performance of large language models, the introduction of non-entity argument roles, and model performance including time and place argument roles.