The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation (2023)

Abstract: Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers. (2) The output space discrepancy of different model paradigms makes different-paradigm EE models lack grounds for comparison and also leads to unclear mapping issues between predictions and annotations. (3) The absence of pipeline evaluation of many EAE-only works makes them hard to be directly compared with EE works and may not well reflect the model performance in real-world pipeline scenarios. We demonstrate the significant influence of these pitfalls through comprehensive meta-analyses of recent papers and empirical experiments. To avoid these pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. To help implement these remedies, we develop a consistent evaluation framework OmniEvent, which can be obtained from https://github.com/THU-KEG/OmniEvent .

## introduction event extraction (ee) is a fundamental information extraction task aiming at extracting structural event knowledge from plain texts. as illustrated in figure 1 , it is typically formalized as a two-stage pipeline @xcite . the first subtask, event detection (ed), is to detect the event triggers (keywords or phrases evoking events, e.g., quitting in figure 1 ) and classify their event types (e.g., end-position). (eae), is to extract corresponding event arguments and their roles (e.g., elon musk and its argument role person) based on the first-stage ed results. ## related work traditional methods @xcite @xcite rely on human-crafted features and rules to extract events. most modern ee models automate feature learning with neural networks @xcite @xcite and adopt different model paradigms to model the ee task. the most common classification-based methods view ee as classifying given trigger and argument candidates into different labels @xcite @xcite @xcite @xcite . sequence labeling methods @xcite @xcite @xcite @xcite do ee by labeling every word following a certain tagging schema such as bio @xcite . recently, some works @xcite @xcite @xcite @xcite propose to cast the task formalization of ee into resource-rich machine reading comprehension tasks and adopt the span prediction paradigm to predict the starting and ending positions of event trigger and argument spans. with the development of generative pre-trained language models @xcite @xcite , there have been works @xcite @xcite @xcite @xcite @xcite exploring the conditional generation paradigm to generate sequences indicating ee results. a few previous works @xcite @xcite have noted that data preprocessing discrepancy may influence evaluation results, but they did not especially study its impact with in-depth analyses. to the best of our knowledge, we are the first to study all three kinds of pitfalls of ee evaluation and propose comprehensive remedies for them. ## pitfalls of event extraction evaluation we first introduce our investigation setup for metaanalysis and empirical analysis ( § 3.1). then we analyze the three pitfalls: data preprocessing discrepancy ( § 3.2), output space discrepancy ( § 3.3), and absence of pipeline evaluation ( § 3.4). ## consistent evaluation framework the above analyses show that the hidden pitfalls substantially harm the consistency and validity of ee evaluation. we propose a series of remedies to avoid these pitfalls and develop a consistent evaluation framework, omnievent. omnievent helps to achieve the remedies and eases users of handling the inconspicuous preprocessing and evaluation details. it is publicly released and continually maintained to handle emerging evaluation pitfalls. the suggested remedies include specifying data preprocessing ( § 4.1), standardizing outputs ( § 4.2), and providing pipeline evaluation results ( § 4.3). we further re-evaluate various ee models using our framework and analyze the results in § 4.4. ## conclusion and future work in this paper, we identify three pitfalls of event extraction evaluation, which are data preprocessing discrepancy, output space discrepancy, and absence of pipeline evaluation. meta-analyses and empirical experiments present a huge impact of these pitfalls, which urges the attention of our research community. to avoid the pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. we develop a consistent evaluation framework omnievent, to help future works implement these remedies. in the future, we will continually maintain it to well handle more emerging ee datasets, model paradigms, and other possible hidden evaluation pitfalls.