T ext EE : Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction (2024)

Abstract: Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied large language models on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TextEE. We believe TextEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research.

## introduction event extraction @xcite has always been a challenging task in the field of natural language processing (nlp) due to its demand for a high-level comprehension of texts. since event extraction benefits many applications @xcite , it has attracted increasing attention in recent years @xcite @xcite @xcite 1 textee benchmark platform is available at https: //github.com/ej0cl6/textee @xcite . however, due to the complicated nature of event extraction datasets and systems, fairly evaluating and comparing different event extraction approaches is not straightforward. recent attempts @xcite point out that the reported scores in previous work might not reflect the true performance in real-world applications because of various shortcomings and issues during the evaluation process. this poses a potential obstacle to the development of robust techniques for research in event extraction. motivated by the evaluation concern, this work aims to establish a standardized, fair, and reproducible benchmark for assessing event extraction approaches. we start by identifying and discussing several significant issues in the current evaluation process. first, we discuss the inconsistency issue caused by discrepant assumptions about data, different preprocessing steps, and the use of external resources. next, we highlight the insufficiency problem of existing evaluation pipelines, which cover limited datasets and rely on fixed data splits, potentially introducing bias when evaluating performance. finally, we emphasize the importance of reproducibility, which indirectly causes the aforementioned inconsistency and insufficiency issues. to address these evaluation concerns, we propose textee, an evaluation platform that covers 16 datasets spanning diverse domains. to ensure fairness in comparisons, we standardize data preprocessing procedures and introduce five standardized data splits. furthermore, we aggregate and re-implement 14 event extraction approaches published in recent years and conduct a comprehensive reevaluation. textee offers the benefits of consistency, sufficiency, reproducibility in evaluation. additionally, we benchmark several large language models (llms) @xcite @xcite for event extraction with textee and show the unsatisfactory performance of llms for this task. based on our reevaluation results and findings, we discuss the role of event extraction in the current era of llms, along with challenges and insights gleaned from textee. specifically, we discuss how event extraction systems can be optional tools for llms to utilize, as well as highlight future challenges, including enhancing generalization, expanding event coverage, and improving efficiency. in summary, our contributions are as follows: (1) we highlight and address the difficulties of fair evaluation for event extraction tasks. (2) we present textee as a benchmark platform for event extraction research and conduct a thorough reevaluation of recent approaches as well as llms. (3) based on our results and findings, we discuss limitations and future challenges in event extraction. ## issues in past evaluation despite a wide range of works in ee, we identify several major issues of the past evaluation. we classify those issues into three categories: inconsistency, insufficiency, and low reproducibility. inconsistency. due to the lack of a standardized evaluation framework, we notice that many studies utilize varied experimental setups while comparing their results with reported numbers in the literature. this leads to unfair comparisons and makes the evaluation less reliable and persuasive. we identify and summarize the underlying reasons as follows: insufficiency. we argue that the existing evaluation process used by the majority of approaches cannot thoroughly evaluate the capabilities of event extraction models due to the following aspects: low reproducibility. because of the complex nature of event extraction tasks, the event extraction models have become increasingly complicated. releasing code and checkpoints for reproducing results has become essential, as many details and tricks need to be taken into account during the reimplementation process. however, many promising approaches do not provide an official codebase @xcite @xcite , which potentially impedes the progress of research in event extraction. ## benchmark and reevaluation to address the issues listed in section 3, we present textee, a framework aiming to standardize and benchmark the evaluation process of event extraction. textee has several advantages as follows. better consistency. we propose a standardized experimental setup for fair comparisons. to avoid the difference caused by variations in python package versions, we use stanza 1.5.0 for tokenization and save all the offsets. our script will load the saved offsets during preprocessing, ensuring that everyone can generate exactly the same data. improved sufficiency. we improve the sufficiency of the evaluation process as follows. most prior works follow lin et al. (2020) and consider trigger f1-score and argument f1-score as the evaluation metrics. specifically, they calculate f1scores regarding the following: (1) ti: if the (start_idx, end_idx) of a predicted trigger match the gold ones. (2) tc: if the (start_idx, end_idx, model task event entity relation pos tags amr verbalization template classification-based models dygie++ (wadden et al., 2019) e2e ✓ ✓ ✓ oneie (lin et al., 2020) e2e ✓ ✓ ✓ amr-ie (zhang and ji, 2021) e2e ✓ ✓ ✓ ✓ eeqa @xcite ed, eae ✓ ✓ rcee @xcite ed, eae ✓ ✓ query&extract @xcite ed, eae ✓ ✓ ✓ tagprime-c @xcite ed, eae ✓ ✓ tagprime-cr @xcite eae ✓ ✓ unist @xcite ed ✓ ✓ cedar @xcite ) ed ✓ ✓ generation-based models degree (hsu et al., 2022) e2e, ed, eae ✓ ✓ ✓ bart-gen (li et al., 2021b) eae ✓ ✓ x-gear (huang et al., 2022b) eae ✓ paie (ma et al., 2022) eae ✓ ✓ ✓ ampere (hsu et al., 2023b) eae ✓ ✓ ✓ ✓ reproducibility. we open-source the proposed textee framework for better reproducibility. additionally, we encourage the community to contribute their datasets and codebases to advance the research in event extraction. ## have llms solved event extraction? given the demonstrated potential of large language models (llms) across various nlp tasks, we discuss their capability in solving event extraction tasks. in contrast to previous studies @xcite , which evaluate a single llm on a single ee dataset, we investigate multiple popular llms across multiple datasets provided by textee. we consider gpt-3.5-turbo as well as some open-source llms that achieve strong performance on chatbot arena @xcite foot_2 , including llama-2-13bchat-hf and llama-2-70b-chat-hf @xcite , zephyr-7b-alpha @xcite , and mixtral-8x7b-instruct @xcite , with vllm framework @xcite . we evaluate them on the pipelined tasks of event detection (ed) and event argument extraction (eae). as part of the prompt, we provide llms with the ## future challenges and opportunities in this section, we discuss the role of event extraction in the current nlp era, as well as some challenges and insights derived from textee. how should we position event extraction in the era of llms? based on the findings in section 5, llms struggle with extracting and comprehending complicated structured semantic concepts. this indicates the need for a dedicated system with specialized design to effectively recognize and extract abstract concepts and relations from texts. we believe that a good event extractor, capable of identifying a wide range of events, could serve as a tool that provides grounded structured information about texts for llms. accordingly, llms can flexibly decide whether they require this information for the following reasoning steps or inference process. to achieve this goal, we expect event extractors to be universal, efficient, and accurate, which introduces the following research challenges. broader event coverage and generalizablity. we anticipate that a strong event extractor can recognize a wide range of events and even identify new event concepts that may not have appeared during training. this requires two efforts: (1) expanding domain coverage in datasets. most existing event extraction datasets suffer from a restricted coverage of event types. for instance, all the datasets incorporated by textee have no more than 200 event types, which is significantly below the amount of human concepts encountered in daily life. although some recent studies have attempted to tackle this issue @xcite , their data often contains label noise and lacks detailed role annotations. we believe that efficiently collecting or synthesizing high-quality data that covers a wild range of events is crucial for enhancing the emerging ability to generalize event recognition. (2) better model design for generalization. most existing event extraction models focus on in-domain performance. therefore, their design can fail when encountering novel events. while exploring prompting in llms shows promise, as discussed in section 5, the re-sults remain unsatisfactory. some recent works @xcite explore learning a unified model across multiple information extraction tasks for improved generalization, but their integration is constrained by limited domains. we expect that textee can serve as a starting point for aggregating diverse datasets and training more robust unified models. enhanced model efficiency. inference time can pose a bottleneck for effective event extraction, especially when the number of event (role) types increases. for instance, well-performing methods in textee (e.g., tagprime and paie) require enumerating all the event (role) types, resulting in multiple times of model inference, which significantly slows down as more events (roles) are considered. similar challenges arise with llms, as we have to prompt them per event. therefore, there is a critical necessity for model designs that not only prioritize performance but also optimize efficiency. ## conclusion in this work, we identify and discuss several evaluation issues for event extraction, including inconsistent comparisons, insufficiency, and low reproducibility. to address these challenges, we propose textee, a consistent, sufficient, and reproducible benchmark for event extraction. we also study and benchmark the capability of five large language models in event extraction. additionally, we discuss the role of event extraction in the current nlp era, as well as challenges and insights derived from textee. we expect textee and our reevaluation results will serve as a reliable benchmark for research in event extraction.