REG en: A Reliable Evaluation Framework for Generative Event Argument Extraction (2025)

Abstract: Event argument extraction identifies arguments for predefined event roles in text. Existing work evaluates this task with exact match (EM), where predicted arguments must align exactly with annotated spans. While suitable for span-based models, this approach falls short for large language models (LLMs), which often generate diverse yet semantically accurate arguments. EM severely underestimates performance by disregarding valid variations. Furthermore, EM evaluation fails to capture implicit arguments (unstated but inferable) and scattered arguments (distributed across a document). These limitations underscore the need for an evaluation framework that better captures models’ actual performance. To bridge this gap, we introduce REGen, a Reliable Evaluation framework for Generative event argument extraction. REGen combines the strengths of exact, relaxed, and LLM-based matching to better align with human judgment. Experiments on six datasets show that REGen reveals an average performance gain of +23.93 F1 over EM, reflecting capabilities overlooked by prior evaluation. Human validation further confirms REGen’s effectiveness, achieving 87.67% alignment with human assessments of argument correctness.

## introduction information extraction is a key area in natural language processing @xcite . event argument extraction (eae) is a core information extraction task that transforms text into structured information. as eae identifies and extracts event-specific arguments from texts, it is essential for a wide range of applications such as document understanding @xcite , misinformation detection @xcite , discourse understanding @xcite , pharmacovigilance @xcite . with the emergence of generative models (e.g., llms), eae has gained significant attention in recent years @xcite . however, previous studies @xcite indicate that llms perform poorly on eae tasks. this is largely due to the disconnect between the nature of generative predictions and the exact span-based evaluation method commonly used for eae @xcite . span-based exact matching (em) significantly underestimates the performance of llms as they often predict accurate arguments in surface forms that differ from the ground truth. for example, if the ground truth annotation for a role is 'pain relief', the model might output terms like [alleviates pain, reducing discomfort, analgesia]. depending on the context, all or multiple of these outputs are correct, but none would be accepted by em. even minor variations would result in no match. authors in @xcite highlighted that this problem is even more pronounced when evaluating the arguments composed of information from different parts of the text (scattered arguments) or the arguments that are not directly mentioned (implicit arguments). previous works have attempted to address these issues using embedding-based relaxed matching, which considers two arguments similar if they have high embedding similarity @xcite . however, this approach fails to capture semantically similar arguments with different lexical forms and wrongly classifies arguments with high token overlap as similar @xcite . for example, in figure 2 for the role 'patient concerns', the ground-truth argument 'limited insurance coverage' and the predicted argument 'coverage limitations for fla and cryotherapy' refer to the same issue. due to lexical variation, relaxed matching fails to capture this. in contrast, consider a role 'date' for which ground-truth and predicted arguments are '18 april 2024' and '20 april 2024', respectively. these two arguments are different, but relaxed matching considers them the same due to high token overlap. context is needed when evaluating these arguments. recent work by @xcite used llms as judges to identify similar arguments. this approach requires a large number of inferences, adding significant computational costs. additionally, without human validation, llm-based judgments can produce unreliable results. relying solely on relaxed match or judgebased approaches can overestimate performance by incorrectly classifying non-match arguments as matches, leading to inflated and unreliable model assessments. a detailed analysis of argument correctness by each method is shown in to address these limitations, we introduce re-gen, a reliable evaluation framework for event argument extraction. regen systematically combines the strengths of exact, relaxed, and llmbased matching by maximizing the evaluation reliability while minimizing the computation costs. figure 2 illustrates the framework, and it is structured into four sequential phases: exact match (em), relaxed match (rm), complex match (cm), and alignment with human judgments. the em level filters arguments that match exactly, reducing computational costs for subsequent stages by eliminating obvious matches. this level does not require human evaluation as exact matches indicate perfect agreement with humans. the rm stage identifies arguments that are semantically similar, making evaluation robust to minor syntactic variations. this matching is performed based on the contextual embedding of the arguments. setting up a high embedding similarity threshold ensures higher reliability and minimizes human evaluation. after filtering out exact and relaxed matches, unmatched arguments are carried forward for complex matching. the cm stage captures semantically similar arguments based on context despite lexical and/or syntactic differences. we leverage llm as a judge @xcite to determine argument similarity. finally, in the judgment alignment stage, we propose a novel judgment aligned match (jam) score to factor in the scores from each level to account for misjudgments based on human validation. this framework ensures evaluation accuracy, cost-effectiveness, and better alignment with human judgments. to the best of our knowledge, this is the first systematic evaluation of llms on popular eae datasets. unlike prior studies @xcite that experimented on small test subsets sampled and merged from multiple datasets, we evaluate the complete test sets of the original datasets. this provides a more reliable assessment of llms' performances on these benchmarks and highlights their potential in solving the eae task, which has been previously underestimated. our key contributions are as follows. gen through extensive evaluation using multiple llms on six widely-used eae datasets, including discourseee (sharif et al., 2024), phee (sun et al., 2022), rams (ebner et al., 2020), geneva (parekh et al., 2023), docee @xcite , and wikievents @xcite . the results show an average improvement of 23.93 f1 points across all datasets while reducing inference costs by 41.2% than the llm-as-judge-only approach @xcite . reproducibility: our code, evaluation framework, the judge and alignment datasets, and other relevant resources are available at https://github.com/omarsharif/regen . 2 regen framework ## results significant improvement in f1-score across all datasets: the increase in f1 score for the geneva dataset was 32.49, representing a 238.52% improvement over the standard em evaluation. similar substantial gains were noted in other datasets, such as 26.71 for discourseee and 24.32 for docee. on average, 41.20% of inferences are reduced under the regen framework: our results in figure 3 and regen framework is more reliable (87.67% alignment): regen shows no/minimal errors in the performance under exact match and relaxed match scoring. while there is some overestimation due to misjudgments in the complex match step, our extensive validation indicates an 87.67% alignment with human judgments (see recall is on average higher than precision in all settings: our fine-grained analysis (see tables 12 to 17 ) reveals llms achieve higher recall than precision. such as the gpt-4o model in the docee dataset achieved a jam recall of 68.41 compared to a precision of only 42.12. this indicates while the models are effective in identifying ground-truth arguments, they tend to over-predict, impacting the overall f1-score. in this work, we used a single prompt for all the models and datasets, which might have contributed to this overprediction. future research should focus on pushing the performance through dataset-and model-specific prompting to enhance precision without sacrificing recall. ## related work generative event argument extraction: early studies on event argument extraction (eae) treated it as an extractive or token-level classification task @xcite . these efforts primarily focused on identifying argument spans directly found in the text @xcite . recently, eae has been formulated as a generative task where pre-trained language models are guided with natural language to fill templates or generate arguments @xcite . @xcite argue that this generative formulation better suits real-world applications as it can capture implicit and scattered arguments better. with the emergence of llms, generative model-based argument extraction gained more traction @xcite @xcite to address this, han et al. (2024) adopts a relaxed matching approach, considering arguments similar if their embedding-based similarly exceeds a threshold of 0.5. similarly, sharif et al. (2024) used a threshold of 0.75. however, this approach has limitations. it fails to capture semantically similar arguments with different lexical or syntactic forms and wrongly classifies arguments with high token overlap as similar. thus, performance reported solely on relaxed matching is unreliable. more recently, lu et al. (2024) employed llms to determine argument similarity. nonetheless, this approach incurs significant computational overhead and demands extensive human validation. our re-gen framework combines the strengths of exact, relaxed, and llm-based matching. it systematically reduces misjudgments, computational costs, and the need for human validation. 7 conclusion this paper presents regen, a novel evaluation framework for eae. our extensive experiments and human validation demonstrate its effectiveness, with a 23.93-point gain in average f1 score across six eae datasets, and reliability, with 87.67% alignment with human judgments. we highlight the limitations of current evaluation approaches and illustrate how regen addresses these issues. furthermore, our analysis reveals that previous studies have underestimated the true performance of llms. we believe that regen fills a critical gap in eae research and motivates future work to explore the generative model's capability in solving other information extraction tasks, e.g., relation extraction, entity extraction, and beyond. ## limitations one limitation of this work is that we did not conduct statistical significance testing on the reported results. we chose not to conduct statistical testing for two reasons. first, our goal is not to conclude which model is best but to highlight performance gaps and show how existing evaluation approaches underestimate model performance. the results clearly demonstrate a significant performance gap with exact match evaluation, which is not diminished by the lack of statistical testing. second, performing statistical tests across all datasets and models with multiple runs is time-consuming and prohibitively expensive. for example, averaging over 3 runs would require an additional 320k inferences. another limitation is that we did not optimize prompts for each model. performance could be improved with dataset-and model-specific prompting. however, we chose to focus on benchmarking a wide range of datasets using model-agnostic prompting. conducting a thorough, prompt engineering for every model and dataset in a single study is not feasible. our results show significant performance gains and future work can explore dataset-and model-specific prompts to further enhance performance. additionally, future work can explore few-shot experiments and find optimal prompting strategies for different datasets, such as self-consistency @xcite . we exclude few-shot experiments as they require selecting demonstration examples through trial and error, finding the optimal order of demonstration, and running multiple iterations, which significantly increases the experimental cost and complexity.