BEMEAE : Moving Beyond Exact Span Match for Event Argument Extraction (2025)

Abstract: Event Argument Extraction (EAE) is a key task in natural language processing, focusing on identifying and classifying event arguments in text. However, the widely adopted exact span match (ESM) evaluation metric has notable limitations due to its rigid span constraints, often misidentifying valid predictions as errors and underestimating system performance. In this paper, we evaluate nine state-of-the-art EAE models on the RAMS and GENEVA datasets, highlighting ESM’s limitations. To address these issues, we introduce BEMEAE (Beyond Exact Span Match for Event Argument Extraction), a novel evaluation metric that recognizes predictions that are semantically equivalent to or improve upon the reference. BEMEAE integrates deterministic components with a semantic matching component for more accurate assessment. Our experiments demonstrate that BEMEAE aligns more closely with human judgments. We show that BEMEAE not only leads to higher F1 scores compared to ESM but also results in significant changes in model rankings, underscoring ESM’s inadequacy for comprehensive evaluation of EAE.

## introduction event argument extraction (eae) is a complex task in natural language understanding that requires deep comprehension of text to accurately identify and classify event arguments @xcite . as a core component of event extraction-the process of transforming unstructured text into structured event representations-eae plays a critical role in enabling various downstream applications such as narrative event prediction @xcite and news summarization @xcite . given its broad applicability and significance, the precise evaluation and meaningful comparison of eae systems is crucial for advancing research. the most commonly used evaluation metric for eae, exact span match (esm), assesses there are no craters, while [the vehicles] reference have their chassis intact and they have not been severely damaged, which would have been the case from an [airstrike] event_trigger , konashenkov said. on tuesday, the un also retreated from its claims that the [convoy] candidate was hit by military planes. event type: conflict.attack.selfdirectedbattle role: target figure 1 : an example from rams where airstrike is the event trigger and the vehicles is the reference argument for the role target . the standard metric, exact span match (esm) is suboptimal. seven models we work with correctly predict convoy as the target of the airstrike, yet esm classifies it as an error. furthermore, vehicles, chassis, their chassis, or the convoy would also receive no credit with esm despite being correct, as none are an exact span match to the vehicles. model predictions by requiring an exact match between the predicted argument spans (candidates) and human-annotated argument spans (references). while this approach offers a clear and objective measure of performance, it suffers from several notable limitations. @xcite and @xcite analyzed candidate arguments flagged as errors and found that many were not genuine mistakes; humans would often consider these arguments valid for the assigned role, even if they did not exactly align with the reference span. for example, as illustrated in figure 1 , although "convoy" is a valid argument for the role "target" of event "airstrike," esm would incorrectly classify it as an error because it does not exactly match the reference argument, "the vehicles." motivated by these observations and the need for a more comprehensive evaluation framework, we introduce bemeae (beyond exact span match for event argument extraction). 1 bemeae addresses key shortcomings of esm by combining deterministic components to handle textual variations with a semantic matcher that evaluates the meaning of candidate arguments relative to reference arguments. our experiments, conducted on nine models across two datasets, show that bemeae provides a more accurate assessment of system performance and leads to significant changes in system rankings. the main contributions of this paper are: 1. bemeae: we introduce a novel evaluation metric that addresses the limitations of esm by incorporating deterministic components and a semantic matcher. 2. comprehensive evaluation: through extensive experiments on nine eae models across rams and geneva datasets, we demonstrate that (a) bemeae aligns more closely with human judgments than esm, and (b) f1 scores under bemeae reveal uneven improvements across models, providing new insights into model performance and leading to significant changes in system rankings. 3. automatic semantic matching: we evaluate various semantic matching methods, including gpt-4. they obtain a fair agreement with human judgments, enabling scalable evaluations without the need for costly manual annotations. a key advantage of our approach is its adaptability to any event argument extraction framework, without the need for additional costly human assessments or extensive retraining. this flexibility allows it to be seamlessly integrated into existing systems, providing a more accurate evaluation of model performance with minimal overhead. ## background event extraction (ee) is a key task in information extraction that focuses on converting unstructured text into structured representations of events, which are defined by an event ontology. an event ontology provides a shared framework to categorize events and their argument roles, ensuring consistency in how events, triggers, and arguments are identified across systems. to ensure robustness, we use datasets with different ontologies. ee typically consists of two sub-tasks: event detection (ed) and event argument extraction (eae). event detection identifies event triggers which are keywords or phrases that indicate an event (e.g., "airstrike" in figure 1 ) and classifies them into event types (e.g., "conflict.attack.selfdirectedbattle" in figure 1 ). event argument extraction (eae) focuses on identifying arguments associated with the event and assigning specific roles to them. for instance, eae would identify "convoy" as an argument for the "target" role (in figure 1 ). in this paper, we focus specifically on evaluating eae. evaluation involves comparing the human-annotated argument with the model's prediction. we adopt the terminology of @xcite and refer to human-annotated arguments as reference arguments and the model-predicted arguments as candidate arguments. ## related work evaluating models in tasks like text generation, question answering, and event argument extraction remains a significant challenge. traditional methods, often based on exact span or text match metrics, tend to overlook the variability of possible correct candidates. in this work, we focus specifically on evaluating event argument extraction. we begin by reviewing prior research on event extraction evaluation, followed by a discussion of the challenges in directly comparing models for ee and eae. finally, we conclude by examining proposed metrics that address the limitations of exact span matching in other nlp tasks. ## limitations of exact span match the most commonly used definition of correctness in event argument extraction is exact span match (esm), where a candidate argument is considered correct only if both the offsets and role exactly match the reference argument. unless specified otherwise, in this paper, "f1 score" refers to the esm f1 score. despite being a de facto standard, as shown in one limitation arises when identical tokens appear multiple times in a document, as in the "mar-seille" example (limitation 1), where both mentions refer to the same entity and should be treated as correct. however, exact span matching fails to account for this. similarly, candidate arguments with slight token variations -such as articles, determiners, or punctuation (limitation 2a) or non-informative tokens (limitation 2b)-are unfairly penalized for these minor differences. more significant limitations arise with informative variations (limitation 3), where additional details, such as "south korean" in "south korean vehicles," provide correct information without changing the core meaning, yet are penalized for including additional information. in both datasets, we observed that coreferent mentions (e.g., pronouns) were sometimes annotated as reference arguments, even when more informative alternatives, such as names, were available. yet, exact span match fails to account for equivalence between mentions like "clinton" and "she" (limitation 4a), which should be considered interchangeable in context. similarly, non-coreferent mentions, such as the metonymic use of "u.s." (limitation 4b), may also be valid, but exact span match lacks the flexibility to capture these nuances. finally, exact span match struggles with lists (limitation 5), where combining elements or listing them individually conveys the same meaning even though esm fails to indicate so. ## bemeae: beyond exact match for event-argument extraction to address some of the limitations outlined in section 5 and assess their impact on model performance and rankings, we propose a novel evaluation metric for event argument extraction: bemeae. this metric introduces (1) a series of deterministic components to account for textual variations that do not affect the correctness of an argument and (2) a semantic matching component to identify additional correct candidate arguments. while bemeae does not resolve all the limitations observed with esm, we demonstrate that by effectively addressing some of the significant limitations, it provides a more accurate assessment of system performance and leads to significant changes in system rankings. to address this issue, we compare candidate and reference arguments (excluding pronouns) based on their uncased textual content rather than their offsets. ## insights from bemeae in this section, we present the results from evaluating nine eae models using bemeae and compare them to the baseline esm. the incremental f1 improvements from each bemeae component are presented in one of the starkest differences appears in the "breaking & aggregating" component, which leads to no gains for most models. however, for ampere, degree, and bart-gen, it results in an average 2-point f1 increase with rams ( bemeae substantially improves overall f1, but uneven gains lead to rank changes. the uneven gains across bemeae components accumulate, leading to significant shifts in model rankings, as seen in for example, when comparing model ranking based on esm and bemeae human , bart-gen's f1 score increases by 15 percentage points with rams (46% to 61%) and 12 percentage points with geneva (64.5% to 76.5%), boosting its rank from 7th to 2nd in rams and from 9th to 6th in geneva. in contrast, eeqa shows a smaller improvement, gaining 6.1 points with rams and 6.2 points with geneva, maintaining its low position in rams while dropping two ranks in geneva. exact span match does not correlate well with human judgments. if f1 improvements had been consistent across models, esm might still serve as a reasonable evaluation metric for ranking models. however, a comparison of model rankings from esm and bemeae human show only moderate correlation (kendall's τ = 0.44 for rams, 0.67 for geneva; results are presented incrementally, adding the deterministic components one at a time (section 6). two versions of the semantic matcher are also compared: human judgments (tedious but reliable) and gpt-4o with expert instructions and reference shared (automatic but less reliable). notably, bemeae, our metric, reveals significant differences in system improvements, which ultimately lead to changes in system rankings ( bemeae human , achieving a high kendall's τ of 0.94 for both datasets. this highlights that esm is limited in its effectiveness to approximate human judgment. by recognizing more candidate arguments semantically equivalent to the reference, bemeae offers a more accurate evaluation that is more reflective of human judgment, which ultimately changes model rankings and provides a better understanding of how models perform against each other. automated methods for semantic matcher are reliable for ranking, but not for estimating absolute f1. the best automated semantic matcher in our study demonstrated only fair agreement with human judgment, achieving a cohen's kappa (κ) of 0.43 with rams ( however, automated matchers fall short when estimating absolute f1 scores. the gap between bemeae human and bemeae best automated is significant, with models like bart-gen showing f1 differences of 2.5 points in rams and 2.2 points in geneva. this highlights that even the best automated matchers in our study, despite their ranking reliability, fail to capture all valid candidate arguments that a human would recognize as correct. as a result, our best automated matcher cannot provide absolute f1 scores, as a "true" f1 would require either a perfect semantic matcher-which does not yet exist-or human evaluation, such as bemeae human . as demonstrated in our analysis, bemeae is a significant improvement over esm, providing an evaluation that aligns much more closely with human judgments. by moving beyond exact span matching and recognizing a broader range of candidate arguments that humans would consider semantically equivalent to the reference and thus correct, bemeae offers a more realistic and faithful assessment of model performance. ## conclusion we evaluated nine eae models on the rams and geneva datasets, highlighting the limitations of the traditional exact span match (esm) evaluation metric. to address these shortcomings, we introduced bemeae, a novel metric that combines deterministic components with both human and automated semantic matching. our results show that bemeae provides a far more accurate and fair evaluation since it accounts for a broader range of argument representations resulting in significantly shifted model ranks. when compared, ranks based on esm show poor correlation with bemeae human , whereas bemeae best automated achieves strong correlation, presenting a scalable and reliable alternative.