On Event Individuation for Document-Level Information Extraction (2023)

Abstract: As information extraction (IE) systems have grown more adept at processing whole documents, the classic task of *template filling* has seen renewed interest as a benchmark for document-level IE. In this position paper, we call into question the suitability of template filling for this purpose. We argue that the task demands definitive answers to thorny questions of *event individuation* — the problem of distinguishing distinct events — about which even human experts disagree. Through an annotation study and error analysis, we show that this raises concerns about the usefulness of template filling metrics, the quality of datasets for the task, and the ability of models to learn it. Finally, we consider possible solutions.

## introduction template filling involves extracting structured objects called templates from a document, each of which describes a complex event and its participants. @xcite the task comprises both template detection -determining the number of templates present in a document and their types -as well as slot filling or role-filler entity extraction (ree) -assigning extracted entities to slots in those templates that characterize the roles entities play in the event. since documents routinely describe multiple events, documents in template filling datasets regularly feature multiple templates, often of the same type. correctly individuating events is thus a crucial and challenging component of the task. here, we argue that whereas there is often more than one reasonable way to individuate events, template filling admits no such pluralism, which undermines its value as a benchmark for document-level ie. to support our claim, we draw evidence from several sources: a bomb exploded today in a lima restaurant, and a second device that had been placed in the same establishment was deactivated by the peruvian national police. there were no victims, and the explosion caused very little damage to the restaurant…guerrillas of the tupac amaru revolutionary movement (mrta) have claimed credit for the terrorist act…. figure 1 : an excerpted document from muc-4. does this describe one bombing or two? we argue that this question is confusing and that disagreements about the "right" answer cause problems for document-level ie. more recently, the iarpa better program has released the better granular dataset (soboroff, 2023), which has begun to receive attention (holzenberger et al., 2022; chen et al., 2023). bet-ter also features six template types, but unlike muc-4, each template has a distinct set of slots, which may have other types of fillers beyond entities. we take both datasets as case studies. @xcite event individuation at the core of template filling is the question of event individuation: systems must determine when one event (i.e. template) is being described or several. this is a profoundly difficult problem that long antedates the field of ie, with a rich history in philosophy and linguistics (lemmon, 1967; kim, 1973; quine, 1985; cleland, 1991; davidson, 2001, i.a.). at this level, the problem is well beyond ie's purview, as ie systems plainly should not be made to take a hard line on vexed metaphysical questions. instead, for ie purposes, we might hope to find some rules of thumb for event individuation that agree broadly with human judgments, and that can be consistently applied during annotation. work on template filling itself has had relatively little to say on such rules, although we highlight some relevant discussion from the muc-4 proceedings @xcite details on each dataset can be found in appendix a. in appendix a. rather, the fullest treatment of this subject comes from work on event coreference by hovy et al. (2013), who posit conditions for full event identity -the inverse of the individuation problem. appealing to the notion of a discourse element (de), an event or entity that is referenced in the text, they stipulate that "two mentions fully corefer if their activity/event/state de is identical in all respects, as far as one can tell from their occurrence in the text. (in particular, their agents, location, and time are identical or compatible)." differences in times, locations, and "agents" (i.e. participants) between two event descriptions are intuitively each sufficient conditions for individuating events and, importantly, accord with the guidelines for individuating templates provided for muc-4 and for better (see appendix a). template types offer another, coarser means of distinguishing events. here, we focus analysis on individuation within type, since this is where most of the difficulty lies, and consider whether these guidelines are adequate to enable annotators and models to converge on similar individuation decisions. ## individuation and evaluation to show why individuation matters for template filling, we first consider how the task is evaluated. in contrast to events in event extraction, which are associated with lexical triggers, templates are not anchored to specific lexical items. given a predicted template, there are often multiple gold templates one could use as a reference, and one must therefore establish an alignment (a) between them. all template filling metrics select the alignment that produces the highest total score, summed over aligned reference and predicted templates t r and t p , given a template similarity function ϕ: and given the constraint that each reference template is aligned to at most one predicted template with matching type. @xcite metrics differ mainly in their choice of ϕ. it is this reliance on alignments that makes individuation so critical: if individuation decisions in the reference templates are ill-grounded, this can yield punishingly large differences in score for otherwise semantically similar predictions. we illustrate this with the example document from muc-4 shown in figure 1 and the gold templates in figure 2 . @xcite although arguably only one bombing is described in the text, the annotations feature two templates -one for the bomb that detonated and one for the bomb that was deactivated. @xcite however, a system that predicts only the left template will be heavily penalized for having "missed" the one on the right, even though it identifies all relevant entities and assigns them the correct roles. under the standard alignment-based metric for muc-4, ceaf-ree (du et al., 2021a), such a system would receive a score of only 57% f1. thus, because systems are tasked with reproducing particular (often idiosyncratic) individuation decisions, distinctions without much semantic difference can reverberate as significant differences in score. alignment-based metrics therefore present a serious obstacle in principle to meaningful model comparison. but in practice, we will see that models struggle to individuate effectively at all ( §5). ## individuation replicability one could argue that the metrics per se are not the problem, but rather questionable individuation decisions embodied in the annotations. if those decisions reflect real differences between events, then the difference between predicting one template and predicting two plausibly should correspond to a large difference in score. accordingly, we next investigate whether these decisions are well-founded in muc-4 and bet-ter. to do so, we conduct a re-annotation study of a random subset of documents from each dataset and evaluate agreement between these new annotations and the original ones. if the original annotations reflect sound individuation decisions, then we would expect them to be recoverable with high accuracy by annotators versed in their ontologies. three of the authors of this paper, each an experienced ie researcher, reannotated a subset of documents from the muc-4 and better training splits for template detection only, indicating how many templates of each type are present in each document. documents were selected via stratified sampling, with the goal of obtaining 10 documents annotated for each template type. within type, we further stratified by number of templates. this @xcite other challenging examples are shown in appendix c. @xcite we note the oddity of the missing annotation of second device as a weapon, but this is not important to the argument. we consider the level of agreement among the three annotators alone (α), and when additionally incorporating the gold annotations (α + ). ## do models learn to individuate? given these observations, we now ask what the consequences are for our ie models. even if experts struggle to replicate the original individuation decisions, it is possible that the datasets exhibit some internal consistency, and that a model could therefore learn their template count distributions. to explore this possibility, we consider three recent top-performing models for template filling: iterx (chen et al., 2023), the current state-ofthe-art; gtt (du et al., 2021b), the previous best model; and tempgen (huang et al., 2021), a third competitive system. we investigate the predictions of all three systems on the muc-4 test split, as well as the predictions of iterx on the better test split. @xcite while some prior work has attempted a complete taxonomy of template filling model errors (das et al., 2022), our concern here is exclusively with detection problems. first, we note that all three models make a significant number of detection errors. even the strongest detection model (gtt) still incorrectly predicts the total number of templates on 32% of muc-4 test documents ( second, we observe that the errors of all three models are overwhelmingly system misses: among documents with detection errors, iterx, gtt, and tempgen predicted fewer templates than the reference on 72.5%, 81.3%, and 80.0% of them, respectively ( if this is in fact what these models are learning to do, then collapsing predicted templates into a single aggregate template per type (taking per-slot unions over their fillers) should not substantially alter overall scores. doing this for all model predictions and recomputing scores bears this out: across the three models, there is an average drop of just over a single point in overall score on muc-4 and a mere 0.3 points on better ( ## solutions the value of template filling as an ie benchmark is undermined both by datasets that make dubious individuation decisions and by metrics that sharply penalize deviations from those decisions. so what ## conclusion this work has considered the problem of event individuation -how to distinguish distinct eventsand its ramifications for document-level event extraction. we have shown both that models for this task struggle to replicate gold event counts ( §5) and, more fundamentally, that even human experts frequently make different individuation decisions ( §4). these considerations motivate alternative modes of evaluation for this task ( §6), which we hope are adopted in future work. more generally, in the spirit of hovy et al. (2013), we hope that our work reminds nlp researchers of the linguistic and conceptual complexity of events, and of the need to take this complexity seriously.