Beyond Benchmarks: Building a Richer Cross-Document Event Coreference Dataset with Decontextualization (2025)

Abstract: Cross-Document Event Coreference (CDEC) annotation is challenging and difficult to scale, resulting in existing datasets being small and lacking diversity. We introduce a new approach leveraging large language models (LLMs) to decontextualize event mentions, by simplifying the document-level annotation task to sentence pairs with enriched context, enabling the creation of Richer EventCorefBank (RECB), a denser and more expressive dataset annotated at faster speed. Decontextualization has been shown to improve annotation speed without compromising quality and to enhance model performance. Our baseline experiment indicates that systems trained on RECB achieve comparable results on the EventCorefBank(ECB+) test set, showing the high quality of our dataset and its generalizability on other CDEC datasets. In addition, our evaluation shows that the strong baseline models are still struggling with RECB comparing to other CDEC datasets, suggesting that the richness and diversity of RECB present significant challenges to current CDEC systems.

## introduction cross-document event coreference (cdec) annotation is a complex and labor-intensive process. many restrictions have to be applied to make it feasible. as a result, existing data sets tend to be small and sparsely annotated. in the widely used benchmark dataset for cdec, eventcorefbank (ecb+) @xcite , 95% of annotated events are non-coreferential @xcite . in 88% of all sentences, no events are annotated @xcite . moreover, existing cdec datasets are often unrepresentative and lack expressiveness. many of these datasets show lower referential diversity and ambiguity that allow high lemma match (conll f1 61.9 for ecb+) @xcite . for instance, ecb+ intentionally introduces lindsey lohan admitted to rehab as a new event instance alongside tara reid admitted to rehab to enhance the ambiguity of the event expression admit to rehab. while this approach artificially enhances diversity, which is essential for creating a more representative dataset, it remains limited to only two individuals associated with the event expression admit to rehab. this constraint fails to reflect the natural distribution of diverse events typically found in a collection of news articles. for example, articles discussing protests in hong kong might reference numerous distinct protests, involving various people or organizations across different districts over the span of several months. improved decontextualization accuracy by large language models (llms) paves the way for a more efficient approach to cdec annotation, which we use to create a more representative, and expressive cdec dataset called richer eventcorefbank (recb) with significantly less time. unlike traditional cdec annotation methods, which require annotators read entire articles to gather contextual information to determine if an event pair is coreferent, our approach streamlines the process by reducing the context to a single, self-contained sentence. this simplifies the annotator's task, as now when performing cdec annotation they only need to evaluate pairs of sentences, with participant, time, and location information all included in the sentence, rather than dispersed throughout the article. as shown in figure 1 , by minimizing the amount of text to process, we significantly reduce the cognitive load on the annotator and increase the annotation speed. as a result, recb provides a more scalable and efficient solution for generating large-scale cdec datasets. while our annotation is performed on decontextualized sentences, we maintain the mapping of the event annotations to the original documents, allowing researchers to reconstruct coreference relations in the original text if necessary. we demonstrate the utility of this new dataset by showing that models trained on it can generalize effectively to other test sets. our experiments reveal that cdec models trained on recb achieve comparable performance on the ecb+ test set to models trained on the ecb+ training set itself. our experiments also show that the recb dataset presents a more difficult challenge for existing cdec systems, as fewer restrictions are imposed on recb data selection compared with other datasets. unlike more curated datasets that focus on one topic or a limited set of events, recb encompasses a broader range of event types, temporal and location variations, and participant roles. this results in greater lexical diversity and more referential ambiguity that is characteristic of more realistic data sets, making it harder for models to rely on simple cues like event trigger words. further model improvements will have to come from improvement in capturing the broader context of the events. we also highlight the effectiveness of decontextualization in both facilitating efficient cdec annotation and enhancing model performance in experiments with recb. decontextualization enables the creation of a larger, more scalable dataset, allowing models to be trained on a wider variety of examples with rich representations, thereby improving their robustness and generalization across different domains and tasks. the key contributions of this paper are as follows: the rest of the paper is organized as follows. in §2, we discuss related work on current cdec datasets and modeling, and recent research on decontextualization. in §3, we describe the selection of news articles for recb. in §4, we describe the methods used in our data preparation. in §5, we describe the annotation process for constructing the recb data set. in §6, we describe and analyze the recb data set by performing a statistical comparison with existing cdec data sets. in §7, we assess the performance of a state-of-the-art model on the recb dataset and compare it to its performance on existing datasets. the experimental results are analyzed in §8, followed by our conclusions in §9. ## related work cdec dataset creation previous work on cdec datasets include ecb+ @xcite , fcct @xcite , mean-time @xcite , eer @xcite . when annotating such datasets, annotators must exhaustively compare each event mention in the dataset against all other event mentions across documents to establish coreference relations. this is a laborintensive process and as a result, existing datasets are all relatively small. in our work, by representing events as decontextualized sentences that can stand alone, there is the potential to create cdec datasets on a much larger scale, as annotators only need to examine a pair of sentences instead of two articles to make coreference decisions. a lot of effort has been put into circumventing the scalability issue of manually created data by creating auto-or semi-automatically annotated cdec datasets. gun violence corpus (gvc) @xcite marks event references using a structured database of known gun violence events in a semi-automatic fashion. it considerably improves annotation efficiency and event variation compared to ecb+, but the method does not apply to broader data topics other than gun violence for which such a database does not exist. hyper-coref @xcite and wec-eng @xcite leveraged article hyperlinks in wikipedia data to create data automatically. however, there is no guarantee that the events marked by the wikipedia contributors will apply a consistent standard when creating such hyperlinks. moreover, they mainly consist of wikipedia-entry worthy or what @xcite call referential event mentions, and do not cover descriptive or anecdotal events that arise in news reports. cdec systems recent advancements in cdec modeling use neural cross encoders for pairwise event mention classification @xcite @xcite @xcite @xcite . these methods generally involve preprocessing steps such as document topic modeling and event argument labeling, followed by the application of neural classifiers to analyze pairs of event mentions. the classification process assigns scores based on the distance between event mentions within specific topics, which are then grouped into coreference event clusters using agglomerative clustering. recent cdec state-of-the-art systems use representation learning of mention pairs @xcite @xcite . @xcite al. (2021 2024 ) induce the model to learn through rationale-centric counterfactual data augmentation. we choose to use the model described in @xcite to evaluate our data set for its ease of use and near state-of-the-art performance. decontextualization event descriptions often span multiple sentences, requiring cdec to interpret events within a broader context. key details like participants, time, or location may be implicit rather than explicitly stated. annotating entire documents can be inefficient, especially when they are too lengthy for annotation or computational models @xcite . to address this, we apply event decontextualization, a method that preserves an event's meaning while making it understandable outside its original context @xcite . other strategies for restoring missing, ellipitical, or underspecified content in text include np enrichment @xcite , and dense paraphrasing @xcite , the latter of which has been used in question answering @xcite , amr generation @xcite and sentence textual similarity @xcite . we are the first to utilize llms for event summarization to decontextualize events for cdec. ## data collection our data collection builds on the dataset of @xcite , which consists of articles covering highly contentious international news with a rich set of events. we incorporated additional data focused on u.s. domestic news to ensure a broader and more diverse range of event types. the resulting dataset includes english news articles covering four topics: al-shifa hospital raid (shifa), putin's 2024 election win (putin), hong kong july 1 protests (hongkong), and kyle rittenhouse acquittal (rittenhouse). as shown in ## methodology the recb data preparation process requires multiple steps to convert a collection of documents into candidate sentence pairs for annotators to work on (figure 2 ). this section describes the technical details in the data preparation pipeline. ## recb annotation in this section, we describe the process for constructing the recb data set, and present the statistics of this dataset. ## dataset analysis we compare recb with the other two widely used cdec datasets. ecb+ @xcite extends the original ecb @xcite by adding more similar and ambiguous events, enhancing the coverage and diversity of ecb. only seminal events and a small number of other event mentions are annotated. the gvc @xcite dataset contains events involved in gun violence incidents (e.g., location, time, victim details) with related news articles. only event mentions related to gun violence are annotated. the three datasets, recb contains the highest number of clusters, despite having a comparable number of positive pairs to ecb+. this suggests that recb features a more fine-grained and densely annotated coreference structure. the increase in clusters is primarily driven by a higher proportion of singleton clusters, where an event mention lacks a coreferential counterpart within the dataset. in contrast, gvc has more positive pairs but from less diverse clusters. a higher fraction of singleton clusters indicates greater event diversity, as recb captures a broader range of distinct events rather than overfitting to a limited set of recurring event types. the lemma-cluster ratio quantifies the number of distinct lemmas within the same cluster. a higher lemma count in recb suggests greater lexical diversity, indicating that clusters encompass a broader range of expressions for each event. the cluster-lemma ratio reflects the referential diversity or ambiguity within the dataset. a higher clusters-per-lemma ratio in recb indicates a broader range of event distinctions for the same lemma. this suggests a higher degree of complexity in event differentiation, making recb particularly well-suited for tasks requiring nuanced cdec. ## experiments we evaluate our proposed recb dataset using two strong cdec baselines: lemma matching and pairwise encoding. model performance is compared across recb, gvc, and ecb+ under a crossevaluation setting. to maintain the integrity of event coreference clusters, we partition documents by topic for train and test splits. specifically, documents from three topics are used for training, while one subtopic from the fourth topic is used for testing, and the other subtopic for validation. during evaluation, test split conll f1 pairwise f1 ecb+ 61.9 9.5 gvc 33.8 36.4 shifa 32.3 6.2 putin 39.2 5.5 hongkong 48.2 4.9 rittenhouse 30.0 5.9 ## conclusion in this paper, we introduce recb, a novel cdec dataset that leverages decontextualization to streamline and scale the annotation process. our experiments demonstrate that recb provides a more representative and expressive dataset compared to existing benchmarks like ecb+ and gvc. models trained on recb achieve competitive performance, highlighting its high quality and strong generalizability across other cdec datasets. in addition, compared to ecb+ and gvc, recb presents greater challenges for identifying corefer-ential events, serving as both a benchmark and a foundation for future model development. additionally, our findings suggest that decontextualization is a highly effective technique for enhancing both annotation efficiency and model performance. this approach offers a scalable solution to meet the growing demands of cdec data in complex real-world scenarios.