New Insights into Cross-Document Event Coreference: Systematic Comparison and a Simplified Approach (2020)

Abstract: Cross-Document Event Coreference (CDEC) is the task of finding coreference relationships between events in separate documents, most commonly assessed using the Event Coreference Bank+ corpus (ECB+). At least two different approaches have been proposed for CDEC on ECB+ that use only event triggers, and at least four have been proposed that use both triggers and entities. Comparing these approaches is complicated by variation in the systems’ use of gold vs. computed labels, as well as variation in the document clustering pre-processing step. We present an approach that matches or slightly beats state-of-the-art performance on CDEC over ECB+ with only event trigger annotations, but with a significantly simpler framework and much smaller feature set relative to prior work. This study allows us to directly compare with prior systems and draw conclusions about the effectiveness of various strategies. Additionally, we provide the first cross-validated evaluation on the ECB+ dataset; the first explicit evaluation of the pairwise event coreference classification step; and the first quantification of the effect of document clustering on system performance. The last in particular reveals that while document clustering is a crucial pre-processing step, improvements can at most provide for a 3 point improvement in CDEC performance, though this might be attributable to ease of document clustering on ECB+.

## introduction cross-document event coreference (cdec) is a clustering problem with a seemingly straightforward objective: assign every event mention in a corpus to exactly one set in which every mention in the set refers to the same real-world event. cdec the data and code for the experiments described herein is available at https://doi.org/10.34703/gzx1-9v95/fqvnqy . for example, consider the statements "yanitza went for a run" and "juan went for a run," describing two distinct events. note that names of the human participants, yanitza and juan, are arguments that distinguish otherwise identical events. the events often also have internal structure: the event trigger contains a light verb construction using went in combination with run. add the complexity that these event mentions might be found in completely different contexts from completely different documents, and this simple example illustrates why event annotation-and the related task of event coreference, cross-document or not-is difficult and prone to error. in seeking to build a cdec system for our own use, we began with a thorough review of prior work. we discovered that prior systems were not well compared or evaluated, and that the performance of the key step of document clustering was often not reported. on the basis of these insights, we developed a system with a focus on simplicity and explainability. we identify issues in the cdec literature that make comparing prior work difficult and suggest best practices to remedy this situation going forward. our system is modeled on the bag of events system described by @xcite , primarily because of its simplicity and strong performance. however, we use a different and significantly smaller feature set to predict pairwise event coreference (4 features instead of 19), we employ a different document clustering scheme independent of gold-standard annotations, we ingest only event trigger annotations (instead of both triggers and entities), and we develop a different event clustering technique while maintaining comparable state-of-the-art performance. the paper proceeds as follows. we begin with an extensive review of the area: the two major corpora, prior work in cdec, as well as some relevant wdec work ( §2). we then describe our own, simplified approach, with careful attention to evaluating all stages of the pipeline ( §3). we discuss our cross-validated results for various scenarios ( §4) and conclude with a list of contributions ( §5). ## simplified approach we based our approach on the bag of events system described by @xcite and discussed above in section 2.3, primarily because of its simplicity and strong performance. however, we made several modifications based on what we learned in our literature survey: