Realistic Evaluation Principles for Cross-document Coreference Resolution (2021)

Abstract: We point out that common evaluation practices for cross-document coreference resolution have been unrealistically permissive in their assumed settings, yielding inflated results. We propose addressing this issue via two evaluation methodology principles. First, as in other tasks, models should be evaluated on predicted mentions rather than on gold mentions. Doing this raises a subtle issue regarding singleton coreference clusters, which we address by decoupling the evaluation of mention detection from that of coreference linking. Second, we argue that models should not exploit the synthetic topic structure of the standard ECB+ dataset, forcing models to confront the lexical ambiguity challenge, as intended by the dataset creators. We demonstrate empirically the drastic impact of our more realistic evaluation principles on a competitive model, yielding a score which is 33 F1 lower compared to evaluating by prior lenient practices.

## introduction cross-document (cd) coreference resolution identifies and links textual mentions that refer to the same entity or event across multiple documents. for example, while subsuming the challenges of withindocument (wd) coreference, cd coreference introduces additional unique challenges. most notably, lexical similarity is often not a good indicator when identifying cross-document links, as documents are authored independently. as shown in in this paper, we show that cd coreference models achieve these numbers using overly-permissive evaluation protocols, namely assuming gold entity and event mentions are given, rewarding singletons and bypassing the lexical ambiguity challenge. accordingly, we present more realistic evaluation principles which better reflect model performance in real-world scenarios. first, following well established standards in wd coreference resolution @xcite , we propose that cd coreference models should be also evaluated on predicted mentions. while recent models unrealistically assume that event mentions are given as part of the input, practical application on new texts and domains requires performing coreference on raw text, including automatic mention detection. using predicted mentions raises a subtle point with regards to singletons (entities which are only referenced once). in particular, we observe that ecb+'s inclusion of singletons inaccurately rewards models for predicting them, by conflating the evaluation of mention identification with that of coreference detection. to address this, we propose reporting of singleton identification performance in a separate metric, while reporting coreference results without singletons. second, we find that ecb+ does not accurately reflect real-world scenarios where prominent events can be referenced in documents spanning different subjects and domains. to facilitate its annotation, ecb+ mimics this phenomenon by artificially grouping documents dealing with the same event (e.g., the nomination of sanjay gupta in finally, we show empirically that both of these evaluation practices artificially inflate results. an end-to end model that outperforms state-of-the-art results on previous evaluation settings drops by 33 f1 points when using our proposed evaluation scheme, pointing at weaknesses that future modelling work could explore. ## background in this work, we will examine the evaluation of cd coreference on the popular ecb+ corpus @xcite , constructed as an augmentation of the eecb and ecb datasets @xcite . as exemplified in the ecb+ evaluation protocol largely follows that of conll-2012, perhaps the most popular wd benchmark @xcite , with two major distinctions. first, barring a few notable exceptions @xcite ), foot_0 most recent cd models have unrealistically assumed that gold entity and event mentions are given as part of the input, reducing the task to finding coreference links between gold mentions @xcite @xcite @xcite . second, while singletons are omitted on conll-2012, they are exhaustively annotated in ecb+. in the following section, we present a more realistic evaluation framework for cd coreference, taking into account the interacting distinctions of ecb+. ## realistic evaluation principles in this paper, we suggest that cd coreference models should perform and be evaluated on predicted mentions. to achieve this, in section 3.1, we will introduce the singleton effect on coreference evaluation and propose to decouple the evaluation of mention prediction from coreference resolution. in section 3.2, we will establish guidelines allowing to better assess how models handle the ubiquitous lexical ambiguity challenge in real-world scenarios. ## experiments we show empirically that each of the previous evaluation practices (using gold mentions, singleton in-clusion, and subtopic clustering) artificially inflates the results ( §4.2). as recent cd coreference models are designed to perform on gold mentions ( §2), we cannot use them to set baseline results on predicted mentions. we therefore develop a simple and efficient end-to-end model for cd coreference resolution by combining the successful single document e2e-coref @xcite with common cd modeling approaches. ## conclusion we established two realistic evaluation principles for cd coreference resolution: (1) predicting mentions and ( 2 ) facing the lexical ambiguity challenge. we also set baseline results for future work on our evaluation methodology using a sota model.