This paper explores the effect that different corpus configurations have on the performance of a coreference resolution system, as measured by MUC, B 3 , and CEAF. By varying separately three parameters (language, annotation scheme, and preprocessing information) and applying the same coreference resolution system, the strong bonds between system and corpus are demonstrated. The experiments reveal problems in coreference resolution evaluation relating to task definition, coding schemes, and features. They also expose systematic biases in the coreference evaluation metrics. We show that system comparison is only possible when corpus parameters are in exact agreement.
REF explore the impact on coreference resolution performance by varying several prominent contextual factors; they measure performance across corpora, languages, annotation schemes and preprocessing.
Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information
ACL
Computer Science