Understanding Mention Detector-Linker Interaction in Neural Coreference Resolution (2021)

Abstract: Despite significant recent progress in coreference resolution, the quality of current state-of-the-art systems still considerably trails behind human-level performance. Using the CoNLL-2012 and PreCo datasets, we dissect the best instantiation of the mainstream end-to-end coreference resolution model that underlies most current best-performing coreference systems, and empirically analyze the behavior of its two components: mention detector and mention linker. While the detector traditionally focuses heavily on recall as a design decision, we demonstrate the importance of precision, calling for their balance. However, we point out the difficulty in building a precise detector due to its inability to make important anaphoricity decisions. We also highlight the enormous room for improving the linker and show that the rest of its errors mainly involve pronoun resolution. We propose promising next steps and hope our findings will help future research in coreference resolution.

## introduction coreference resolution identifies mentions in a document that co-refer to the same entity. it is an important task facilitating many applications such as reading comprehension @xcite ) and text summarization @xcite . @xcite proposed the first neural end-toend architecture for coreference resolution. most recent systems use it as a backbone and employ better scoring functions @xcite , pruning procedures @xcite , or token representations @xcite ). 1 despite this usage, little in-depth analysis has been done to better understand the inner workings of such an influential system. @xcite analyzed the effect of the high-order inference, while @xcite and @xcite 1 except @xcite which has not seen wide adoption. respectively examined its generalizability and gender bias. few work has inspected the interaction between its components. @xcite conducted oracle experiments that are related to ours, but without fine-grained control over confounding factors affecting oracle mentions. such an understanding is important: for example, @xcite 's dissection of the then-best classical coreference systems inspired many important follow-up works @xcite @xcite . however, it is unknown if observations on such classical feature-based and often pipelined systems extend to current neural end-to-end models. we consider the best instantiation of this model family, spanbert @xcite ) + c2fcoref @xcite , and investigate the interaction between its two components: mention detector and mention linker. we study how their errors independently or jointly affect the final clustering. using the @xcite and preco @xcite ) datasets, we highlight the low-precision, high-recall nature of the detector. while traditionally only recall is emphasized for the detector as a design decision @xcite , we show huge degradation from noisy mentions and, perhaps surprisingly, increasing the number of candidates considered by the baseline linker only deteriorates the performance. while some classical coreference pipelines focused on detector precision (uryupina, 2009), it is rarely emphasized for current end-toend systems. we hence stress the importance of a precision-recall balance for the detector and demonstrate how pruning hyperparameters, in addition to reducing computational complexity, control this trade-off. however, we show the difficulty of obtaining a precise detector by demonstrating the importance of anaphoricity decisions and the inability of the detector to make them. finally, we highlight the high potential of the linker and that the remaining errors mainly involve pronoun resolution. we hope this work sheds light on the internals of the mainstream coreference system and, with our proposed next steps, catalyze future research. we believe some of our findings may also transfer to other tasks with a similar joint span detection and span (pair) classification architecture, such as srl @xcite , ie @xcite , and entity linking @xcite . see @xcite which subsumes many other tasks under such a span-based framework. data conll-2012 is the most common dataset to test coreference models. however, it lacks singleton mention annotation @xcite . singleton, or non-anaphoric, mentions do not co-refer with other spans, e.g. "the dog" in "[the dog] barks." however, they may become anaphoric in another context, e.g. "[the dog] barks at @xcite ." being a mention is a span's inherent property, while anaphoricity, whether or not a mention co-refers, is context-dependent. we use "all mentions" to refer to the union of singleton and anaphoric mentions. to understand the effect of singleton mentions, we heuristically generate all mentions for conll-12 ( §b) for relevant experiments. we also experiment with preco, a coreference dataset with annotated singleton mentions. we do all analyses on development sets and report dataset statistics in §a. ## precision-recall trade-off for the mention detector traditionally, coreference systems heavily favor recall over precision for the detector @xcite as the linker cannot recover missed mentions. similarly, our c2f-coref system @xmath0. we therefore explore if detector recall is always more important than its precision. if more spans are considered by increasing the max span width l or the number of spans considered per word λ, will the system performance necessarily improve? in the extreme case, if we hypothetically the aggregated importance of precision for all oracles in to analyze how higher detector precision helps the linker, we examine the coreference score the linker assigns to every span-antecedent pair. the anaphoric mention re-trained perfect precision oracle has an average score of -13.0 on conll-12, higher than -15.1 with perfect recall. among only correct span-antecedent pairs, these scores are 11.7 and 7.1, with the same pattern. this indicates that the noise with perfect recall prevents the linker from reliably assigning high coreference scores, even for correct links. the effect of higher coreference scores also shows in that, compared with perfect recall, the perfect precision oracle produces on average larger (4.44 vs. 4.26 entities) and longerdistance (154 vs. 152 tokens spanned) clusters. we also see this effect by examining the amount of improvement with reduced noise in from the linker's increased confidence in assigning coreference scores when not tasked with ignoring non-mentions (and singletons) in noisy candidates. ## conclusion we analyzed the complex interaction between the mention detector and linker in the mainstream coarse-to-fine coreference system. using oracle experiments, we showed that, while detector recall is important, higher anaphoric mention precision would lead to dramatically better linker performance, though achieving this is difficult. we also demonstrated that the oracle linker performance is near perfect and that the vast majority of remaining linker errors besides anaphoricity decisions are about pronoun resolution. we hope these discoveries will help future coreference research.