On Generalization in Coreference Resolution (2021)

Abstract: While coreference resolution is defined independently of dataset domain, most models for performing coreference resolution do not transfer well to unseen domains. We consolidate a set of 8 coreference resolution datasets targeting different domains to evaluate the off-the-shelf performance of models. We then mix three datasets for training; even though their domain, annotation guidelines, and metadata differ, we propose a method for jointly training a single model on this heterogeneous data mixture by using data augmentation to account for annotation differences and sampling to balance the data quantities. We find that in a zero-shot setting, models trained on a single dataset transfer poorly while joint training yields improved overall performance, leading to better generalization in coreference resolution models. This work contributes a new benchmark for robust coreference resolution and multiple new state-of-the-art results.

## introduction coreference resolution is a core component of the nlp pipeline, as determining which mentions in text refer to the same entity is used for a wide variety of downstream tasks like knowledge extraction @xcite , question answering @xcite , and dialog systems @xcite . as these tasks span many domains, we need coreference models to generalize well. meanwhile, models for coreference resolution have improved due to neural architectures with millions of parameters and the emergence of pretrained encoders. however, model generalization across domains has always been a challenge @xcite @xcite . since these models are usually engineered for a single dataset, they capture idiosyncrasies inherent in that dataset. as an example, ontonotes @xcite , a widely-used general-purpose dataset, provides metadata, like the document genre and speaker information. however, this assumption cannot be made more broadly, especially if the input is raw text @xcite . furthermore, while there are datasets aimed at capturing a broad set of genres @xcite @xcite , they are not mutually compatible due to differences in annotation guidelines. for example, some datasets do not annotate singleton clusters (clusters with a single mention). ideally, we would like a coreference model to be robust to all the standard datasets. in this work, we consolidate 8 datasets spanning multiple domains, document lengths, and annotation guidelines. we use them to evaluate the off-theshelf performance of models trained on a single dataset. while they perform well within-domain (e.g., a new state-of-the-art of 79.3 f1 on litbank), they still perform poorly out-of-domain. to address poor out-of-domain performance, we propose joint training for coreference resolution, which is challenging due to the incompatible training procedues for different datasets. among other things, we need to address (unannotated) singleton clusters, as ontonotes does not include singleton annotations. we propose a data augmentation process to add predicted singletons, or pseudosingletons, into the training data to match the other datasets which have gold singleton annotations. concretely, we contribute a benchmark for coreference to highlight the disparity in model performance and track generalization. we find joint training highly effective and show that including more datasets is almost "free", as performance on any single dataset is only minimally affected by joint training. we find that our data augmentation method of adding pseudo-singletons is also effective. with all of these extensions, we increase the macro average ## datasets we organize our datasets into three types. training datasets (sec. 2.1) are large in terms of number of tokens and clusters and more suitable for training. evaluation datasets (sec. 2.2) are out-of-domain compared to our training sets and are entirely held out. analysis datasets (sec. 2.3) contain annotations aimed at probing specific phenomena. ## results among the longdoc baseline models trained on one of ontonotes, preco, or litbank, we observe a sharp drop in out-of-domain evaluations. the lit-bank model is generally substantially worse than the models trained on ontonotes and preco, likely due to both a smaller training set and a larger domain shift. interestingly, the litbank model performs the best among all models on qbc, which can be attributed to both lb and qbc being restricted to a similar set of markable entity types. meanwhile, all ontonotes-only models perform well on wc and gap, possibly due to the more diverse set of genres within on and because wc also does not contain singletons. for models trained on ontonotes, we find that the addition of speaker tokens leads to an almost 9 point increase on ci, which is a conversational dataset, but has little impact for non-conversational evaluations. surprisingly, the addition of genre embeddings has almost no impact on the overall evaluation. 6 finally, the addition of pseudo-singletons leads to consistent significant gains across almost all the evaluations, including ontonotes. the joint models, which are trained on a combination of ontonotes, litbank, and preco, suffer only a small drop in performance on ontonotes and preco, and achieve the best performance for litbank. like the results observed when training with only ontonotes, we see a significant performance gain with pseudo-singletons in joint training as well, which justifies our intuition that they can bridge the annotation gap. the "joint + ps 30k" model also achieves the state of the art for wc. ## analysis impact of singletons singletons are known to artificially boost the coreference metrics @xcite , and their utility for downstream applications is arguable. to determine the impact of singletons on final scores, we present separate results for singleton and non-singleton clusters in qbc in the poor performance of on models for singletons is expected, as singletons are not seen during training. adding pseudo-singletons improves the performance of both the on and the joint model for singletons. interestingly, adding pseudo-singletons also leads to a small improvement for non-singleton clusters. the pc model has the best performance for nonsingleton clusters while the lb 0 model, which performs the best in the overall evaluation, has the worst performance for non-singleton clusters. this means that the gains for the lb 0 model can be all but attributed to the superior mention detection performance which can be explained by the fact that both lb and qbc are restricted to a similar set of markable entity types. ## related work joint training is commonly used in nlp for training robust models, usually aided by learning dataset, language, or domain embeddings (e.g., @xcite for parsing; @xcite for machine translation). this is essentially what models for ontonotes already do with genre embeddings @xcite . unlike prior work, our test domains are unseen, so we cannot learn test-domain embeddings. for coreference resolution, @xcite augment annotations using relation extraction systems to better incorporate world knowledge, a step towards generalization. @xcite use adversarial training to target names, with improvements on gap. moosavi and strube (2018) incorporate linguistic features to improve generalization to wc. recently, @xcite proposed the ontogum dataset which consists of multiple genres. however, compared to the datasets used in our work, ontogum is much smaller, and is also restricted to a single annotation scheme. to the best of our knowledge, our work is the first to evaluate generalization at scale. missing singletons in ontonotes has been previously addressed through new data annotations, leading to the creation of the arrau @xcite and preco @xcite corpora. while we include preco in this work, arrau contains additional challenges, like split-antecedents, that further increase the heterogeneity, and its domain overlaps with ontonotes. pipeline models for coreference resolution that first detect mentions naturally leave behind unclustered mentions as singletons, although understanding singletons can also improve performance @xcite . recent end-to-end neural models have been evaluated on ontonotes, and therefore conflate "not a mention" with "is a singleton" @xcite @xcite . for datasets with singletons, this has been addressed explicitly through a cluster-based model @xcite . for those without, they can be implicitly accounted for with auxiliary objectives @xcite . we go one step further by augmenting with pseudo-singletons, so that the training objective is identical regardless of whether the training set contains annotated singletons. ## conclusion our eight-dataset benchmark highlights disparities in coreference resolution model performance and tracks cross-domain generalization. our work begins to address cross-domain gaps, first by handling differences in singleton annotation via data augmentation with pseudo-singletons, and second by training a single model jointly on multiple datasets. this approach produces promising improvements in generalization, as well as new state-of-the-art results on multiple datasets. we hope that future work will continue to use this benchmark to measure progress towards truly general-purpose coreference resolution.