Moving on from O nto N otes: Coreference Resolution Model Transfer (2021)

Abstract: Academic neural models for coreference resolution (coref) are typically trained on a single dataset, OntoNotes, and model improvements are benchmarked on that same dataset. However, real-world applications of coref depend on the annotation guidelines and the domain of the target dataset, which often differ from those of OntoNotes. We aim to quantify transferability of coref models based on the number of annotated documents available in the target dataset. We examine eleven target datasets and find that continued training is consistently effective and especially beneficial when there are few target documents. We establish new benchmarks across several datasets, including state-of-the-art results on PreCo.

## introduction starting initially with neurally-learned features @xcite , end-to-end neural models for coreference resolution (coref) @xcite have been developed and imbued with the benefits from contextualized language modeling @xcite and additional pretraining @xcite . at the same time, the number of parameters used in these models have increased, raising questions of overfitting our research to a specific dataset. several studies show that fully-trained neural models on preexisting large datasets do not transfer well to new domains @xcite @xcite , and that rule-based baselines can still be superior @xcite . further, while prior work has analyzed fully-trained models for mention pairs, like gender bias @xcite @xcite , there has not been a comprehensive comparison analyzing transfer across datasets for document-level coref. we bridge the current gap in understanding between the strength of pretrained models in contrast to the value of annotated target data, in light of the strong few-shot capabilities demonstrated by pretrained language models @xcite . while transfer in other nlp tasks have been studied more in-depth, transfer in coref has scarcely been examined despite recent models containing hundreds of millions of parameters. we investigate model transfer across datasets with continued training, in which a fullytrained model on a source dataset is further trained on a small number of target dataset examples @xcite . 1 we contribute the first study of neural coref transfer, showing that continued training is effective on eleven datasets spanning different domains, annotation guidelines, and languages. we find evidence that ontonotes, a widely-used but licenserequiring dataset for benchmarking coref, is no better at model transfer than the freely-available preco. we establish modern benchmarks on several understudied datasets, including state-of-theart results on preco. additionally, we analyze practical considerations regarding model selection, catastrophic forgetting, and parameter sharing. 2 ## coreference resolution entity coreference resolution is the task of finding clusters of mentions within a document that all refer to the same entity. it still remains a difficult challenge in nlp due to several factors like ambiguity @xcite and dependence on real-world knowledge @xcite . there are several large annotated datasets for coreference resolution. annotation guidelines for coref differ across these datasets based on the intended goals of the creators, resulting in differ-dataset example comments ontonotes (general) judging from the americana in @xcite 1 "a wild sheep chase" [kodansha]2, 320 pages, $18.95]3, baby boomers on both sides of the pacific have a lot in common. only coreferring mentions are marked (no singletons). ## methods in this section, we describe the method, model, datasets, and initialization methods used to investigate the effectiveness of continued training. ## experiments and results for a single source model and target dataset, we train several models using a different number of input training examples. the exact details for training set sizes and preprocessing are in appendix b while training details and hardware are in appendix c. we evaluate coreference using the average f 1 between muc, b 3 and ceaf φ 4 , following prior work (pradhan et al., 2012). 7 4.1 how effective is continued training for domain adaptation? continued training figure 1 shows that it is always beneficial to perform continued training on a source model, even if there is a large amount of target data. however, intuitively the differences are most pronounced in low-resource settings (with 10 fully-annotated documents) where it is still possible to adapt a strong model to perform non-randomly. these conclusions for coreference are similar to those drawn by @xcite on the effectiveness of domain-and task-pretraining of encoders for language classification tasks. these findings also support the intuition used by @xcite , who choose preco as a pretraining corpus for arrau. continued training (and finetuning) is a core component of most nlp models, as text embeddings are typically derived from large pretrained models. @xcite find that model adaptation with contextualized word embeddings only requires a small set of partial annotations in the new domain for syntactic parsing. @xcite and @xcite find that pretrained language models can effectively learn a broad suite of sentence-level understanding, translation, and question-answering tasks with just a few examples. we corroborate their findings for a document-level information extraction task, since our models, based on strong pretrained encoders, perform well with just 5 or 10 training documents. ontonotes vs. preco we find that ontonotes (transfer (on)), despite being the benchmark dataset, is on par (or worse) as a pretraining dataset compared to preco (transfer (pc)). one possibility is that because preco annotates for singletons, it is closer to the target datasets that also annotate singletons. this is evident when we compare the mention detection accuracy of the two models coref (gold ment.) figure 1 : each subplot shows the test performance for each model and (english) dataset when trained with a different number of documents. the first and second rows are coreference and mention boundary f 1 in the endto-end setting, while the third row is the coreference f 1 with gold mentions. spanbert is a pretrained encoder, while the spanbert-on encoders are further finetuned on ontonotes by @xcite , with base and large designating its size. unlike these (dashed lines) models for which we initialize the encoder, the transfer models (solid lines) use continued training and initialize the full model with one that has already been trained on a source dataset, either ontonotes (on) or preco (pc). in low-data settings (e.g. litbank or qbcoref at 5 examples). however, we subsequently explore the case when all models are given gold mention boundaries in pretraining, continued training, and testing, which would effectively evaluate just the linker. we find in this case that preco outperforms ontonotes even more on qbcoref, litbank, as well as arrau rst . this suggests preco as a preferred pretraining dataset over ontonotes when there are few annotated documents. ## conclusion we comprehensively examine the transferability of neural coreference resolution models. we explore several model initialization methods across a wide set of domains and languages, and with a different number of training examples, to demonstrate the universal effectiveness of continued training. additionally, this method results in improved performance over prior work on many of these datasets. furthermore, we find that preco can be effectively used for pretraining, suggesting a viable alternative to ontonotes. in our analysis, we find that: given a fixed number of annotated documents, few need to be allocated for model selection; continued training also suffers from catastrophic forgetting; and continued training is effective with partially frozen encoders. this study and its set of benchmarks serve as a reference for future work in coreference resolution model adaptation, especially for scenarios where annotation can be expensive or data may be scarce. the @xmath0, while the @xmath1. note that if we know @xmath2, then we always make the correct "cluster" decision by ignoring it for the remainder of the algorithm, which allows for this simplification. this is different from simply adding an objective maximizing p (m ), since that would incorrectly handle cases when @xmath3. in practice, however, we found that this makes no difference in performance on the task, though pruning spans earlier resulted in a substantially faster model.