Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution (2023)

Abstract: Although recent neural models for coreference resolution have led to substantial improvements on benchmark datasets, it remains a challenge to successfully transfer these models to new target domains containing many out-of-vocabulary spans and requiring differing annotation schemes. Typical approaches involve continued training on annotated target-domain data, but obtaining annotations is costly and time-consuming. In this work, we show that adapting mention detection is the key component to successful domain adaptation of coreference models, rather than antecedent linking. We also show annotating mentions alone is nearly twice as fast as annotating full coreference chains. Based on these insights, we propose a method for efficiently adapting coreference models, which includes a high-precision mention detection objective and requires only mention annotations in the target domain. Extensive evaluation across three English coreference datasets: CoNLL-2012 (news/conversation), i2b2/VA (medical notes), and child welfare notes, reveals that our approach facilitates annotation-efficient transfer and results in a 7-14% improvement in average F1 without increasing annotator time.

## introduction neural coreference models have made substantial strides in performance on standard benchmark datasets such as the conll-2012 shared task, where average f1 has improved by 20% since 2016 @xcite @xcite . modern coreference architectures typically consist of an encoder, mention detector, and antecedent linker. all of these components are optimized end-to-end, using only an antecedent linking objective, so expensive coreference chain annotations are necessary for training @xcite . these results have encouraged interest in deploying models in domains like medicine and child protective services, where a small number of practition-figure 1 : model coreference performance (avg f1) as a function of continued training on limited target domain data requiring varying amounts of annotator time. the source domain is news/conversation (ontonotes) and the target domain is medical notes (i2b2/va). using our method to adapt coreference models using only mentions in the target domain, we achieve strong coreference performance with less annotator time. further, coreference models trained on standard benchmarks are not robust to differences in annotation schemes for new domains @xcite . for example, ontonotes does not annotate singleton mentions, those that do not corefer with any other mention. a system trained on ontonotes would implicitly learn to detect only entities that appear more than once, even though singleton retrieval is often desired in other domains @xcite . also, practitioners may only be interested in retrieving a subset of domain-specific entities. continued training on target domain data is an effective approach @xcite , but it requires costly and time-consuming coreference chain annotations in the new domain @xcite . annotating data in high-stakes domains like medicine and child protective services is particularly difficult, where privacy needs to be preserved, and domain experts have limited time. our work demonstrates that annotating only mentions is more efficient than annotating full coreference chains for adapting coreference models to new domains with a limited annotation budget. first, through timed experiments using the i2b2/va medical notes corpus @xcite , we show that most documents can be annotated for mention detection twice as fast as for coreference resolution ( §3). then, we propose how to train a coreference model with mention annotations by introducing an auxiliary mention detection objective to boost mention precision ( §4). with this auxiliary objective, we observe that fewer antecedent candidates yields stronger linker performance. continuity with previous featurebased approaches @xcite @xcite suggests this relationship between high-precision mention detection and strong coreference performance in low-resource settings extends beyond the architecture we focus on @xcite . we evaluate our methods using english text data from three domains: ontonotes @xcite , i2b2/va medical notes @xcite , a new (unreleased) corpus of child welfare notes obtained from a county-level department of human services (dhs). we experiment with standard benchmarks for reproducibility, but we focus primarily on real-world settings where there is interest in deploying nlp systems and limited capacity for in-domain annotations @xcite . for a fixed amount of annotator time, our method consistently out-performs continued training with target domain coreference annotations when transferring both within or across annotation styles and vocabulary. our primary contributions include: timing experiments showing the efficiency of mention annotations ( §3), and methodology to easily integrate mention annotations ( §4) into a common coreference architecture @xcite . furthermore, to the best of our knowledge, this is the first work to examine coreference resolution in child protective settings. with empirical results demonstrating 7-14% improvements in f1 across 3 domains, we find that our approach for adaptation using mention annotations alone is an efficient approach for practical, real-world datasets. ## timed annotation experiments in §2 we established that adapting just the mention detection component of a coreference model to a new domain can be as effective as adapting both mention detection and antecedent linking. in this section we demonstrate that annotating mentions is approximately twice as fast as annotating full coreference chains. while coreference has been established as a time-consuming task to annotate for domain experts @xcite , no prior work measures the relative speed of mention versus full coreference annotation. our results suggest, assuming a fixed annotation budget, coreference models capable of adapting to a new domain using only mention annotations can leverage a corpus of approximately twice as many annotated documents compared to models that require full coreference annotations. we recruited 7 in-house annotators with a background in nlp to annotate two tasks for the i2b2/va dataset. for the first mention-only annotation task, annotators were asked to highlight spans corresponding to mentions defined in the i2b2/va annotation guidelines. for the second full coreference task, annotators were asked to both highlight spans and additionally draw links between mention pairs if coreferent. all annotators used in-ception @xcite and underwent a 45 minute training session to learn and practice using the interface before beginning timed experiments. 2 in order to measure the effect of document length, we sampled short (~200 words), medium (~500), and long (~800) documents. each annotator annotated four documents for coreference resolution and four documents for mention identification (one short, one medium, and two long, as most i2b2/va documents are long). each document was annotated by one annotator for coreference, and one for mention detection. this annotation configuration maximizes the number of documents annotated (as opposed to the number of annotators per document), which is necessary due to the high variance in style and technical jargon in the medical corpus. in total 28 documents were annotated. agreement is slightly higher for mention detection, albeit differences in agreement for the two tasks are not significant due to the small size of the experiment, agreement is higher for mention detection. although results may vary for different interfaces, we show empirically that mention annotation is faster than coreference annotation. ## model given the evidence that a large benefit of continued training for domain adaptation is concentrated in the mention detector component of the coreference system ( §2.3), and that mention annotations are much faster than coreference annotations ( §3), in this section, we introduce methodology for training a neural coreference model with mention annotations. our approach includes two core components focused on mention detection: modification to mention pruning ( §4.2) and auxiliary mention detection training ( §4.3). we also incorporate an auxiliary masking objective ( §4.4) targeting the encoder. ## experiments we evaluate our model on transferring between data domains and annotation styles. to facilitate reproducibility and for comparison with prior work, we conduct experiments on two existing public data sets. we additionally report results on a new (unreleased) data set, which reflects a direct practical application of our task setup and approach. ## results and analysis ## related work previous work has used data-augmentation and rule-based approaches to adapt coreference models to new annotation schemes with some success @xcite @xcite . in many cases, adapting to new annotation schemes is not enough -performance degradation persists for out-of-domain data even under the same annotation scheme @xcite , and encoders (spanbert) can struggle to represent domain specific concepts well, resulting in poor mention recall @xcite . investigation of the popular @xcite architecture has found that coreference systems generally rely more on mentions than context @xcite , so they are especially susceptible to small perturbations. relatedly, @xcite find that mention detection precision has a strong positive impact on overall coreference performance, which is consistent with findings on pre-neural systems @xcite and motivates our work. despite challenges associated with limiting source domain annotation schema, with enough annotated data, coreference models can adapt to new domains. @xcite show that continued training is effective with at least 100 target documents annotated for coreference. however, it is unclear how costly it would be to annotate so many documents: while @xcite focus on the best way to use annotated coreference target examples, we focus on the most efficient way to spend an annotation budget. a related line of work uses active learning to select target examples and promote efficient use of annotator time @xcite @xcite . however, since these annotations require link information, there is a persistent trade-off in active learning between reading and labeling @xcite . since our method does not require link annotations for adaptation, our annotation strategy circumvents the choice between redundant labeling or reading. ## limitations annotation speed for mention detection and coreference is dependent on many variables like annotation interface, domain expertise of annotators, annotation style, document length distribution. so, while our finding that coreference resolution is approximately 2x slower to annotate than mention detection held for two domains (i2b2, cn), there are many other variables that we do not experiment with. we also experiment with transfer between domains with varying semantic similarity and annotation style similarity. but, our notion of annotation style is narrowly focused on types of mentions that are annotated (i.e. singletons, domain applicationspecific mentions). however, since our method is focused on mention detection, our findings may not hold for transfer to annotation styles with different notions of coreference linking (i.e. split-antecedent anaphoric reference @xcite ). we also focus on one common coreference architecture @xcite with encoder spanbert. however, there have been more recent architectures surpassing the performance of @xcite over benchmark on @xcite . our key finding that transferring the mention detector component can still be adopted. ## ethical concerns we develop a corpus of child welfare notes annotated for coreference. all research in this domain was conducted with irb approval and in accordance with a data-sharing agreement with dhs. throughout this study, the data was stored on a secure disk-encrypted server and access was restricted to trained members of the research team. thus, all annotations of this data were conducted by two authors of this work. while this work is in collaboration with the dhs, we do not view the developed coreference system as imminently deployable. prior to considering deploying, at a minimum a fairness audit on how our methods would reduce or exacerbate any inequity would be required. deployment should also involve external oversight and engagement with stakeholders, including affected families. ## conclusion through timing experiments, new model training procedures, and detailed evaluation, we demonstrate that mention annotations are a more efficient use of annotator time than coreference annotations for adapting coreference models to new domains. our work has the potential to expand the practical usability of coreference resolution systems and highlights the value of model architectures with components that can be optimized in isolation.