Improving Span Representation for Domain-adapted Coreference Resolution (2021)

Abstract: Recent work has shown fine-tuning neural coreference models can produce strong performance when adapting to different domains. However, at the same time, this can require a large amount of annotated target examples. In this work, we focus on supervised domain adaptation for clinical notes, proposing the use of concept knowledge to more efficiently adapt coreference models to a new domain. We develop methods to improve the span representations via (1) a retrofitting loss to incentivize span representations to satisfy a knowledge-based distance function and (2) a scaffolding loss to guide the recovery of knowledge from the span representation. By integrating these losses, our model is able to improve our baseline precision and F-1 score. In particular, we show that incorporating knowledge with end-to-end coreference models results in better performance on the most challenging, domain-specific spans.

## introduction recent work has achieved high performance on coreference resolution in standard benchmark datasets like ontonotes @xcite @xcite . however, in many real world settings where coreference resolution would be valuable, text differs greatly from these standard datasets. for example, coreference resolution over clinical notes can enable tracking a patient's progress and treatment history. however, clinical notes contain acronyms and medical terminology. annotating new training data for every domain of interest is expensive and time-consuming, and coreference models trained on existing benchmark datasets perform worse on other domains @xcite @xcite . in this work, we develop a domainadaptation model for coreference resolution that requires only a small number of target training examples and target domain knowledge. our primary approach involves incorporating domain-knowledge into the span representations learned by an end-to-end neural system @xcite . a span representation is a vector representation of a contiguous set of tokens. when determining if a given mention refers to an antecedent, span representations are used by the model to (1.) select a set of candidate mentions and (2.) select an antecedent from the candidates for the given mention. thus, a high-quality span representation encodes the semantic meaning of the span tokens and their local context. @xcite introduced span-bert, a pre-training method extending bert, designed to improve performance on span-selection tasks that involves masking contiguous spans rather than tokens. span representations are derived by concatenating the pre-trained transformer outputs at the boundary tokens with an attention-weighted vector over the span tokens. these representations are fed into a coreference resolution model, thus integrating spanbert into an end-to-end coreference resolution system. spanbert is able to capture coreference structure implicitly in rich span representations. the expressiveness of the spanbert representation is apparent from extrinsic coreference performance, but also through probing tasks that have shown that span representations can capture headedness, coreference arcs, and other linguistic features of coreference @xcite . the best coreference performance and span representations are obtained by training the end-to-end model with spanbert using labeled coreference data. when adapting a coreference model to a new domain, fine-tuning or continued training can greatly improve performance, but this approach can be computationally expensive and requires a large amount of labelled documents from the target domain @xcite . neural models have also been criticized for largely relying on shallow heuristics in the text, suggesting this data-driven learning method requires many target examples to learn a new target distribution @xcite . the presence of out-of-vocabulary words in a new domain can create additional challenges. span-bert uses wordpiece tokenization, which can lead to misleading meaning representation for spans when a single wordpiece belongs to spans with different meanings @xcite . consider for example, the spans euthmyia and dementia, both of which are common medical terminology but out-of-vocabulary words for span-bert, which tokenizes them to contain a common wordpiece: "##ia". as described by @xcite , this can lead to a coreference model incorrectly predicting the spans coreferent, since the suffix "##ia" is commonly associated with diseases. a coreference model could correct this by learning a more meaningful representation for the prefix tokens and downweighting the suffix "##ia" token, but this would take many target domain training examples. instead, we propose a more efficient method for integrating domain-specific knowledge into spanbert-based span representations, which requires only a small number of target training samples and leverages domain-specific concept knowledge. we take a set of spans with some similarity in meaning to be a concept, and we use concepts of varying granularity (e.g., problem, headache). first, we introduce a retrofitting loss ( §2.4) which guides the representation learning of span pairs to satisfy a knowledge-based distance function. this distance function reflects pairwise span relationships from our concept knowledge. as a result, we are able to align the target domain coreference structure encoded in the span representation with the global meanings of the spans. this allows the end-to-end model to more efficiently build more meaningful span representations for the target domain. we also introduce an auxiliary scaffolding loss ( §2.5) for a concept prediction task in order to ensure that knowledge relevant to the coreference task can be recovered from the span representation. this usage of an auxiliary task to produce a useful inductive bias was introduced in @xcite to add a syntactic labeling loss for coref-erence resolution since syntactic constituents are often coreferent. spans belonging to the same concept within our concept knowledge usually corefer, so we generalize this technique to a broad, knowledge-based lexicon in our domain adaptation setting. while our retrofitting loss integrates relative knowledge into the span representation, we are able to supplement the span representation with global meaning using the scaffolding loss. to evaluate our models, we take ontonotes as our source domain and the i2b2/va corpus of clinical notes as our target domain. we train our model on the source domain and 200 examples from the target domain, we evaluate model performance on the target domain. our knowledge concepts are from the i2b2/va dataset and the umls metathesaurus. first, we describe our methodology and introduce our retrofitting and scaffolding loss functions in §2. then, we describe our experiments to quantify model performance on our target domain in §3, and finally we demonstrate the performance improvement over our baseline and on rare/oov spans §4. the main contribution of this work is in developing methods to integrate concept knowledge into coreference resolution systems to improve domain adaptation. we outperform our baseline primarily by improving precision, and in doing so, we demonstrate the utility of a set of knowledge concepts for adapting span representations to a new domain. we show that our model's performance does not deteriorate on highly domain-specific spans containing ood frequently subdivided vocabulary. ## model the objective of coreference resolution is to identify a set of coreference clusters from a document containing entities, where each coreference cluster contains mentions referencing a single entity. the set of mention candidates are referred to as spans, where a span is any contiguous set of tokens in the document. we can use the coreference clusters {c 1 , c 2 , . . . , c m } to define an unconnected graph where the set of spans are the vertices @xmath0, s 2 , . . . , s n }, and there are edges only between spans that belong to the same cluster. first, we describe the basic model setup for a neural coreference resolution system, and we then describe our proposed approach, that involves two new losses. figure 1 : span representations produced by spanbert for mentions "euthymia" and "dementia" are fed into a co-reference resolution model. the baseline @xcite , which uses a single coreference loss, cl, produces similar span representations for "euthymia" and "dementia" (top). when we incorporate knowledge concepts, c k 1 = {euthymia, pyrexia, . . .}, c k 2 = {dementia, polymyalgia}, . . ., into the span representation using our proposed losses sl and rl, the span representations for "euthmyia" and "dementia" are further apart (bottom), accurately reflecting that "euthmyia" is a symptom, while "dementia"is a disease, even though they share a wordpiece "ia". ## results for successful domain adaptation, our model aims to demonstrate the value of incorporating concept knowledge. we evaluate overall coreference performance improvement as a result of the span retrofitting and auxiliary concept identification task in §4.1. then, we inspect whether representations for highly domain-specific spans are better for the coreference resolution task in our model than in the baseline ( §4.2). ## related work prior to the introduction of end-to-end neural models in @xcite , coreference resolution for the clinical domain used pipelined approaches, allowing for the propagation of errors from other core nlu tasks and relying on hand-crafted rules for there has been some limited work in domain adaptation for coreference resolution. @xcite adapts a model trained on the muc-6 and ace 2005 datasets to the biomedical domain using an active learning approach, applying data augmentation and pruning techniques. @xcite propose a feature-based active learning method to learn cross-domain knowledge. unlike these works, we take advantage of the modern expressive power of the spanbert representation. with the introduction of spanbert, there was a marked performance improvement for several nlu tasks including coreference resolution. @xcite showed that spanbert could be fine-tuned to perform well on several datasets, e.g., glu and ace @xcite . however, @xcite found that span-bert coreference resolvers generally rely more on mentions than context, so they are susceptible to small perturbations (e.g., changing all the names/nominal mentions in the test set). more generally, for nlu tasks several studies have found that neural models rely heavily on shallow heuristics in the text rather than learning the underlying structure of the linguistic phenomenon @xcite , leading to the misinterpretation of context @xcite . this poses a challenge for adapting a coreference model to noisy domains like clinical text and mostly ood spans. with the recent success of finetuning for domain adaptation, a natural approach would be to fine-tune the spanbert representation, the coreference model, or both @xcite . fine-tuning the pretrained spanbert method alone can be expensive - @xcite showed that the best performing scheme requires 180k documents. in settings where there are fewer documents available in the target domain, it is still possible to fine-tune a coreference model with spanbert @xcite . however, a persisting challenge with adapting the span representation is associated with the wordpiece tokenization employed by spanbert. for highly technical domain-specific language, it is natural that there is a higher average number of subwords per span, since it is unlikely that many spans or spans split only once belong to the limited 300,000 word spanbert vocabulary. @xcite show that wordpiece tokenization in the biomedical domain can lead to misleading span representations. purely fine-tuning approaches fail to address this issue, since the spanbert vocabulary is constant. at the same time, spanbert implicitly learns a geometry that encodes rich information related to the coreference task. @xcite show that it is possible to learn a linear projection space from bert embeddings to capture linguistic phenomena like syntactic dependencies. in @xcite , the authors use a feedforward neural network (ffnn) to probe for properties of coreference structure, finding evidence that the spanbert representation can be used to predict coreference arcs and the head word of the span @xmath1. therefore, it is likely that overall performance greatly depends on the quality of span representation. in our work, we develop multiple techniques to adapt the span representation to a new domain with concept knowledge, allowing the model to be fine-tuned with fewer target domain examples and perform better on highly domain-specific entities. in particular, by incorporating knowledge into the span representation, we are able to restore a global, non-contextual meaning to excessively subtokenized spans. ## conclusion we present methods to efficiently adapt coreference resolution models to a new domain using domainspecific concept knowledge. we demonstrate that we can integrate knowledge into the span representation using two losses to (1) retrofit the span representation to the concept knowledge and (2) ensure that knowledge can be recovered from the span representation using an auxiliary concept identification task. using these methods, we are able to improve the performance of our baseline, especially for highly domain-specific spans.