CD ˆ2 CR : Co-reference resolution across documents and domains (2021)

Abstract: Cross-document co-reference resolution (CDCR) is the task of identifying and linking mentions to entities and concepts across many text documents. Current state-of-the-art models for this task assume that all documents are of the same type (e.g. news articles) or fall under the same theme. However, it is also desirable to perform CDCR across different domains (type or theme). A particular use case we focus on in this paper is the resolution of entities mentioned across scientific work and newspaper articles that discuss them. Identifying the same entities and corresponding concepts in both scientific articles and news can help scientists understand how their work is represented in mainstream media. We propose a new task and English language dataset for cross-document cross-domain co-reference resolution (CDˆ2CR). The task aims to identify links between entities across heterogeneous document types. We show that in this cross-domain, cross-document setting, existing CDCR models do not perform well and we provide a baseline model that outperforms current state-of-the-art CDCR models on CDˆ2CR. Our data set, annotation tool and guidelines as well as our model for cross-document cross-domain co-reference are all supplied as open access open source resources.

## introduction cross-document co-reference resolution (cdcr) is the task of recognising when multiple documents mention and refer to the same real-world entity or concept. cdcr is a useful nlp process that has many downstream applications. for example, cdcr carried out on separate news articles that refer to the same politician can facilitate inter-document sentence alignment required for stance detection and natural language inference models. furthermore, cdcr can improve information retrieval and multi-document summarisation by grouping documents based on the entities that are mentioned within them. recent cdcr work @xcite @xcite has primarily focused on resolution of entity mentions across news articles. despite differences in tone and political alignment, most news articles are relatively similar in terms of grammatical and lexical structure. work based on modern transformer networks such as bert @xcite and elmo @xcite have been pre-trained on large news corpora and are therefore well suited to news-based cdcr @xcite . however, there are cases where cdcr across documents from different domains (i.e. that differ much more significantly in style, vocabulary and structure) is useful. one such example is the task of resolving references to concepts across scientific papers and related news articles. this can help scientists understand how their work is being presented to the public by mainstream media or facilitate fact checking of journalists' work @xcite . a chatbot or recommender that is able to resolve references to current affairs in both news articles and user input could be more effective at suggesting topics that interest the user. finally, it may be helpful for e-commerce companies to know when product reviews gathered from third party websites refer to one of their own listings. the work we present here focuses on the first crossdocument, cross-domain co-reference-resolution (cd 2 cr) use case, namely co-reference resolution between news articles and scientific papers. the objective of cd foot_1 cr is to identify coreferring entities from documents belonging to different domains. in this case co-reference resolution is made more challenging by the differences in language use (lexical but also syntactic) across the dif-ferent domains. specifically, authors of scientific papers aim to communicate novel scientific work in an accurate and unambiguous way by using precise scientific terminology. whilst scientific journalists also aim to accurately communicate novel scientific work, their work is primarily funded by newspaper sales and thus they also aim to captivate as large an audience as possible. therefore journalists tend to use simplified vocabulary and structure, creative and unexpected writing style, slang, simile, metaphor and exaggeration to make their work accessible, informative and entertaining in order to maximise readership @xcite . success at the cd 2 cr task in this setting is dependent on context sensitive understanding of how the accessible but imprecise writing of journalists maps on to precise terminology used in scientific writing. for example, a recent study has found that "convalescent plasma derived from donors who have recovered from covid-19 can be used to treat patients sick with the disease" foot_0 . a news article 2 discussing this work says that "...blood from recovered covid-19 patients in the hope that transfusions...[can help to treat severely ill patients]" . in this example the task is to link 'blood' to 'convalescent plasma' and 'recovered covid-19 patients' to 'donors'. these cross-document, cross-domain coreference chains can be used as contextual anchors for downstream analysis of the two document settings via tasks such as natural language inference, stance detection and frame analysis. the contributions in this paper are the following: ## dataset creation our dataset is composed of pairs of news articles and scientific papers gathered automatically (section 3.1). our annotation process begins by obtaining summaries of the news and science document pairs (extractive news summaries and scientific abstracts, respectively) (section 3.2). candidate coreference pairs from each summary-abstract pair are identified and scored automatically. (section 3.3). candidate co-reference pairs are then presented to human annotators via a bespoke annotation interface for scoring (section 3.4). annotation quality is measured on an ongoing basis as new candidates are added to the system (section 3.5). ## model below we describe several baseline models including state of the art cdcr models that we used to evaluate how well current approaches can be used in our cd 2 cr task setting. ## results and discussion we evaluate each of the model baselines described in section 4 above on the test subset of our cd 2 cr corpus. results are shown in for the purposes of evaluation, we use named entity spans from the manually annotated cd 2 cr as the "gold standard" in all experiments rather than using the end-to-end named entity recognition capabilities provided by some of the models. we evaluate the models using the metrics described by @xcite (henceforth muc) and bagga and baldwin (1998) (henceforth b 3 ). muc f1, precision and recall are defined in terms of pairwise co-reference relationships between each mention. b 3 , f1, precision and recall are defined in terms of presence or absence of specific entities in the cluster. when measuring b 3 , we remove entities with no co-references (singletons) from the evaluation to avoid inflation of results @xcite . the threshold baseline (bcos) gives the highest muc recall but also poor muc precision and poorest b 3 precision. the b 3 metric is highly specific with respect to false-positive entity mentions and strongly penalises bcos for linking all noncoreferent pairs with cossim (m 0 , m 1 ) ≥ 0.65. furthermore, fig. 2 shows that a thresholding strategy is clearly sub-optimal given that there is a significant overlap of co-referent and non-co-referent pairs with only a small minority of pairs at the top and bottom of the distribution that do not overlap. therefore, despite its promising muc f1 score, it is clear that bcos is not useful in practical terms. whilst our thresholding baseline above uses bert, roberta is used by @xcite as the basis for their state-of-the-art model and thus for our models based on their work. although the two models have the same architecture, roberta has been shown to outperform bert at a range of tasks @xcite . however, as shown in figure 4 , the cosine similarity distribution of mention pair embeddings produced by roberta is compressed to use a smaller area of the potential distribution space compared to that of bert (figure 2 ). this compression of similarities may imply a reduction in roberta's ability to discriminate in our task setting. @xcite explain that their byte-pair-encoding (bpe) mechanism, which expands roberta's sub-word vocabulary and simplifies pre-processing, can reduce model performance for some tasks, although this is not further explored in their work. we leave further exploration of roberta's bpe scheme and its effects on the cd 2 cr task setting to future work. all of the models specifically trained on the cd 2 cr corpus (ca-v, ca-ft, ca-s) outperform the ca model by a large margin. furthermore, the ca-v model (without pre-training on ecb+ corpus) outperforms the ca-ft model (with ecb+ pre-training) by 6% muc and 3% b 3 . these results suggest that the cd 2 cr task setting is distinct from the cdcr and ecb+ task setting and that this distinction is not solvable with fine-tuning. in terms of both muc and b 3 , ca-s performs much worse than ca-v suggesting that scibert embeddings are less effective than roberta embeddings in this task setting. we hypothesise that scibert's specialisation towards scientific embeddings may come at the cost of significantly worse news summary embeddings when compared to those produced by roberta. we next evaluate our best performing cd 2 cr baseline model (ca-v) at the entity resolution cdcr task using the ecb+ test corpus, to see how well it generalises to the original cdcr task. results are presented in 5 along-side cattan et al's original model results (ca). the ca-v model still shows good performance, despite a small drop, when compared to the original ca model. the drop in b 3 f1 is more pronounced than muc but is still broadly in line with other contemporary cdcr systems @xcite . the ca-v model demonstrates a promising ability to generalise beyond our corpus to other tasks and reveals an interesting correspondence between cdcr and cd 2 cr settings. finally, the best model (ca-v) is analysed using a series of challenging test cases inspired by ribeiro et al (2020). these test cases were created using 210 manually annotated mention-pairs found in the test subset of the cd 2 cr corpus according to the type of relationship illustrated (anaphora & exophora, subset relationships, paraphrases). we collected a balanced set of 30-40 examples of both co-referent and non-coreferentbut-challenging pairs for each type of relationship (exact numbers in ## conclusion we have defined cross-document, cross-domain co-reference resolution (cd 2 cr), a special and challenging case of cross-document co-reference resolution for comparing mentions across documents of different types and/or themes. we have constructed a specialised cd 2 cr annotated dataset, available, along with our annotation guidelines and tool, as a free and open resource for future research. we have shown that state-of-the-art cdcr models do not perform well on the cd 2 cr dataset without specific training. furthermore, even with task-specific training, models perform modestly and leave room for further research and improvement. finally, we show that the understanding of semantic relatedness offered by current generation transformer-based language models may not be precise enough to reliably resolve complex linguistic relationships such as those found in cd 2 cr as well as other types of co-reference resolution and relationship extraction tasks. the use of semantic enrichment techniques (such as those discussed in section 2.3) to improve model performance in the cd 2 cr task should be investigated as future work.