Contrastive Representation Learning for Cross-Document Coreference Resolution of Events and Entities (2022)

Abstract: Identifying related entities and events within and across documents is fundamental to natural language understanding. We present an approach to entity and event coreference resolution utilizing contrastive representation learning. Earlier state-of-the-art methods have formulated this problem as a binary classification problem and leveraged large transformers in a cross-encoder architecture to achieve their results. For large collections of documents and corresponding set of n mentions, the necessity of performing n^{2} transformer computations in these earlier approaches can be computationally intensive. We show that it is possible to reduce this burden by applying contrastive learning techniques that only require n transformer computations at inference time. Our method achieves state-of-the-art results on a number of key metrics on the ECB+ corpus and is competitive on others.

## introduction coreference resolution is the fundamental nlp task of finding all mentions that refer to the same real world entity or event in text. it is an important step for higher level nlp tasks involving natural language understanding, such as text summarization @xcite , information extraction @xcite , and question-answering @xcite . historically, coreference resolution of entities in the same text document -within document (wd) coreference resolutionhas received the most attention, though more recently focus has moved toward cross-document (cd) coreference resolution. cd coreference resolution has recently gained renewed interest for its application in multi-document analysis tasks. cd coreference resolution presents unique challenges not found in the wd context. spans of text come from different documents without any inherent linear order, and there is no notion that antecedents for a given expression typically occur before the expression, as in a single document. coreferent expressions also cannot be assumed to occur near one another. furthermore, documents are also assumed to be authored independently and about different-though lexically similar-topics. for instance, the event described in the sentences from topic 19 in another important aspect of cd coreference resolution is the potential scale of the problem. in certain applications, the number of documents can be large and ever growing. in particular, for applications that merge information from across documents, such as multi-document summarization @xcite or multi-hop question answering @xcite , the corpus in question can be both large and dynamically increasing in size. past methods of cd coreference resolution have treated the problem as a binary classification task: given two pairs of mentions, classify them as referring to the same entity or not @xcite @xcite . in more recent works, contextual embeddings using a cross-encoder architecture have been leveraged to obtain state-of-theart results @xcite @xcite on the ecb+ corpus. despite achieving state-of-the-art results on the benchmark dataset, a shortcoming of these approaches is the fact they use a transformer as a cross-encoder -two sentences are passed to through the transformer network and a label is predicted. for n mentions in a corpus, these approaches require n 2 comparisions at inference time. as @xcite noted when using bert in a cross-encoder architecture, finding the most similar pair of sentences in a collection of @xmath0, which they estimated to take 65 hours using a v100 ## related work most recent work on cd coreference resolution has focused on the ecb+ corpus @xcite , which we also use in this work. the ecb+ corpus, which is an extension of the event coreference bank (ecb), consists of documents from google news clustered into topics and annotated for event coreference @xcite . ecb+ increases the difficulty level of the original ecb dataset by adding a second set of documents for each topic (subtopic), discussing a different event of the same type (e.g. riots in greece over teenagers death vs riots in brooklyn over teenagers death; see most approaches to cd coreference resolution address the problem as a binary classification problem between all pairs of events and entities. early works utilized hand engineered lexical features (e.g. head lemma, word embedding similarities, etc.) @xcite . more recent works have relied on neural network methods, utilizing character-based embeddings (huang et al., 2019; kenyon-dean et al., 2018) or contextual embeddings (yu et al., 2020; cattan et al., 2020; zeng et al., 2020; caciularu et al., 2021; allaway et al., 2021). recent approaches by yu et al. (2020) and caciularu et al. (2021) leveraging roberta and longformer transformer models have set strong benchmarks. a drawback of these approaches is the necessity to consider all pairs of n mentions in a corpus in a cross encoder architecture. each unique pair of entities (separated by a special token) is passed through a transformer to generate a similarity score. this requires n 2 transformer computations. this can be computationally expensive and several works have sought to address this. @xcite introduced a model that clusters mentions sequentially at inference time. they achieved competitive results using a bert-base model and without using a hierarchical clustering algorithm to generate coreference chains. @xcite adapted the model of @xcite to the crossdocument context. specifically, they pruned document spans down to the gold mentions and encode each resulting pared document using a robertalarge model. a pairwise (feed-forward network) scorer then generates a score for each pair of spans. they also considered an end-to-end system where they use their model to predict mention spans instead of using gold mentions. in this work, we consider gold mentions only as has been done in earlier works. in this work, we introduce a method leveraging contrastive learning using a roberta-large model as the base encoder. at inference time, our method requires n passes of the transformer, like earlier methods by @xcite ; @xcite . our method surpasses their methods on the benchmark ecb+ dataset and is competitive with more expensive cross-encoder approaches of yu et al. (2020); zeng et al. (2020); caciularu et al. (2021). ## results and discussion we evaluate our model using four different measures as is common in earlier works. specifically, we evaluated our model performance using muc @xcite , b 3 @xcite , ceaf-e @xcite , and lea (moosavi and strube, 2016) metrics. we also evaluate our model using the conll f1, the average of the muc, b 3 , and ceaf-e f1 scores. as a baseline, we also show results from a lemma model that takes each span in question and utilizes spacy 3 to lemmatize each token. mentions are clustered based on whether their lemmatized tokens are exact matches or not. evaluations on ecb+ test corpus are not without controversy, and we discuss these subtleties in detail below. for the reader familiar with these issues, our main results are discussed in §4.2 and §4.3. we also conduct an ablation study with results in §4.5. ## conclusions in this paper, we proposed a new model for withinand cross-document coreference resolution. we demonstrated that contrastive learning approaches are effective at learning representations for coreference resolution. we evaluated our model on gold topics and at the corpus level of the ecb+ corpuswith and without singleton mentions-and found that our approach surpasses current state-of-the-art methods by large margins. we also evaluated our models with an initial document clustering method and found that our model was competitive with earlier works. we presented extensive ablations of our model and discussed limitations of our work including model size, training time, application to formal text domains (i.e. news articles and wikipedia), and use of agglomerative clustering to generate final coreference clusters. interesting directions for future work would be testing the textrank algorithm in less formal contexts (i.e. beyond news articles and wikipedia articles), investigating higher-order tuples (e.g. triplets) to speed up model convergence, and extending our work to predicted mentions as opposed to gold mentions as has been done by others @xcite . marc vilain, john burger, john aberdeen, dennis connolly, and lynette hirschman. 1995. a modeltheoretic coreference scoring scheme. in proceedings of the 6th conference on message understanding, muc6 '95, page 45-52, usa. association for computational linguistics. bishan yang, claire cardie, and peter frazier. 2015. a hierarchical distance-dependent bayesian model for event coreference resolution. transactions of the association for computational linguistics, 3(0):517-528. xiaodong yu, wenpeng yin, and dan roth. 2020. paired representation learning for event and entity coreference. dmitry zelenko, chinatsu aone, and jason tibbetts. 2004. coreference resolution for information extraction. in proceedings of the conference on reference resolution and its applications, pages 24-31, barcelona, spain. association for computational linguistics. yutao zeng, xiaolong jin, saiping guan, jiafeng guo, and xueqi cheng. 2020. event coreference resolution with their paraphrases and argument-aware embeddings. in proceedings of the 28th international conference on computational linguistics, pages 3084-3094, barcelona, spain (online). international committee on computational linguistics.