Coreference in Long Documents using Hierarchical Entity Merging (2024)

Abstract: Current top-performing coreference resolution approaches are limited with regard to the maximum length of texts they can accept. We explore a recursive merging technique of entities that allows us to apply coreference models to texts of arbitrary length, as found in many narrative genres. In experiments on established datasets, we quantify the drop in resolution quality caused by this approach. Finally, we use an under-explored resource in the form of a fully coreference-annotated novel to illustrate our model’s performance for long documents in practice. Here, we achieve state-of-the-art performance, outperforming previous systems capable of handling long documents.

## introduction coreference resolution has significant time and memory requirements which, in currently released models, typically increase at least quadratically with the length of the document, resulting in inefficient systems. these substantial computational requirements make coreference resolution impractical for long documents such as novels. the task of establishing coreference links in such texts is important to enable a wide range of downstream tasks such as extracting character interaction networks (e.g. @xcite . we propose a novel hierarchical algorithm for coreference resolution, to conserve computational resources while still achieving good performance. our approach allows the models to -in principle -scale to documents of arbitrary length, outperforming existing long-document approaches. our proposed approach works by splitting a long document into multiple splits and then running an existing coreference resolution model on each split, thereby extracting the entities in each of them. we ## background & related work coreference resolution is the task of identifying corefering spans in a text, that is to say, those groups of spans that refer to the same entity. traditional coreference resolution comprises of two phases: span extraction from the text and the subsequent identification of coreference links among the extracted spans. the word-level coreference (subsequently wl-coref) resolution model (dobrovolskii, 2021) separates the task of coreference resolution from span extraction and solves it on the word level, hence lowering the time complexity of the model to o(n 2 ), where n is the length of the document. the span extraction is performed separately only for those words that are found to be coreferent to some other words. we will base our experiments on this model, but note that our hierarchical approach can in principle also be used with the other major coreference resolution model architectures, with the only requirement being that mentions can be represented by fixed-length embeddings. the models by @xcite ; @xcite are recent sequence-to-sequence coreference models, but are impractical for long documents due to their memory requirements, with even short document processing being very resource-intensive. our proposed hierarchical merging strategy could potentially help to apply these models to long documents. we train our english model on the ontonotes dataset @xcite and the litbank @xcite dataset, the latter containing coreference annotations for literary texts. we evaluate our approach on litbank, and observe competitive performance to the state-of-the-art coreference models while being memory efficient. for training our german models, we employ the tüba-dz news dataset @xcite and the droc literature dataset @xcite . efficient coreference resolution in long documents is a task that current models struggle with, due to their memory-intensive nature. traditional incremental coreference resolution models use global entity representations, but their performance lags behind compared to other models @xcite . @xcite propose a scalable coreference resolution approach that works on the token level instead of the span level, and drops non-essential candidate antecedents to improve memory and time requirements. they test their system on a long book for which no annotations are available. their code is not available as open source, which prevented us from testing it on the full book data that we use. ## methodology in this section, we describe our coreference approach in detail. we build upon the wl-coref model @xcite , picking it over alternatives due to its quick inference and relative simplicity in conjunction with competitive performance. our code is publicly available. * we split each document evenly into n splits, ensuring an equal number of sentences in each, where @xmath0, 4, 8}. treating each split as an independent document, the model performs inference as normal on each of them, ideally yielding one cluster for each entity in each split. that is to say, the model predicts pairwise scores for whether two words co-refer and builds a transitive closure over those connections that exceed a certain threshold. now, since some entities in one split may refer to the same entity as entities in another split, we develop a merging approach to merge such coreferring entities across splits. this process is similar to linking two mentions, except that instead of passing mention embeddings, we pass entity embeddings to the model. hence, we need a way to represent individual entities as vectors. in this work, we only evaluate the approach of creating entity embeddings by means of averaging over all embeddings in a cluster. next, we merge the entities from two splits at a time, by passing the entity embeddings in the same way that the model takes in the word embeddings to create links among them. this results in an entity list that spans both the splits. the merging process is repeated for all such disjoint pairs of splits in each level, resulting in n/2 splits in the subsequent level if the previous level had n splits. consequently, the last level of our hierarchical approach results in entities that encompass the entire document. this is illustrated in figure 1 . ## experiments we conduct a set of experiments to evaluate the efficacy of the proposed hierarchical merging approach. first, we train the standard wl-coref model, which is pre-trained on the ontonotes dataset, for an additional 10 epochs on the litbank dataset. the resulting model is evaluated on the test split of litbank, from which we exclude the singleton mentions since the original wl-coref architecture removes singletons during the span extraction step. that is to say we evaluate on a version of the texts without singleton mentions. in terms of merging approaches, we follow the process laid out in section 3. we document the results under three scenarios: (a) without merging entities across splits, (b) merging entities without training the model for merging (i.e. using the same model twice), and (c) merging entities after training the model for merging. we expect the merging approach to have a negative impact on prediction quality, at least for short documents. to quantify said impact of merging on the prediction quality, we split the documents into n equal-sized splits and experiment with different values of n, specifically 2, 4, and 8. we set a baseline result by refraining from merging the entities across splits. for the merging module, we experimented with and without training the model specifically for merging. for training data, we use 2-way splits and use gold entities in the individual splits, rather than predicted ones. we subsequently average the span embeddings in each entity to obtain the entity embeddings in each split. as these embeddings are handled analogously to word embeddings in the original setup, the model creates links between the entities in the same way that it does so for the words. we evaluate the entities now spanning the entire documents on the gold data. our merging model is trained for 10 epochs using 2-way split documents, on top of the existing word-level weights. as we found the model to lose its span prediction capabilities after training the merging module, we use two distinct sets of weights for the two tasks. for the first level of our approach, where the model generates an entity list, we employ the first sets of weights, which was trained on litbank but not trained specifically for merging entities (this is denoted as "coref" in figure 1 ). for subsequent levels of our approach, where entity lists from two splits are merged to produce an entity list that spans both the splits, we use the second set of weights, which was specifically trained for merging entities (indicated by "merge" in figure 1 ). we refer to the recursive application of the merging step as hierarchical entity merging. our approach is primarily evaluated on the standard conll-f1 score. additionally, we provide lea, an evaluation metric that focuses on coreference links and resolves several issues that conll-f1 constituent scores suffer from (moosavi and strube, 2016). ## results all the scores have been obtained using the official @xcite . we perform experiments with varying numbers of splits, specifically splitting the document into 1 (i.e. not splitting it), 2, 4, and 8 segments, as can be seen in figure 2 . for the unsplit document, we achieve a conll-f1 score of 78.45. for two splits, we achieve a score of 76.06 if we employ merging without specifically training the merging module (using only first sets of weights). we achieve an improved f1 score of 77.30 if we additionally train the merging module (using two sets of weights). as a control experiment where we do not merge entity lists at all, we unsurprisingly obtain a much worse f1 score of 70.27. similarly, for four and eight splits, the scores increase by two points on average as we go from one set of weights approach to the two sets of weights approach. the best f1 score we achieve is 74.52 and 73.96 respectively, by using two sets of weights. the scores when not conducting a merging step are unsurprisingly far below those with a in practice, our model is very run-time efficient and inference for the whole book "effi briest" takes just under two minutes on a single a6000 gpu. ## conclusion our results demonstrate a potential path for applying state-of-the-art models to longer text without vast increases in memory requirements. we use existing systems to provide a baseline result for coreference resolution on a fully-annotated novel and set new state-of-the-art results. there is further headroom, as we are -for reasons of simplicitynot using the best available model for within-split resolution. our training process could also be improved, for example by randomly splitting documents (rather than into roughly equal portions) to create more training data. while our approach exhibits reasonable coreference resolution quality for novels, it still does not suffice for use as a preprocessing step in most literary computing applications. we suspect that a substantial improvement of our approach would be possible by creating specialized embeddings for merging rather than averaging mention embeddings; additional error analysis will be needed to verify this intuition. our general idea of hierarchical merging could also potentially be adapted to the very recent seq2seq coreference architectures @xcite where they could help overcome the model's maximum input lengths.