Incremental Neural Coreference Resolution in Constant Memory (2020)

Abstract: We investigate modeling coreference resolution under a fixed memory constraint by extending an incremental clustering algorithm to utilize contextualized encoders and neural components. Given a new sentence, our end-to-end algorithm proposes and scores each mention span against explicit entity representations created from the earlier document context (if any). These spans are then used to update the entity’s representations before being forgotten; we only retain a fixed set of salient entities throughout the document. In this work, we successfully convert a high-performing model (Joshi et al., 2020), asymptotically reducing its memory usage to constant space with only a 0.3% relative loss in F1 on OntoNotes 5.0.

## introduction coreference resolution is a core task in nlp for both model analysis and information extraction. at the sentence level, ambiguities in pronoun coreference can be used to probe a model for common sense @xcite or gender biases @xcite . at the document level, coreference resolution is commonly used in information extraction pipelines, but can be applied to reading comprehension @xcite or literature analysis @xcite . models for this task typically encode the entire text before scoring and subsequently clustering candidate mention spans, either found by a parser @xcite or learned jointly @xcite . prior work has primarily focused on improving pairwise span scoring functions @xcite @xcite and methods for decoding into globally consistent clusters @xcite @xcite . recent models have also benefited from pretrained encoders used to create high-dimensional input text (and span) representations, and improvements in contextualized encoders appear to translate directly to coreference resolution @xcite @xcite . these models typically rely on simultaneous access to all spans -θ(n) for a document with length n -for scoring and all scores -up to θ(n 2 ) -for decoding. as the dimensionality of contextualized encoders, and therefore the size of span representations, increases, this becomes computationally intractable for long documents or under limited memory. given these constraints, expensive scoring functions are increasingly difficult to explore. further, prior models depart from how humans incrementally read and reason about coreferent mentions; @xcite argue in favor of a limited memory constraint as a more psycholinguistically plausible approach to reading and model coreference resolution via shift-reduce parsing. motivated by scalability and armed with advances in neural architectures, we revisit that intuition. following prior work, our model begins with a spanbert encoding of a text segment to form a list of proposed mention spans @xcite . clustering is performed online: each span either attaches to an existing cluster or begins a new one. we substantially minimize memory usage during inference by storing only the embeddings of active entities in the document and a small set of candidate mention spans. our two contributions of online clustering and storing a constant size set of active entities result in an end-to-end trainable model that uses o(1) space with respect to document length while sacrificing little in performance (see figure 1 ). 1 2 model our algorithm revisits the approach taken by @xcite for incrementally making coreference resolution decisions (online clustering). the major differences lie in explicit entity representations, neural components, and learning. baseline first, we summarize the coreference resolution model described by @xcite , which itself extends from earlier work @xcite . for each document, this model enumerates and scores all spans up to a chosen width. the span representations are formed using bert @xcite encodings of input text by concatenating the first, last, and an attention-weighted average of the token representations within the span. these spans are ranked and pruned to the top θ(n) mentions. both the maximum span width and fraction of remaining spans are hyperparameters. for each remaining span, the model learns a distribution over its possible antecedents (via a pairwise scorer) and the training objective maximizes the probability of its gold labeled antecedents. the entire model (including finetuning the encoder) is trained end-to-end over ontonotes 5.0. this model is further improved by @xcite , who introduces spanbert and uses it as the underlying encoder instead. the spanbertlarge version of @xcite is the baseline model used in this paper. inference our method (algorithm 1) stores a permanent list of entities (clusters), each with its own representation. for a given sentence or segment, the model proposes a candidate set of spans. for each span, a scorer scores the span representation against all the cluster representations. this is used to determine to which (if any) of the pre-existing clusters the current span should be added. upon inclusion of the span in the cluster, the cluster's representation is subsequently updated via a (learned) function. periodically, the model evicts less salient entities, writing them to disk. under this algorithm, each clustering decision is permanent. 2 concretely, our model uses a contextualized encoder, spanbert @xcite , to encode an entire segment. given a segment, spans returns candidate spans, a result of enumerating all spans up to a fixed width, encoding spans as a combination of the embeddings within the span, algorithm 1 findclusters(document) create an empty entity list, e for @xmath0, e) top score ← max(scores) top e ← argmax(scores) if top @xmath1, m) else add new entity(e, m) evict(e) return e pairscore is a feedforward scorer which takes as input the concatenation of a mention span and entity representation along with additional embeddings for distance and genre. update updates the entity representation (e top e ) with the newly linked span representation (e m ). in this work, we use a learned weight, α = σ(ff([e top e , e m ])) and update e top e ← αe top e + (1 -α)e m . 3 here, ff is a feedforward network and σ is the sigmoid function. to ensure constant space, evict moves some entities from e to cpu. these entities are never revisited; the offsets are stored on cpu solely for evaluation purposes. we evict based on cluster size and distance from the end of the segment. the algorithm is independent of these components, so long as they satisfy the correct interface. specifically, our algorithm is compatible with the recent model by @xcite . they use a querybased pairwise scorer, which could be adopted in place of the feedforward pairwise scorer. our use of abstract components also allows for comparison of different encoders or update rules. training similar to prior work @xcite , our training objective is to maximize the probability of the correct antecedent (cluster) for each mention span. however, rather than considering all correct antecedents, we are only interested in the cluster for the most recent one. 4 for each mention m, scores is treated as an unnormalized probability distribution p (e | m) for @xmath2, where e is the entity list that includes an ε target label which represents the action of starting a new cluster. the exact objective is to maximize p (@xmath3; e gold is the gold cluster of m (i.e., the cluster the most recent antecedent was assigned to). however, the entirely sequential algorithm also introduces sample inefficiency, as most mentions have the same label (ε) and barely accrue loss. we speed up training by accumulating gradients periodically, trading computation time for space. this tradeoff is similar to that of batching by documents, which is impractical for our model from a memory perspective. like prior work, we update parameters once per document (and not once per mention). we lean on pretrained components: we reuse not only encoder weights that are already finetuned on this dataset, but also the mention and pairwise scorers from @xcite as initialization for our encoder, spans and pairscore. 5 3 experiments since we reuse weights from @xcite (our baseline), our primary experiment is to compare their model to our constant space adaptation in both task performance and memory usage. additionally, we analyze document and segment length, conversational genre, and explicit clusters. ## conclusion we present an online algorithm for space efficient coreference resolution that incorporates contributions from recent neural end-to-end models. we show it is possible to transform a model which performs document-level inference into an incremental algorithm. in so doing, we greatly reduce the memory usage of the model during inference at virtually no cost to performance, thereby providing an option for researchers and practitioners interested in modern coreference resolution models for tasks constrained by memory, like the modeling of book-length texts.