Coreference Resolution without Span Representations (2021)

Abstract: The introduction of pretrained language models has reduced many complex task-specific NLP models to simple lightweight layers. An exception to this trend is coreference resolution, where a sophisticated task-specific model is appended to a pretrained transformer encoder. While highly effective, the model has a very large memory footprint – primarily due to dynamically-constructed span and span-pair representations – which hinders the processing of complete documents and the ability to train on multiple instances in a single batch. We introduce a lightweight end-to-end coreference model that removes the dependency on span representations, handcrafted features, and heuristics. Our model performs competitively with the current standard model, while being simpler and more efficient.

## introduction until recently, the standard methodology in nlp was to design task-specific models, such as bidaf for question answering @xcite and esim for natural language inference @xcite . with the introduction of pretraining, many of these models were replaced with simple output layers, effectively fine-tuning the transformer layers below to perform the traditional model's function @xcite . a notable exception to this trend is coreference resolution, where a multi-layer taskspecific model @xcite is appended to a pretrained model @xcite . this model uses intricate span and span-pair representations, a representation refinement mechanism, handcrafted features, pruning heuristics, and more. while the model is highly effective, it comes at a great cost in memory consumption, limiting the amount of examples that can be loaded on a large gpu to a single document, which often needs to * equal contribution. we present start-to-end (s2e) coreference resolution: a simple coreference model that does not construct span representations. instead, our model propagates information to the span boundaries (i.e., its start and end tokens) and computes mention and antecedent scores through a series of bilinear functions over their contextualized representations. our model has a significantly lighter memory footprint, allowing us to process multiple documents in a single batch, with no truncation or sliding windows. we do not use any handcrafted features, priors, or pruning heuristics. experiments show that our minimalist approach performs on par with the standard model, despite removing a significant amount of complexity, parameters, and heuristics. without any hyperparameter tuning, our model achieves 80.3 f1 on the english ontonotes dataset @xcite , with the best comparable baseline reaching 80.2 f1 @xcite , while consuming less than a third of the memory. these results suggest that transformers can learn even difficult structured prediction tasks such as coreference resolution without investing in complex task-specific architectures. foot_0 ## background: coreference resolution coreference resolution is the task of clustering multiple mentions of the same entity within a given text. it is typically modeled by identifying entity mentions (contiguous spans of text), and predicting an antecedent mention a for each span q (query) that refers to a previously-mentioned entity, or a null-span ε otherwise. @xcite introduce coarse-to-fine (c2f), an end-to-end model for coreference resolu-tion that predicts, for each span q, an antecedent probability distribution over the candidate spans c: here, f (c, q) is a function that scores how likely c is to be an antecedent of q. this function is comprised of mention scores f m (c), f m (q) (i.e. is the given span a mention?) and a separate antecedent score f a (c, q): our model (section 3) follows the scoring function above, but differs in how the different elements f m (•) and f a (•) are computed. we now describe how f m and f a are implemented in the c2f model. ## model we present start-to-end (s2e) coreference resolution, a simpler and more efficient model with respect to c2f (section 2). our model utilizes the endpoints of a span (rather than all span tokens) to compute the mention and antecedent scores f m (•) and f a (•, •) without constructing span or span-pair representations; instead, we rely on a combination of lightweight bilinear functions between pairs of endpoint token representations. furthermore, our model does not use any handcrafted features, does not prune antecedents, and prunes mention candidates solely based on their mention score f m (q). our computation begins by extracting a start and end token representation from the contextualized representation x of each token in the sequence: we then compute each mention score as a biaffine product over the start and end tokens' representations, similar to @xcite : the first two factors measure how likely the span's start/end token q s /q e is a beginning/ending of an entity mention. the third measures whether those tokens are the boundary points of the same entity mention. the vectors v s , v e and the matrix b m are the trainable parameters of our mention scoring function f m . we efficiently compute mention scores for all possible spans while masking spans that exceed a certain length ℓ. 2 we then retain only the top-scoring λn mention candidates to avoid o(n 4 ) complexity when computing antecedents. similarly, we extract start and end token representations for the antecedent scoring function f a : then, we sum over four bilinear functions: however, computing the factors directly bypasses the need to create n 2 explicit span representations. thus, we avoid a theoretical space complexity of o(n 2 d), while keeping it equivalent to that of a transformer layer, namely o(n 2 + nd). ## experiments dataset we train and evaluate on two datasets: the document-level english ontonotes 5.0 dataset @xcite , and the gap coreference dataset @xcite . the ontonotes dataset contains speaker metadata, which the baselines use through a hand-crafted feature that indicates whether two spans were uttered by the same speaker. instead, we insert the speaker's name to the text every time the speaker changes, making the metadata available to any model. pretrained model we use longformer-large @xcite as our underlying pretrained model, since it is able to process long documents without resorting to sliding windows or truncation. baseline we consider @xcite expansion to the c2f model as our baseline. specifically, we use the implementation of @xcite with minor adaptations for supporting longformer. we do not use higher-order inference, as @xcite demonstrate that it does not result in significant improvements. we train the baseline hyperparameters all models use the same hyperparameters as the baseline. the only hyperparameters we change are the maximum sequence length and batch size, which we enlarge to fit as many tokens as possible into a 32gb gpu. 3 for our model, we use dynamic batching with 5,000 max tokens, which allows us to fit an average of 5-6 documents in every training batch. the baseline, however, has a much higher memory footprint, and is barely able to fit a single example with longformer-base (max 4,096 tokens). when combining the baseline with spanbert-large or longformer-large, the baseline must resort to sliding windows to process the full document (512 and 2,048 tokens, respectively). ## related work recent work on memory-efficient coreference resolution sacrifices speed and parallelism for guarantees on memory consumption. @xcite and @xcite present variants of the c2f model @xcite that use an iterative process to maintain a fixed number of span representations at all times. specifically, spans are processed sequentially, either joining existing clusters or forming new ones, and an eviction mechanism ensures the use of a constant number of clusters. while these approach constrains the space complexity, their sequential nature slows down the computation, and slightly deteriorates the performance. our approach is able to alleviate the large memory footprint of c2f while maintaining fast parallel processing and high performance. corefqa @xcite propose an alternative solution by casting the task of coreference resolution as one of extractive question answering. it first detects potential mentions, and then creates dedicated queries for each one, creating a pseudo-question-answering instance for each candidate mention. this method significantly improves performance, but at the cost of processing hundreds of individual context-question-answer instances for a single document, substantially increasing execution time. our work provides a simple alternative, which can scale well in terms of both speed and memory. ## conclusion we introduce a new model for coreference resolution, suggesting a lightweight alternative to the sophisticated model that has dominated the task over the past few years. our model is competitive with the baseline, while being simpler and more efficient. this finding once again demonstrates the spectacular ability of deep pretrained transformers to model complex natural language phenomena.