Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends (2024)

Abstract: Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks. However, the urge to attain superior results has, at times, led to the premature replacement of carefully designed task-specific approaches without exhaustive experimentation. The Coreference Resolution task is no exception; all recent state-of-the-art solutions adopt large generative autoregressive models that outperform encoder-based discriminative systems. In this work, we challenge this recent trend by introducing Maverick, a carefully designed – yet simple – pipeline, which enables running a state-of-the-art Coreference Resolution system within the constraints of an academic budget, outperforming models with up to 13 billion parameters with as few as 500 million parameters. Maverick achieves state-of-the-art performance on the CoNLL-2012 benchmark, training with up to 0.006x the memory resources and obtaining a 170x faster inference compared to previous state-of-the-art systems. We extensively validate the robustness of the Maverick framework with an array of diverse experiments, reporting improvements over prior systems in data-scarce, long-document, and out-of-domain settings. We release our code and models for research purposes at https://github.com/SapienzaNLP/maverick-coref.

## introduction as one of the core tasks in natural language processing, coreference resolution aims to identify and group expressions (called mentions) that refer to the same entity @xcite . given its crucial role in various downstream tasks, such as knowledge graph construction @xcite , entity linking @xcite , question answering @xcite @xcite , machine translation @xcite @xcite ) and text summarization @xcite @xcite , inter alia, there is a pressing need for both high performance and efficiency. however, recent works in coreference resolution either explore methods to obtain reasonable performance optimizing time and memory efficiency @xcite @xcite , or strive to improve benchmark scores regardless of the increased computational demand @xcite . efficient solutions usually rely on discriminative formulations, frequently employing the mentionantecedent classification method proposed by @xcite . these approaches leverage relatively small encoder-only transformer architectures @xcite to encode documents and build on top of them task-specific networks that ensure high speed and efficiency. on the other hand, performance-centered solutions are nowadays dominated by general-purpose large sequence-to-sequence models @xcite . a notable example of this formulation, and currently the state of the art in coreference resolution, is @xcite , which proposes a transition-based system that incrementally builds clusters of mentions by generating coreference links sentence by sentence in an autoregressive fashion. although sequence-to-sequence solutions achieve remarkable performance, their autoregressive nature and the size of the underlying language models (up to 13b parameters) make them dramatically slower and memory-demanding compared to traditional encoder-only approaches. this not only makes their usage for downstream applications impractical, but also poses a significant barrier to their accessibility for a large number of users operating within an academic budget. in this work we argue that discriminative encoder-only approaches for coreference @xcite have not yet expressed their full potential and have been discarded too early in the urge to achieve state-of-the-art performance. in proposing maverick, we strike an optimal balance between high performance and efficiency, a combination that was missing in previous systems. our framework enables an encoder-only model to achieve toptier performance while keeping the overall model size less than one-twentieth of the current stateof-the-art system, and training it with academic resources. moreover, when further reducing the size of the underlying transformer encoder, maverick performs in the same ballpark as encoder-only efficiency-driven solutions while improving speed and memory consumption. finally, we propose a novel incremental coreference resolution method that, integrated into the maverick framework, results in a robust architecture for out-of-domain, data-scarce, and long-document settings. ## related work we now introduce well-established approaches to neural coreference resolution. specifically, we first delve into the details of traditional discriminative solutions, including their incremental variations, and then present the recent paradigm shift for approaches based on large generative architectures. ## methodology in this section, we present the maverick framework: we propose replacing the preprocessing and training strategy of coarse-to-fine models with a novel pipeline that improves the training and inference efficiency of coreference resolution systems. furthermore, with the maverick pipeline, we elim-inate the dependency on long-standing manuallyset hyperparameters that regulate memory usage. finally, building on top of our pipeline, we propose three models that adopt a mention-antecedent classification technique, namely maverick s2e and maverick mes , and a system that is based upon a novel incremental formulation, maverick incr . ## conclusion in this work, we challenged the recent trends of adopting large autoregressive generative models to solve the coreference resolution task. to do so, we proposed maverick, a new framework that enables fast and memory-efficient coreference resolution while obtaining state-of-the-art results. this demonstrates that the large computational overhead required by sequence-to-sequence approaches is unnecessary. indeed, in our experiments maverick systems demonstrated that they can outperform large generative models and improve the speed and memory usage of previous best-performing encoder-only approaches. furthermore, we introduced maverick incr , a robust multi-step incremental technique that obtains higher performance in the out-of-domain and long-document setting. by releasing our systems, we make state-of-the-art models usable by a larger portion of users in different scenarios and potentially improve downstream applications.