Seq2seq is All You Need for Coreference Resolution (2023)

Abstract: Existing works on coreference resolution suggest that task-specific models are necessary to achieve state-of-the-art performance. In this work, we present compelling evidence that such models are not necessary. We finetune a pretrained seq2seq transformer to map an input document to a tagged sequence encoding the coreference annotation. Despite the extreme simplicity, our model outperforms or closely matches the best coreference systems in the literature on an array of datasets. We consider an even simpler version of seq2seq that generates only the tagged spans and find it highly performant. Our analysis shows that the model size, the amount of supervision, and the choice of sequence representations are key factors in performance.

## introduction the seminal work by @xcite popularized end-to-end models for coreference resolution based on searching over all possible spans and their clustering. however, even with substantial refinement and simplification in followup works @xcite @xcite , the models are highly task-specific, involving many specialized hyperparameters such as how many candidates to retain in the mention proposal phase. there is a recent line of works that take an alternative approach, leveraging advances in pretrained sequence-to-sequence (seq2seq) models. @xcite propose asp, an autoregressive pointerbased model with a multitasking head for bracket pairing and span labeling. @xcite propose a transition-based system with carefully designed states and actions, simulated by a seq2seq model that processes one state at a time. however, these seq2seq-based models are still task-specific, requiring a modification of the seq2seq architecture or a derivation of a transition-based system with state manipulation. a natural question is: are such task-specific models necessary for coreference resolution, or can we approach it as a standard seq2seq problem? there have been previous efforts to reduce coreference resolution as a seq2seq problem @xcite , but they underperform task-specific models, suggesting that task-specific models are perhaps necessary. in this work, we present the first full seq2seq reduction of coreference resolution that matches or outperforms the best coreference systems in the literature, demonstrating that task-specific models are not necessary to obtain state-of-the-art performance and questioning the need to develop task-specific solutions. our approach is extremely simple. we treat the raw document as a source sequence and the coreference annotation as a target sequence, then finetune a pretrained encoder-decoder model like t5 @xcite or t0 @xcite without any modification to the architecture. there is a great deal of flexibility in the choice of target sequence representation. our main model represents the coreference annotation as a sequence of actions that either copy a token from the source sequence, start/end a mention span, or tag a predicted mention span with an integer. at test time, the model always produces a valid coreference clustering by constrained beam search. we consider an even simpler version in which the model generates only the tagged spans, and find that it also yields surprisingly high performance. this simpler version is advantageous because it results in faster inference. our seq2seq model obtains strong results on an array of coreference datasets. on english ontonotes @xcite , with a 3bparameter t0 the model obtains 82.9 test average f1, outperforming the corresponding asp @xcite initialized from the same base model (82.3). with a 11b-parameter t0, our model achieves 83.2 f1, outperforming corefqa @xcite (83.1) and getting close to the best known result using a 13b-parameter model (83.3). on preco @xcite , the model achieves 88.5, outperforming the task-specific model of @xcite (87.8). on lit-bank @xcite , which has substantially smaller training data, the model obtains 78.3 cross-validation f1 lagging behind @xcite who report 79.2. but when trained on the union of litbank, ontonotes, and preco, it obtains 81.2 split-0 f1, significantly outperforming their 78.2. our analysis shows that the model size, the amount of supervision, and the choice of sequence representations are key factors in performance. we make our code publicly available at: https: //github.com/wenzhengzhang/seq2seqcoref . ## related work in this section, we give a more focused treatment of previous works on seq2seq-style approaches to coreference resolution to make our contributions more clear. @xcite propose framing coreference resolution as a seq2seq task, but their work is a proof of concept. they naively predict boundaries, cluster numbers, and blanks (e.g., "(0 --0) -(1 -(2) | 1)") with suboptimal modeling choices (e.g., their decoder only receives these symbols and no document content). they only report results on the arrau corpus and significantly fall behind existing works (e.g., 66.5 vs 78.8 under b 3 ). in contrast, we solve a much more challenging problem of developing state-of-the-art seq2seq coreference systems. there are recent works approaching structured prediction with a general seq2seq-style solution. one example is tanl @xcite , which frames entity/relation extraction, semantic role labeling, and coreference resolution as seq2seq tasks. again, tanl fails to demonstrate competitive performance on standard coreference resolution datasets, obtaining only 72.8 average f1 on ontonotes (compared to the current state-of-theart performance level which @xmath0. their target sequence representation corresponds to our full linearization with token action except that they tag the cluster information by preceding mention strings (e.g., "[ his | barack obama ]"), which yields long target sequences and introduces clustering ambiguity. while we improve the performance of tanl to 79.6 in our own implementation, achieving the best performance (83.2) requires our sequence def-initions (section 3). asp @xcite is an autoregressive pointer-based model for structured prediction. it is a modified transformer that at step t conditions on the input document x and a sequence representation of the annotation so far @xmath1, β t , γ t ) used for structure building. for instance, β @xmath2. . . t -1} is a bracket pairing action parameterized with a feedforward layer that consumes the previous hidden states (h t , h βt ) (i.e., a pointer network). asp obtains state-of-the-art results on ontonotes (up to 82.5 average f1). we outperform asp with a standard transformer. @xcite develop a transition-based coreference system that can be implemented by a seq2seq model. the system autoregressively maps a state to a prediction, where a state is previous coreference-annotated sentences along with the next sentence and a prediction is system actions (e.g., link and append). while the system can be reduced to seq2seq predictions, it processes one sentence at a time by applying the predicted actions to the current state. we show that such an explicit state-by-state transition is not necessary, and that a standard transformer can directly reason with all coreference clusters simultaneously. ## seq2seq methods let v denote the vocabulary. for any sequence @xmath3. . . t }, we use the prefix notation @xmath4. . . a t-1 ) ∈ v t-1 and @xmath5. . . a t ) ∈ v t . we assume a generalized seq2seq setting in which @xmath6, @xmath7.e., a sequence of labels), and @xmath8, @xmath9, @xmath10. the "generalized" model uses parameters θ to define the conditional distribution we emphasize that the generalization above does not modify the standard seq2seq framework. during training, we feed (x, z) to the model as the usual source-target sequence pair and minimize the cross-entropy loss using y as per-token labels. at test time, we apply the mapping f at each step by postprocessing predictions. all our methods use (1) and only change the variable definitions. the encoder input x is always the document to be processed. the decoder input z is a sequence representation or "linearization" of the coreference annotation of x. the decoder output y is any action sequence from which z can be extracted deterministically at each step. ## discussion having presented our technical approach, we discuss how it relates to other approaches to coreference resolution that also use sequence-based models. our approach is pure seq2seq because we use both the standard architecture and the system design that is applicable to any seq2seq task, such as machine translation and text summarization. in contrast, the approach of @xcite is not considered pure seq2seq because their transitionbased system is specifically designed for the coreference task (e.g., their "link" and "append" actions are meant to cluster mentions together), even though the system itself is implemented using the standard seq2seq architecture. to understand the practical difference, note that their system requires m encoder forward passes for a single document during inference where m is the number of sentences, whereas ours requires one. the success of the transition system of @xcite does not imply the success of the general seq2seq system in our work. we adopt the generalized seq2seq framework that may require postprocessing during generation (e.g., for copy action) for improved modeling flexibility and performance, but the postprocessing step does not change the standard seq2seq architecture and system design. furthermore, our model without postprocessing is nearly as effective. specifically, our "full linear + token action + t0 3b " model in we use constrained decoding during generation to ensure valid coreference annotation, which is a standard practice @xcite . the details can be found in appendix a. ## conclusions we have presented a highly performant seq2seq reduction of coreference resolution. unlike in previous works that rely on task-specific approaches to obtain strong performance, we use a standard encoder-decoder model that receives the document as an input sequence and predicts a sequence representation of its coreference annotation as the target sequence. contrary to previously reported weak results using seq2seq reductions, we show for the first time that with suitable definitions of the sequence representation of the task, it is possible to achieve state-of-the-art performance, reaching 83.2 test average f1 on english ontonotes with an 11b-parameter t0 pp initialization. our model's strong results fundamentally challenge the default task-specific approach to coreference resolution.