Coreference Resolution through a seq2seq Transition-Based System (2023)

Abstract: Most recent coreference resolution systems use search algorithms over possible spans to identify mentions and resolve coreference. We instead present a coreference resolution system that uses a text-to-text (seq2seq) paradigm to predict mentions and links jointly. We implement the coreference system as a transition system and use multilingual T5 as an underlying language model. We obtain state-of-the-art accuracy on the CoNLL-2012 datasets with 83.3 F1-score for English (a 2.3 higher F1-score than previous work [Dobrovolskii, 2021]) using only CoNLL data for training, 68.5 F1-score for Arabic (+4.1 higher than previous work), and 74.3 F1-score for Chinese (+5.3). In addition we use the SemEval-2010 data sets for experiments in the zero-shot setting, a few-shot setting, and supervised setting using all available training data. We obtain substantially higher zero-shot F1-scores for 3 out of 4 languages than previous approaches and significantly exceed previous supervised state-of-the-art results for all five tested languages. We provide the code and models as open source.1

## introduction there has been a great deal of recent research in pretrained language models that employ encoderdecoder or decoder-only architectures (e.g., see @xcite @xcite ), and that can generate text using autoregressive or text-to-text (seq2seq) models (e.g., see t5, mt5 @xcite ). these models have led to remarkable results on a number of problems. coreference resolution is the task of finding referring expressions in text that point to the same entity in the real world. coreference resolution is a core task in nlp, relevant to a wide range of 1 https://github.com/google-research /google-research/tree/master/coref_mt5 . the state-of-the-art models on coreference problems are based on encoder-only models, such as bert @xcite or spanbert @xcite . all recent state-of-the-art coreference models (see this paper describes a text-to-text (seq2seq) approach to coreference resolution that can directly leverage modern encoder-decoder or decoder-only models. the method takes as input a sentence at a time, together with prior context, encoded as a string, and makes predictions corresponding to coreference links. the method has the following advantages over previous approaches: figure 1 : example of one of our transition-based coreference systems, the link-append system. the system processes a single sentence at a time, using an input encoding of the prior sentences annotated with coreference clusters, followed by the new sentence. as output, the system makes predictions that link mentions in the new sentence to either previously created coreference clusters (e.g., ''@xmath0, to previous mentions (e.g., ''the @xmath1. the system predicts ''shift'' when processing of the sentence is complete. note in the figure we use the word indices 2 and 17 to distinguish the two incidences of ''i'' in the text. as input, and outputs an action corresponding to a set of coreference links involving that sentence as its output. figure 1 gives an overview of the highest performing system, ''link-append'' which encodes prior coreference decisions in the input to the seq2seq model, and predicts new conference links (either to existing clusters, or creating a new cluster) as its output. we provide the code and models as open source. we describe results on the conll-2012 data set in section 4. in addition, section 5 describes multilingual results, in two settings: first, the setting where we fine-tune on each language of interest; second, zero-shot results, where an mt5 model fine-tuned on english alone is applied to languages other than english. zero-shot experiments show that for most languages, accuracies are higher than recent translation-based approaches and early supervised systems. ## related work most similar to our approach is the work of @xcite , who use a shift-reduce transition-based system for coreference resolution. the transition system uses two data structures, a queue initialized with all mentions and a list. the shift transition moves from the queue a mention to top of the list. the reduce transition merges the top mentions with selected clusters. @xcite consider the approach to better reflect human cognitive processing, to be simple and to have small memory requirements. @xcite use this transition-based system together with a neural approach for mention identification and transition prediction; this neural model @xcite gives higher accuracy scores (see another important method to gain higher accuracy is to use stronger pretrained language models, which we follow in this paper as well. a number of recent coreference resolution systems kept the essential architecture fixed while they replace the pretrained models with increasingly stronger models. @xcite used elmo @xcite including feature tuning and show an impressive improvement of 5.1 f1 on the english conll 2012 test set over the baseline score of @xcite . the extension from an end-to-end to the differentiable higher-order inference provides an additional 0.7 f1-score on the test set, which leads to a final f1-score of 73.0 for this approach. @xcite use the same inference model and explore how to best use and gain another significant improvement of 3.9 points absolute and reach a score of 76.9 f1-score on the test set (see similarly, @xcite reduce the high memory footprint of mention detection by using the start-and end-points of mention spans to identify mentions with a bilinear scoring function. the top λn scored mentions are used to restrict the search space for coreferences prediction using again a bilinear function for scoring. the algorithm has a quadratic complexity since each possible coreference pair has to be scored. @xcite cast coreference resolution as question answering and report gains originating from pretraining on quoref and squad 2.0 of 1 f1-score on the development set. the approach first predicts mentions with a recall-oriented objective, then creates queries for these potential mentions for the cluster prediction. this procedure requires the application of the model for each mention candidate multiple times per document, which leads to high execution time. our work makes direct use of t5-based models @xcite . t5 adopts the idea of treating tasks in natural language processing uniformly as ''text-to-text'' problems, which means to only have text as input and generate text as output. this idea simplifies and unifies the approach for a large number of tasks by applying the same model, objective, training procedure, and decoding process. ## experimental setup we train a mt5 model to predict from an input a target text. we use the provided training, development, and test splits as described in section 4.1. for the preparation of the input text, we follow previous work and include the speaker in the input text before each sentence @xcite as well as the text genre at the document start if this information is available in the corpus. we apply as described in section 3 the corresponding transitions as an oracle to obtain the input and target texts. we shorten the text at the front if the input text is larger as the sentence piece token input size of the language model and add further context beyond the sentence i when the input space is not filled up (note that as described in section 3, we use the pipe symbol | to mark the start of the focus sentence and the end with two asterisk symbols **). ## discussion in this section, we analyze performance factors with an ablation study, analyze errors, and reflect on design choices that are important for the model's performance. ## conclusions in this paper, we combine a text-to-text (seq2seq) language model with a transition-based systems to perform coreference resolution. we reach 83.3 f1-score on the english conll-2012 data set surpassing previous sota. in the text-to-text framework, the link-append transition system has been superior to hybrid mention-link-append transition system with mixed prediction of mentions, links and clusters. our trained models are useful for future work as they could be used to initialize models for continuous training or zero-shot transfer to new languages.