Sequence to Sequence Coreference Resolution (2020)

Abstract: Until recently, coreference resolution has been a critical task on the pipeline of any NLP task involving deep language understanding, such as machine translation, chatbots, summarization or sentiment analysis. However, nowadays, those end tasks are learned end-to-end by deep neural networks without adding any explicit knowledge about coreference. Thus, coreference resolution is used less in the training of other NLP tasks or trending pretrained language models. In this paper we present a new approach to face coreference resolution as a sequence to sequence task based on the Transformer architecture. This approach is simple and universal, compatible with any language or dataset (regardless of singletons) and easier to integrate with current language models architectures. We test it on the ARRAU corpus, where we get 65.6 F1 CoNLL. We see this approach not as a final goal, but a means to pretrain sequence to sequence language models (T5) on coreference resolution.

## introduction coreference resolution is a natural language processing (nlp) task which consists on identifying and clustering all the expressions referring to the same real-world entity in a text. nlp tasks that include language understanding such as text summarisation @xcite , chatbots @xcite , sentiment analysis @xcite or machine translation @xcite can benefit from coreference resolution. and until recently, coreference resolution has been a critical task on the pipelines of those systems. however, with the recent rising trend of building end-to-end deep neural networks, for any nlp task where the data available in that language or domain is huge, current models are able to learn the end task without any explicit training on coreference resolution. this is even more evident in the case of the huge unsupervisedly pretrained language models (lm) that are already able to resolve coreference @xcite , as bert @xcite , roberta @xcite , t5 @xcite , or gpt3 @xcite which are used to boost results on any downstream task. those pretrained language models have also improved notably the results obtained at coreference resolution. combining the sota neural coreference resolution system @xcite at the time with pretrained language models (elmo, bert, spanbert) improves results by a large margin. despite coreference resolution was already useful in nlp end tasks before the irruption of deep learning in nlp, and getting very significant improvements on the results with it, nowadays most of the tasks that require deep language understanding, are approached without having coreference resolution in mind. src: even the smallest person can change the course of history . trg: in this paper, we introduce a new approach to solve coreference resolution as a sequence to sequence task (as shown in we test our approach on the english arrau corpus @xcite , which includes singletons. we train our model on coreference resolution as a sequence to sequence task, where the neural network learns to produce the coreference relations as output from the raw text in the source. in the following section 2 we review the state of the art of the field. in section 3 we describe how we approached coreference resolution as a sequence to sequence task, we present the neural architecture and corpora we used. in section 4 we report our results, and lastly, we present our conlusions and future work in section 5. ## state of the art the sota for english coreference resolution, improved a lot since the revolution of deep learning in nlp. the first end-to-end neural model @xcite obtained big improvements over previous models. since then, pretrained lms improved a lot those results; adding elmo @xcite , bert @xcite and spanbert @xcite to the model, improved by a large margins the sota at the moment @xcite @xcite . furthermore, we would like to underline different approaches as reinforcement learning @xcite and neural mcdm and fuzzy weighting techniques @xcite , which improved results. there have been only two works which already have tried to combine language models and coreference resolution at training. in the first one, t5 @xcite , they use coreference resolution among other tasks to train a neural language model on text to text, but the coreference task is approached as a simple binary mention-pair task, which does not reflect all the advances done at resolving coreference. in the second one, corefqa @xcite , they adress coreference resolution as query-based span prediction for which they convert coreference resolution into a qa task, where the model has to find the coreferential mentions in the text. although they get the best results obtained to this day, their approach still uses a windowing technique of length 512, and needs to create questions automatically from the text. models f1 @xcite 68.8 @xcite 73.0 @xcite 73.8 @xcite 76.6 @xcite 77.1 @xcite 79.6 @xcite 80.0 @xcite 83.1 we should keep in mind that, apart of the well studied english language, there are lots of other less researched languages. yet we already have neural models for some of those languages: polish @xcite , japanese @xcite , french @xcite , basque @xcite , telegu (annam et al., 2019), russian @xcite persian @xcite and cross-linguals @xcite with varied results depending on corpus sizes and architectures. ## sequence to sequence coreference resolution coreference resolution has been historically divided in two subtasks. the first one is mention detection, where posible candidates for a mention are located in the text. the second one would be to find those which have coreferential relations, among the mentions. this second task has been approached as a clustering problem, where mention-pair models evolved into entity-mention models, and their respectives ranking models. some of this approaches have issues with making the correct global decisions, and those who handle this more appropriately, have higher computational cost. in the following subsection, we present our approach, which solves these two subtasks at once in a simpler way. ## results for the evaluation of our new sequence to sequence approach and the transformer model we built, we use the coreference official scorer @xcite to get the results of the most used metrics on the task on the arrau testing split. we obtain 77.2 f1 at mention detection (md), 64.9 f1 at muc, 66.5 f1 at b 3 , 65.3 f1 at ceaf e and 65.6 f1 on the conll metric. they are quite good results for a simple approach which does not use any external information as pretrained word embeddings or lms, or any linguistic, distance or speaker features other than the auxiliary dataset we used, which just added the amount of raw text and its coreferential relations we had. our model is able to detect most of the mentions, including singletons, and it does cluster correctly correferential mentions to a certain extent, including those that are at a very long distance foot_2 . the best results on the arrau dataset are those presented at @xcite . results obtained in this work are not completely comparable with our work, as we do not process documents longer than 1024 tokens (∼800 words, keeping 72% of the documents), while they only test their system with the rst subset of the test set. however, we include the comparison in ## conclusions and future work all in all, in this work we present a novel approach, as far as we know, the first time where coreference resolution has been learned as a simple sequence to sequence task, using just a transformer, an architecture that rules the nlp field. we got 65.6 f1 conll on the arrau corpus, and despite not getting the best results on the dataset, we proved that a transformer is enough to learn the task, from raw text, without any features or pre-trained word-embeddings or lms. the results obtained are quite good, as this approach have room for improvements at architecture level, hyperparameter tuning, and the integration of pretrained lms. this approach may help at unifing the coreference resolution with other nlp models, where this task could be used at pretraining sequence to sequence lms (t5). our code and model are available at: https://github.com/gorka96/text2cor . there are many aspects of this approach worth to continue researching. to begin with, we limited the maximum length of the sequences to 1024 tokens for simplicity, nevertheless, to be able to process longer documents, we will need to train transformer models with longer maximum positions. to handle the increment in memory and computational costs, architectures that do not use full attention as reformer @xcite or longformer @xcite ) could be considered. moreover, we would like to verify that this method is as universal as we said here, trying datasets without singletons, lowresourced languages, and multilingual or cross-lingual settings. finally, using this approach to train a sequence to sequence language model like t5, would be interesting.