M c G ill at CRAC 2023: Multilingual Generalization of Entity-Ranking Coreference Resolution Models (2023)

Abstract: Our submission to the CRAC 2023 shared task, described herein, is an adapted entity-ranking model jointly trained on all 17 datasets spanning 12 languages. Our model outperforms the shared task baselines by a difference in F1 score of +8.47, achieving an ultimate F1 score of 65.43 and fourth place in the shared task. We explore design decisions related to data preprocessing, the pretrained encoder, and data mixing.

## introduction the goal of the crac 2023 shared task @xcite is to evaluate coreference resolution models on the corefud 1.1 collection of datasets @xcite . in this paper, we describe our submission to the task, which is an adaptation of the entity-ranking model described in @xcite with some exploration of the design decisions needed to apply this model to multiple datasets spanning multiple languages. our final submission achieves fourth place out of nine submissions based on head-match f1 score, and third place based on exact-match f1 score. the crac 2023 shared task is specifically based on the public portion of corefud 1.1, which includes 17 datasets spanning 12 languages: catalan, czech, english, french, german, hungarian, lithuanian, norwegian, polish, russian, spanish, and turkish. for the final evaluation, gold and predicted mentions are considered matching if they have overlapping head words, referred to as headmatch score, and the conll f1 head-match score is then macro-averaged over all 17 datasets. for our submission, we adapt the model described in @xcite , which is based on the entity-ranking model originally proposed by @xcite . we explore design decisions necessary to apply this english-based model to multilingual coreference resolution: data preprocessing steps, the pretrained language model encoder, and methods of joint training. our best configuration outperforms the shared task baselines by a difference in head-match f1 score of +8.47, achieving an ultimate score of 65.43. ## related work shared tasks have been instrumental in the development and evaluation of coreference resolution systems. previous examples include conll 2011 @xcite @xcite , and gap @xcite . the crac 2023 shared task builds off the previous iteration, crac 2022 @xcite , with some modification of the datasets and evaluation procedure. entity-ranking models @xcite of coreference resolution function by ranking a set of candidate entities to which each mention might refer. @xcite proposed a competitive neural entity-ranking model that processes mentions incrementally left-to-right. we analyze this method as implemented by @xcite . in contrast to existing work, we explore the potential of this model for multilingual generalization. the best model of the previous crac 2022 shared task was that of @xcite , which consists of two stages: mention detection and coreference linking. the authors found that jointly training on multiple datasets led to better performance on the shared task than training several models, one per each individual dataset. the same finding was found in other submissions as well @xcite . existing analyses have considered the generalization of entity-ranking models across datasets, including when jointly trained on multiple datasets @xcite @xcite . although such work has focused on english-language coreference and not evaluated generalization to a multilingual collection of datasets. it is not clear, a priori, how well the constraints of an entity ranking model will generalize to phenomena not present in english coreference datasets such as zero anaphora. ## model we evaluate the entity-ranking model implemented by @xcite . in this section, we first overview the model configuration and then outline the design decisions that we explore related to preprocessing, the pretrained encoder, and joint training. the high-level idea of the model is to first use a mention scorer to produce a set of mention candidates, then process the mentions left-to-right to determine if they refer to either a new or existing entity. configuration we start with the implementation and hyperparameters of @xcite . the model calculates coreference clusters for a document in the following way: first, embeddings are calculated for all spans @xmath0. each span embedding is scored using the mention scoring head described in @xcite , which is based on that originally proposed by @xcite . this scoring head is trained with binary cross entropy loss to assign a positive score to annotated mentions and a negative score to all other spans. the top 0.4 × ℓ spans are considered as mention candidates and kept for the next step, where ℓ is the length of the document in terms of subword tokens. this set of mention candidates is further filtered by removing all spans with a negative score. then, the set of entities is initialized as @xmath1. when processed, each candidate m is scored against all entities @xmath2, e). if ∀@xmath3, s(m, e) < 0 then m is added to the set e as a new entity. otherwise, m is said to belong to the entity representation with the highest score e * = argmax @xmath4, e) and the representation of e * is updated to be the mean of all mention representations that the entity represents thus far. this method is referred to as the unbounded memory (u-mem) model in the original work. for training we use the default hyperparameters except for those that are specific to the pretrained encoder or number of training steps. we use the default optimizer of adamw with a learning rate of 1e-5 for the pretrained encoder and 3e-4 for all other parameters. ## results in this section we first present the results experimenting with each design decision, and then present the final submission performance. in preliminary experiments, we micro-average conll f1 scores across all datasets for simplicity. for the final evaluation, conll f1 scores are macroaveraged across datasets. ## conclusion we adapt an entity-ranking coreference resolution model to multilingual coreference resolution for the crac 2023 shared task. we explore the method of training and joint encoder, finally using xlm-r large and a rescaled dataset weighting in our submission. this method achieved fourth place of nine submissions in the shared task.