ÚFAL C or P ipe at CRAC 2022: Effectivity of Multilingual Models for Coreference Resolution (2022)

Abstract: We describe the winning submission to the CRAC 2022 Shared Task on Multilingual Coreference Resolution. Our system first solves mention detection and then coreference linking on the retrieved spans with an antecedent-maximization approach, and both tasks are fine-tuned jointly with shared Transformer weights. We report results of finetuning a wide range of pretrained models. The center of this contribution are fine-tuned multilingual models. We found one large multilingual model with sufficiently large encoder to increase performance on all datasets across the board, with the benefit not limited only to the underrepresented languages or groups of typologically relative languages. The source code is available at https://github.com/ufal/crac2022-corpipe .

## introduction coreference resolution is a task of identifying and clustering multiple occurrences of entities across a textual document. the crac 2022 shared task on multilingual coreference resolution @xcite features coreference resolution on 13 datasets in 10 languages, originating from the core-fud 1.0 multilingual dataset @xcite . coreference resolution is often divided into two subtasks, mention detection and coreference linking (also clustering). our contribution solves these tasks neither as a purely pipeline approach with two separate sequential models, nor as an end-to-end system @xcite , as is recently more common (e.g., the baseline; @xcite , but somewhere in between: we first solve mention detection and then coreference linking with an antecedent-maximization algorithm, but both tasks are jointly fine-tuned in one shared large language model. this circumvents the explosion of possible spans in an end-to-end approach, allows for a single retrieval of mentions only, while keeping the benefit of sharing the weights and training only one model for two highly related tasks (contribution 1). our architecture is a fine-tuned large language model, with experimental results leaning toward large pretrained models with better multilingual representation. we experimented with a wide range of pretrained language models @xcite @xcite @xcite @xcite @xcite @xcite @xcite , of which @xcite proved the most effective (contribution 2). we found multilingual models at the center of our research attention in the crac 2022 shared task. the shared task featured datasets with sizes ranging from tiny (457 training sentences in de and en parcorfull) to relatively large (nearly 40k training sentences in cs pcedt and cs pdt), all of them evaluated with equal weight (macro average). this implied that special care must be devoted to leveling the performance on all datasets. we experimented with various combinations of fine-tuned multilingual models and various sampling strategies. although our motivation was to mitigate the poor performance on smaller specimens, we surprisingly found that one large multilingual model with sufficiently large encoder improves results on all datasets, not only the small or linguistically related ones (contribution 3). to sum up, our contributions are the following: 1. we present a jointly trained pipeline approach for coreference resolution. 2. although many monolingual base models surpass their multilingual base counterparts, in the end, one large multilingual pretrained model gives the best performance over base, albeit specifically pretrained monolingual encoders. ## related work coreference resolution is often divided into two subtasks: mention detection and coreference linking (or clustering). these can be solved either separately (pipeline approach) or, more recently, in an end-to-end fashion @xcite . such was also the approach of the baseline @xcite . our proposal takes what we hope is advantageous from both approaches: we solve both tasks sequentially, but the weights are trained jointly in a shared network. as in all other nlp areas, deep learning with representations from large language models represents the current state-of-the-art @xcite @xcite . we build on these works which use bert @xcite by comparing bert with its successors, the language-specific mutations of bert @xcite @xcite @xcite @xcite @xcite @xcite , and multilingual variants: mbert @xcite , xlm-r base and xlm-r large @xcite , and @xcite . there are mixed accounts in the literature on globally decoding the entities (clusters) via higherorder methods. @xcite improved state-of-the-art on the conll-2012 shared task with differentiable end-to-end manner enabling higher-order inference: mentions are represented as the sum of all mentions of the entity (entity equalization). other higher-order coreference linking methods include attended antecedent @xcite @xcite . on the other hand, @xcite thoroughly investigated the contribution of higher-order methods to the models performance and conclude that with modern encoders, higher-order methods contribute only marginally or negatively. as we model all antecedent links during training in a dotproduct attention matrix, we inherently "equalize" entities, although not with an explicit algorithm. ## methods an overview of the model architecture is shown in figure 1 . in the following sections, we describe the components of the model in detail, with reference to the corresponding parts of figure 1 . ## results official results of the crac 2022 shared task on the test set can be found in ## conclusions we presented a jointly trained pipeline approach as a winning contribution to the crac 2022 shared task on multilingual coreference resolution @xcite . we published a thorough comparison of pretrained large language models for the task. finally, we focused on multilingual models and we conclude that one multilingual, all-data model with large encoder outperformed individual monolingual fine-tuned models on all datasets. the source code is available at https: //github.com/ufal/crac2022-corpipe.