ÚFAL C or P ipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution (2023)

Abstract: We present CorPipe, the winning entry to the CRAC 2023 Shared Task on Multilingual Coreference Resolution. Our system is an improved version of our earlier multilingual coreference pipeline, and it surpasses other participants by a large margin of 4.5 percent points. CorPipe first performs mention detection, followed by coreference linking via an antecedent-maximization approach on the retrieved spans. Both tasks are trained jointly on all available corpora using a shared pretrained language model. Our main improvements comprise inputs larger than 512 subwords and changing the mention decoding to support ensembling. The source code is available at https://github.com/ufal/crac2023-corpipe.

## introduction the goal of coreference resolution is to identify and cluster multiple occurrences of entities in the input text. the crac 2023 shared task on multilingual coreference resolution @xcite aims to stimulate research in this area by featuring coreference resolution on 17 corpora in 12 languages from the corefud 1.1 dataset @xcite . the current shared task is a reiteration of the previous year's crac 2022 shared task @xcite . corpipe, our entry to the crac 2023 shared task, is an improved version of our earlier multilingual coreference pipeline @xcite , which was the winner of the last year's shared task. our system first performs mention detection, followed by the coreference linking via an antecedent-maximization approach on the retrieved spans. however, corpipe is not a pure pipeline, because we train both tasks jointly using a shared pretrained language model. performing mention detection first avoids the challenge of end-to-end systems that need to consider an overwhelming number of possible spans, and also permits recognition of single-mention entities. finally, all our models are multilingual and are trained on all available corpora. our contributions are as follows: ## related work while coreference resolution was traditionally carried out by first performing mention detection followed by coreference linking (clustering), recent approaches are often end-to-end @xcite . likewise, the baseline of crac 2022 and 2023 shared tasks @xcite as well as the crac 2022 second-best solution @xcite follow this approach. the recent work of @xcite pushes the end-to-end approach even further, solving both mention detection and coreference linking jointly via a text-to-text paradigm, reaching state-of-the-art results on the conll 2012 dataset @xcite . given that our system uses the same pretrained encoder but a custom decoder designed specifically for coreference resolution instead of a general but pretrained decoder, it would be interesting to perform a direct comparison of these systems. coreference linking charles university is the oldest university in czech republic . it is ... mt5 large/xl token representation, dim charles university is the oldest university in czech republic . it is mention detection dense layer + relu, dim tag logits: dense layer, dim number_of_tags push pop 1 0 0 0 push, pop 1 0 push pop 1 0 push, pop 1 0 first token rep. last token rep. first token rep. last token rep. first token rep. last token rep. first token rep. last token rep. pretrained masked language model hidden: dense layer + relu, dim (mention representation): dense layer, dim hidden: dense layer + relu, dim (antecedent representation): dense layer, dim charles university is the oldest university in czech republic . it is inference only: dynamic decoding algorithm producing correctly balanced sequence of instructions ## corpipe architecture the corpipe architecture is based heavily on our earlier system @xcite , which won the crac 2022 shared task @xcite . we describe just the changes we propose; please refer to @xcite for the description of our original system. in short, our system first obtains a contextualized representation of the input by employing a pretrained model. these representations are then used first to perform mention detection, and then, together with the predicted mentions, to perform coreference linking. the mentions are predicted one sentence at a time, but both previous and following contexts are included up to the specified context length. the architecture overview is displayed in figure 1 . ## ablations on the development set to evaluate the effect of various hyperparameters, we perform further experiments on the development set. because we observed a significant variance with different random seeds and we also observed divergence in some training runs, we devised the following procedure to obtain credible results: for each configuration, we perform 7 training runs and keep only the 5 ones with the best overall performance. we then want to perform early stopping for every corpus. however, choosing for every corpus a different epoch in every run could lead to maximization bias in case the results oscillate considerably -therefore, for every corpus, we choose the single epoch achieving the highest average 5-run score (i.e., we use this epoch for all 5 runs). finally, we either average or ensemble the 5 runs for every corpus.