Findings of the Second Shared Task on Multilingual Coreference Resolution (2023)

Abstract: This paper summarizes the second edition of the shared task on multilingual coreference resolution, held with the CRAC 2023 workshop. Just like last year, participants of the shared task were to create trainable systems that detect mentions and group them based on identity coreference; however, this year’s edition uses a slightly different primary evaluation score, and is also broader in terms of covered languages: version 1.1 of the multilingual collection of harmonized coreference resources CorefUD was used as the source of training and evaluation data this time, with 17 datasets for 12 languages. 7 systems competed in this shared task.

## introduction the idea of a shared task focused on resolving coreference for multiple languages goes back to semeval-2010 (recasens et al., 2010) with seven languages and conll-2012 (pradhan et al., 2012) with three languages included. the amount of languages has been extended to 10 languages (with multiple datasets for some of them) in the multilingual coreference resolution shared task at crac 2022 (žabokrtský et al., 2022), making use of the corefud 1.0 collection (nedoluzhko et al., 2022). this paper reports on the second edition of this shared task organized in 2023, 1 associated with crac again. in brief, the most important improvements in this year's edition are the following. first, the shared 1 https://ufal.mff.cuni.cz/corefud/ crac23 task employs a newer version of the corefud collection. corefud 1.1 contains updated versions of 13 datasets (for 10 languages) already included in corefud 1.0, one new dataset for (already included) hungarian, and 3 new datasets for newly added languages: 2 for norwegian and 1 for turkish. second, the original morpho-syntactic features in the development and test sets were replaced by the output of udpipe 2 (straka, 2018) to make the evaluation scheme more realistic (with gold feature values being available, coreference prediction might be simplified to some extent, compared to real-world application scenarios). third, we use the head-matching approach for mentions in the primary score in this year's edition instead of partial matching. last year, partial matching led several teams to optimize their predicted mentions by reducing them to their syntactic heads, thereby losing the information about full mention spans. the remainder of the paper is structured as follows. section 2 focuses on changes of this shared task's data compared to the previous edition. section 3 explains the evaluation metrics -the primary score as well as the supplementary onesemployed in the shared task. section 4 describes the baseline system and the 7 participating systems. section 5 summarizes the results. section 6 concludes. ## datasets like the previous year, the shared task draws its training and evaluation data from the public part of the corefud collection (nedoluzhko et al., 2022), foot_0 now in its latest release (1.1). @xcite there are 17 datasets for 12 languages (3 language families). compared to corefud 1.0, which was used in the previous year of the shared task, there are 4 new datasets and 2 new languages (1 new language family): hungarian korkor, norwegian narc (bokmål and nynorsk versions), and turkish itcc. corefud ensures that the datasets are unified at the file format level: they use the conll-u format with extra annotation in the last column. @xcite the data have not been sufficiently harmonized at the level of annotation guidelines (for example, different datasets may have different rules for the extent of a mention). we follow the official train/dev/test splits of corefud 1.1. ## evaluation metrics systems participating in the shared task are evaluated with the corefud scorer. @xcite the primary evaluation score is the conll f 1 score with singletons excluded and using head mention matching, which is a change to the last year's edition, where partial mention matching was used in the primary score. in addition, we calculate several other supplementary scores to compare the shared task submissions. ## conclusions and future work both editions of the shared task attracted a substantial number of participants and led to an increase in the state of the art. hence, the success of the two completed shared tasks supports us in the idea of continuing this initiative in the future. however, there are challenges, too. for instance, the underlying data collection is still somewhat limited from the typological perspective, and thus our ambition is to add more languages with substantially different typological structures, experiment with other writing systems, or add a historical perspective with data from classical languages. there are also more technical questions that would deserve a discussion in the future, such as whether weightless macro-averaging is the best approach for data collections with order-ofmagnitude differences in training and testing data sizes. similarly, substantial differences in internal annotation consistency in individual resources is also an issue from the evaluation viewpoint, since, for example, optimizing performance for a lowquality resource might lead to substantial performance gains, which, however, may correspond to systematic deficiencies present in the data rather than objective quality. finally, we aim to progress to a fully realistic evaluation setup which starts from raw or pretokenized text. participants would be then expected to reconstruct zeros.