Findings of the Shared Task on Multilingual Coreference Resolution (2022)

Abstract: This paper presents an overview of the shared task on multilingual coreference resolution associated with the CRAC 2022 workshop. Shared task participants were supposed to develop trainable systems capable of identifying mentions and clustering them according to identity coreference. The public edition of CorefUD 1.0, which contains 13 datasets for 10 languages, was used as the source of training and evaluation data. The CoNLL score used in previous coreference-oriented shared tasks was used as the main evaluation metric. There were 8 coreference prediction systems submitted by 5 participating teams; in addition, there was a competitive Transformer-based baseline system provided by the organizers at the beginning of the shared task. The winner system outperformed the baseline by 12 percentage points (in terms of the CoNLL scores averaged across all datasets for individual languages).

## introduction multilingual shared tasks are an important source of momentum in various subfields of nlp research, with the conll-x shared task on multilingual dependency parsing @xcite ) being one of the most successful and influential examples. clearly, the limiting factor for organizing such shared tasks is the availability of multilingual data whose annotations are harmonized at least to some extent, so that the experiments on individual languages can be performed and evaluated in a uniform way. in the coreference world, one of the first multilingual shared tasks were semeval-2010 @xcite with seven languages and @xcite , in which ontonotes data for three languages (english, chinese, and arabic) were included. with the recent advance of the corefud collection @xcite , harmonized coreference data for 10 languages (covered in corefud's publicly available edition) became available. hence, corefud is the source of data for the present shared task; more information about the collection is given in section 2. in brief, participants of this shared task are supposed to (a) identify mentions in texts and (b) predict which mentions belong to the same coreference cluster (i.e., refer to the same entity or event), using the corefud data both for training and evaluation of their coreference resolution systems. a specific feature of corefud is that it combines coreference with dependency syntax, using the annotation scheme (and file format too) of the universal dependencies (ud) project @xcite . in all datasets included in the collection, the coreference annotation is manual and the dependency annotation is either manual too, if available, or produced by a dependency parser. empirical evidence showing advantages of such symbiosis of coreference and dependency syntax is presented in two case studies @xcite . participants of this shared task can employ the dependency annotation for determining mention spans (as mentions often correspond to syntactically meaningful units) and for determining core parts of mentions (which correspond to syntactic head in corefud). to the best of our knowledge, this is the first shared task on multilingual coreference resolution that accepts zeros (e.g. elided subjects) as potential members of coreference chains. 1 zeros are an integral part of some of the corefud datasets, using empty nodes in enhanced ud representation to annotate them. we keep all annotated zeros, encouraging participants to resolve coreference also for this type of potential mentions. as with other shared tasks, evaluation is crucial. unfortunately, and unlike e.g. in dependency parsing, there is no simple and easily interpretable accuracy metric for coreference. we adhere to using the conll score developed in former coreference shared tasks. more specifically, we use an average of the f 1 values of muc, b 3 and ceaf-e scores as the main evaluation metric. more details concerning evaluation are presented in section 3. a transformer-based coreference prediction system @xcite was provided as a strong baseline to the shared task participants. the baseline system as well as 8 systems submitted by the participants are briefly described in section 4 and some of the systems are described in more detail in separate papers in this volume. their results are summarized in section 5. possible directions for future editions of the shared task are outlined in section 6. ## datasets for training and evaluation purposes, the present shared task uses 13 coreference datasets for 10 languages as available in the public edition of the corefud 1.0 collection @xcite and follows the train/dev/test split of the collection, too. ## evaluation metrics systems participating in the shared task are evaluated with the corefud scorer. 3 the primary evaluation score is the conll f 1 score with singletons excluded and using partial mention matching. we also assess the shared task submissions by multiple supplementary scores. official scorer we use our modification of the coreference scorer -corefud scorer. it is based on the universal anaphora (ua) scorer @xcite foot_3 reusing the implementations of all generally used coreferential measures without any modification. this guarantees that the measures are computed in exactly the same way. however, our scorer is capable of processing the coreference annotation files in the corefud 1.0 format. among other things, it allows evaluation of coreference for zeros. 5 moreover, it re-defines matching of key and response mentions in the way to be able to handle potentially discontinuous mentions, which are present in some corefud datasets. last but not least, we proposed and implemented the mm score to measure the accuracy of mention matching (see below). ## results and comparison the straka system by the úfal corpipe team is clearly the winner of the shared task. it surpasses other systems not only in terms of the primary score (see the primary column in in comparison to the baseline system, most systems ouperformed it by a relatively large margin. the winning straka system achieves over 12 points in the primary score, which is more than 20% improvement over the baseline performance. this is an extremely beneficial effect of the shared task, which may drive further development in multilingual coreference resolution. results unsurprisingly also confirm a doubtless dominance of machine learning approaches. although rule-based postprocessing has been employed by some teams (also encouraged by availability of the baseline predictions), its incorporation is either motivated by the nature of the primary score (straka* systems) or it improves upon the baseline only marginally (the berulasek system) or not at all (the moravec system). we observe almost the same picture in evaluation with singletons (see ## conclusions and future work this paper summarizes the outcomes of the multilingual coreference resolution shared task held with the crac 2022 workshop. we hope that the presented shared task establishes a new state of the art in multilingual coreference resolution. possible future editions of the shared task could be improved or extended along the following directions: