Multilingual coreference resolution: Adapt and Generate (2023)

Abstract: The paper presents two multilingual coreference resolution systems submitted for the CRAC Shared Task 2023. The DFKI-Adapt system achieves 61.86 F1 score on the shared task test data, outperforming the official baseline by 4.9 F1 points. This system uses a combination of different features and training settings, including character embeddings, adapter modules, joint pre-training and loss-based re-training. We provide evaluation for each of the settings on 12 different datasets and compare the results. The other submission DFKI-MPrompt uses a novel approach that involves prompting for mention generation. Although the scores achieved by this model are lower compared to the baseline, the method shows a new way of approaching the coreference task and provides good results with just five epochs of training.

## introduction coreference resolution is a task of finding all mentions referring to the same physical or abstract entity in the given piece of text. e.g., in sentences "i've never been to london before. but i heard it is a lovely place" the words london and it both refer to the real-world entity the city of london, and are called an antecedent and an anaphor respectively. coreference resolution includes two sub-tasks that can be done either in a pipeline manner, or jointly: mention detection and mention clustering. they are quite challenging: (i) antecedents can be split; (ii) mentions can be discontinuous; (iii) one needs to consider the semantics of the context; (iv) there are long-distance coreference relations, etc. coreference resolution contributes to the correct automatic text understanding, and is important for many nlp tasks, including text summarization and paraphrasing, information extraction, machine translation, question answering, etc. ## related work in this section we outline the main achievements in the area of multilingual coreference resolution, and present the approaches that are similar to our work. most progress in the area of multilingual coreference resolution was made due to the introduction of shared tasks. semeval-2010 task 1 @xcite was designed to evaluate and compare methods of coreference resolution in six languages (catalan, dutch, english, german, italian, and spanish) and used four different metrics: muc @xcite , b 3 @xcite , ceaf @xcite and blanc @xcite . there were six systems submitted for this task, all of them rely on feature extraction and machine learning algorithms, like maximum entropy, decision trees, support vector machines (svm), etc. only two systems, ubiu @xcite and sucre @xcite work for all the languages. @xcite was dedicated to predicting coreference in the ontonotes corpus @xcite which includes data in english, chinese, and arabic. the evaluation metrics included metrics used for semeval-2010 and a conll score representing an unweighted average of muc, b 3 and entity based ceaf. there were 16 systems submitted for conll-2012. the majority of them combine machine learning approaches mentioned earlier with the rule-based ones. the latter are typically used for mention detection. the best performing systems also heavily rely on feature engineering. as far as we can judge, most of the systems assume training a separate model for each language. in contrast to the previous shared tasks, crac-2022 @xcite offered much more data in different languages. the corefud 1.0 collection @xcite included 13 datasets in czech, english, polish, french, russian, german, catalan, spanish, lithuanian and hungarian which were harmonized to the same annotation scheme and data format. the primary evaluation metric was the conll score. the organizers offered a strong transformer-based baseline @xcite , which was also used for the current shared task. there were eight systems submitted.the absolute majority use deep learning approaches and rely on large pre-trained models. importantly, most of the systems present crosslingual models trained on all the multilingual data. it is actually difficult to compare all these models in terms of numbers and judge how much progress has been made since semeval-2010 for multilingual coreference resolution. first, the models were trained on quite different data. second, despite the unification of the annotations, the definition of a mention varies across the datasets. third, the evaluation criteria are also different, in the first place for mention boundaries detection. our dfki-adapt system uses a combination of different settings that includes pre-trained adapters. as far as we know, adapters @xcite have not been well researched for multilingual coreference resolution. adapters represent a small amount of additional parameters that can be added as trainable task-specific weights at each layer of the transformer architecture @xcite . they have been successful on a variety tasks including speech recognition @xcite , cross-lingual transfer @xcite and classification tasks @xcite @xcite but there is very little research on using adapters for coreference resolution and the only work that we are aware of uses parallel data for training @xcite . the idea of prompting llms for the task of coreference resolution is relatively new. there are not so many papers on this topic. e.g., @xcite do few-shot prompting to resolve anaphora that requires commonsense knowledge using the winograd schema challenge (wsc) corpus @xcite . @xcite perform similar experiments on the wsc and wino-grande @xcite -on ecb+ @xcite . @xcite and @xcite try prompting for coreference resolution in scientific protocols and medical domain, respectively. @xcite experiment with few-and zero-shot anaphora resolution in the multilingual xwinograd corpus @xcite . in contrast to our approach, all these models do not perform prompt fine-tuning, instead they typically include a few demonstrations into their prompts (therefore few-shot) and use much larger models, like xglm @xcite , gpt-2 @xcite , gpt-3 @xcite or in-structgpt @xcite . moreover, we use prompting only for mention generation. a somewhat similar approach but without prompting was presented by @xcite , who used the t5 model @xcite to generate semantic roles when doing frame-semantic parsing. ## data all our experiments are done using an officially provided public version of the corefud 1.1 data, which extends corefud 1.0 with new datasets in hungarian, norwegian and turkish. in total, this version of corefud consists of 17 datasets in 12 languages. the datasets vary a lot in their sizes (see intuitively, the quality of mention extraction and subsequent coreference resolution depends not only on the training data size, but also on length and complexity of the sentences and mentions, the number of mentions (including nested) in a sentence, the amount of unique named entities, etc. to get an idea about difficulty of the task, we collected some basic statistical facts about the relevant data properties. this information can be found in ## multilingual coreference resolution with dfki-adapt our submission dfki-adapt is based on the baseline provided by the organizers but extended in different ways to accommodate the multi-lingual nature of the task. the dfki-adapt system integrates character embeddings, joint pre-training and fine-tuning on the datasets of the related languages. it also includes additional re-training on the documents with the higher loss and uses adapter modules that were pre-trained for each dataset. the goal of the dfki-adapt submission is to demonstrate how one could get a substantial improvement over the baseline (+4.9 f1 points on the test and +9.07 f1 on the development partitions) without any additional data or larger models, just by leveraging the existing annotations. all experiments are performed with standard multilingual bert and the official crac data. the following sections introduce our baselines, the experiments with individual settings and the final results achieved by dfki-adapt. since the test data are not publicly available our evaluation is performed on the crac development set. the evaluation results on 12 datasets for different languages foot_0 are summarized in ## multilingual coreference resolution with dfki-mprompt in this section we first present our approach to mention identification as generation, then explain how we adapt the baseline to work with the mentions generated by our model. we discuss the results obtained by our system, analyse the mistakes and outline possible improvements. ## conclusion in this paper we presented our systems for multilingual coreference resolution. our dfki-adapt submission leverages the existing data in different ways including joint pretraining, integrating adapters, adding character embeddings and loss-based re-training. it achieves 61.86 f1 on the official test set and 68.06 f1 on the development set. we provide a comparison of different settings for 12 languages from the crac shared task. based on our analysis, joined pretraining with further fine-tuning on the respective dataset is the most beneficial setting per se but the largest gains can be achieved with the combination of different settings as implemented in the dfki-adapt system. our experiments also show that while injecting the pre-trained adapter weights can be helpful for many languages, these pre-trained weights should not be further updated during training. in the future we would like to experiment more with the language-specific vs. task-specific adapters and test whether cross-lingual transfer via adapters could further improve the performance on the coreference resolution task. our second submission dfki-mprompt relies on a novel prompt-based approach for mention identification. it generates all possible mention strings together with their indices, given a sentence. although the obtained scores were lower than baseline scores for the majority of the datasets, our method still has some potential. first, it can be improved by applying a better template, more optimal hyperparameters and a larger model. second, it could be used as an additional tool helping spanbased mention-ranking state-of-the-art models find mentions that are especially challenging for them, like split antecedents or discontinuous mentions. as a possible next step we plan experiments to check if our approach is capable of such a task.