Multilingual Coreference Resolution with Harmonized Annotations (2021)

Abstract: In this paper, we present coreference resolution experiments with a newly created multilingual corpus CorefUD (Nedoluzhko et al.,2021). We focus on the following languages: Czech, Russian, Polish, German, Spanish, and Catalan. In addition to monolingual experiments, we combine the training data in multilingual experiments and train two joined models - for Slavic languages and for all the languages together. We rely on an end-to-end deep learning model that we slightly adapted for the CorefUD corpus. Our results show that we can profit from harmonized annotations, and using joined models helps significantly for the languages with smaller training data.

## introduction coreference resolution is the task of finding language expressions that refer to the same real-world entity (antecedent) of a given text. sometimes the corefering expressions can come from a single sentence. however, the expressions can be one or more sentences apart as well. it is necessary to see the whole document in some hard cases to judge whether two expressions are corefering adequately. this task can be divided into two subtasks. identifying entity mentions, and grouping the mentions together according to the real-world entity they refer to. the task of coreference resolution is closely related to anaphora resolution -see @xcite to compare these two tasks. one of the challenging difficulties of coreference resolution lay in linguistically complicated annotations. some examples of linguistic complications are split antecedents (a mention refer to more than one real-word entities), near identity relations, anaphoric and cataphoric relations, etc @xcite . in this paper, we rely on a corefud corpus @xcite of harmonized annota-tions. this corpus enables us to battle linguistic complications since it simply presents corefering mentions in clusters. since the corpus is compiled from 11 different corpora in 8 different languages, we can conduct multilingual experiments in this work. our research goal is to evaluate whether the harmonized annotations open the possibility to obtain some performance gain by joint learning on multiple languages. we aim to compare the harmonized annotations with the original corpora as well. ## related work in agreement with many other nlp tasks, deep learning models prevail in the coreference resolution task. @xcite were first to introduce the end-to-end approach that many following papers adopted (they obtained an average of 67.2 of f1 score). the task experienced a big leap in performance with the introduction of large pre-trained models. bert based models deliver the best results; @xcite @xcite f1 76.9. @xcite came up with a new pretraining task focused on better span representations. their model called spanbert brings additional improvements in the coreference resolution task (f1 79.6). xu and choi (2020) question the importance of modeling higherorder inference (hoi). they show that with advanced encoders, hoi has only a minor effect on the performance of models. research is significantly less evolved for other languages than english. however, some notable experiments were published in recent years. @xcite describe multilingual experiments (for english, catalan and spanish, dutch, german and italian) similarly to our paper. however, the annotations were not harmonized as in our case. therefore, they provide no experiments with joint training. other cross-lingual experiments include portuguese by learning from spanish @xcite ; spanish and chinese relying on an english corpus @xcite ; and basque based on an english corpus as well @xcite . all these approaches employ neural networks, and they transfer the model via cross-lingual word embeddings. treex cr @xcite is a coreference resolution module in the treex nlp framework 1 . it produces an advanced syntactic analysis with semantic features that the tool uses to find coreference relations -offers models for czech and english. other non-english experiments include polish @xcite , russian @xcite , and german @xcite . ## dataset for our experiments, we use the harmonized multilingual coreference dataset corefud @xcite . the dataset was created by converting 17 existing datasets for 11 different languages into a common format on the top of universal syntactic annotations -universal dependencies. for coreference representation, a cluster-based approach was selected instead of the link-based approach. it is simpler and moreover the most frequently used dataset for english -ontonotes adopt this approach too. in a cluster-based approach, every mention belongs to one cluster, represented by a unique id. in a link-based approach, coreferences are expressed by the links between corefering mentions. in the link-based approach, coreference structures form a chain, but there are more complex coreference structures in some cases @xcite . datasets that use the link-based approach were converted to cluster-based at the cost of some information loss. there are some notable differences between the datasets. one of the most prominent ones is the presence of singletons. singletons are clusters that contain only one mention. singletons are not present in any coreference relation. however, they are annotated as mentions in all datasets. discontinued mentions represent another notable difference. a discontinuous mention consists of a sequence of words that is interrupted at least once with some words that do not belong to the mention. such mentions can cause problems to models that assume 1 https://github.com/ufal/treex mentions to be continuous (such as our model). ## model we use the basic end-to-end model from @xcite with no higher-order inference (hoi), so it is the same model as it was proposed by @xcite . in the model, we start by modeling the probability p (y i |d) of a mention i corefering with the antecedent y i in a document d. since the model adopts the end-to-end approach, the mentions are identified together with the coreference links. we consider every continuous sequence of words as a mention i. therefore, we work with @xmath0, where t is the number of words in a document d. we model the score of a mention i corefering with an antecedent y i as a combination of two types scores s m (i) and s a (i, y i ). the s m is a score of a sequence of words (spans) i being a mention. the s a (i, y i ) score is the score of a span y i being an antecedent of span i. the scores are combined as a sum of s m (i), s m (y i ) and s a (i, y i ) as follows: the probability of an antecedent y i can be expressed as the sof tmax normalization over all possible antecedents @xmath1: the formula for all antecedents uses a product of multinomials of all individual antecedents: in the training phase, we maximize the marginal log-likelihood of all correct antecedents: ## experiments first, we perform monolingual experiments with the model described in section 4 on several largest datasets from corefud. namely czech, russian, polish, spanish, catalan, and german-potsdamcc. the employed datasets are summarized in as the next step, we perform multilingual experiments, where we train one model for all the slavic languages (czech, russian, and polish) and another model for all the languages (czech, russian, polish, german, spanish, and catalan). multilingual results in comparison with the monolingual ones are shown in the results in we report the average f1 measure from the official scoring script foot_0 for the coreference resolution task in conll. the metric is computed as the average of m u c, b 3 and cleaf 4 . definition of these metric can be found in @xcite . the f1 scores are reported with 95% confidence intervals measured from 5 runs. we use the same set of hyperparameters for all the languages and for all models. we train the models for approximately for 100k steps. we employ the adam optimizer with the learning rate of 0.00001 for bert layers and 0.0002 for other layers. ## discussion from the result (see for the smallest datasets (russian and german), multilingual models outperform the monolingual ones by a large margin (cca 2 -6 % f1). we can see that at least in small training datasets, using joined models definitely helps, and we can profit from the harmonized coreference annotations. it is also worth noticing that the confidence intervals for these datasets are significantly wider than for other datasets. after the singleton filtering the employed model achieves good results for all languages tested. for german, there are 6.3% of discontinuous entity mentions. the model iterates over all possible continuous spans. therefore, it is not able to identify discontinuous mentions at all. for geraman, the effect of singletons and discontinuous mentions ## future work currently, we experimented only on a subset of languages available in corefud. this was caused mainly by the resource-exhaustive training. we need 32gb graphic cards to capture long-enough contexts. we plan to experiment with the rest of the languages in the future. additionally, it would be interesting to explore the possibilities of zero-shot cross-lingual transfer in corefud, where we will not use the training data for the target language at all. ## conclusion we performed pilot experiments to evaluate interlanguage transferability of the models based on the corefud dataset. to do so, we used an end-to-end coreference resolution model based on bert-like models. our experiments show that learning from multiple languages in corefud annotation scheme helps significantly especially for languages with smaller training data (like russian and german data in corefud).