End-to-end Multilingual Coreference Resolution with Mention Head Prediction (2022)

Abstract: This paper describes our approach to the CRAC 2022 Shared Task on Multilingual Coreference Resolution. Our model is based on a state-of-the-art end-to-end coreference resolution system. Apart from joined multilingual training, we improved our results with mention head prediction. We also tried to integrate dependency information into our model. Our system ended up in third place. Moreover, we reached the best performance on two datasets out of 13.

## introduction coreference resolution is the task of finding language expressions that refer to the same real-world entity (antecedent) of a given text. sometimes the corefering expressions can come from a single sentence. however, the expressions can be one or more sentences apart as well. it is necessary to see the whole document in some hard cases to judge whether two expressions are corefering adequately. this task can be divided into two subtasks. identifying entity mentions, and grouping the mentions together according to the real-world entity they refer to. the task of coreference resolution is closely related to anaphora resolution -see @xcite to compare these two tasks. this paper describes our approach to the crac 2022 shared task on multilingual coreference resolution. the task is based on the corefud dataset @xcite . the corefud corpus contains 13 different datasets for ten languages in a harmonized scheme. as the corefud is meant to be the extension of universal dependencies with coreference annotation, all the datasets in corefud are treebanks. for some languages, human annotators provided the dependency annotations. for others, the annotation is created automatically with a parser. the coreference annotation is built upon the dependencies. this means that the mentions are subtrees in the dependency tree and can be represented with the head. in fact, in some of the datasets, there are non-treelet mentions -the mentions which do not form a single subtree. but even for these non-treelet mentions, a single headword is selected. there are some notable differences between the datasets. one of the most prominent ones is the presence of singletons. singletons are clusters that contain only one mention. singletons are not present in any coreference relation. however, they are annotated as mentions. for details about the dataset, please see @xcite or @xcite . the task was simplified to predict only non-singleton mentions and group them into entity clusters. for evaluation, the corefud scorer foot_0 is provided. the primary evaluation score is the conll f 1 score with partial mention matching and singletons excluded. in the corefud scorer, a system mention matches a gold mention if all its words are included in the gold mention, and one of them is the key head. this means that the minimal correct span is the head, and it might be beneficial to predict mentions as only the heads. ## model our model builds on the official transformer-based end-to-end baseline @xcite . the underlying neural end-to-end coreference resolution model was originally proposed by @xcite . the model predicts the antecedents directly from all possible mention spans without a previous discrete decision about mentions. in the training phase, it maximizes the marginal log-likelihood of all correct antecedents: the model achieves state-of-the-art performance on the ontonotes dataset where singletons are not annotated. we believe the model is optimal for the corefud dataset as well since some of the datasets of the corefud do not contain singletons. moreover, the evaluation metric ignores singletons, so it does not matter that the model is not able to predict them. ## training we trained all the models on nvidia a40 graphic cards using online learning (batch size 1 document). we limit the maximum sequence length to 6 nonoverlapping segments of 512 tokens. during training, if the document is longer than 6 × 512 tokens, a random segment offset is sampled to take random continuous block of 6 segments, and the rest of them is discarded. during prediction, longer documents are split into independent sub-documents (for simplicity, non-overlapping again). we train a model for each dataset for approximately 80k updates in our monolingual experiments. for joinedpre-trained models, we use 80k steps for model pre-training on all the datasets and approximately 30k for fine-tunning on each dataset. each training took from 8 to 20 hours. ## results & discussion results of several variants of our model are presented in monoling column shows the result of the monolingual model specific for each language. xlm-r column presents results of xlm roberta large trained for each dataset separately. joined is the joined model described in the model section. the columns marked with + mean the best model from all to the left, with the additional feature. +dev means that the dev data part was added to training data, +s2h is the model with mention head prediction described earlier. both methods for mention head prediction have statistically equal performance (we cannot tell which one is better). the reported numbers are for the first one. the results in column +tree correspond to adding the dependency structure as described. it is not surprising that employing a larger model (xlmr-r large or monolingual) significantly improved the performance of the baseline. the results of the joined model are much more interesting. we can see that for some smaller datasets (e. g. german), the performance gain is huge. but if we have a look at ## conclusion we successfully applied an end-to-end neural coreference resolution system to the crac 2022 shared task. there are two main outcomes of our work. 1) joined training helps a lot. our experiments support the fulfillment of the goals of the corefud dataset to help the models by harmonizing the annotation schemas. 2) for the official scoring metric, predicting only the mention heads increases performance. this means that syntactic structure helps to identify mentions. of course, such evaluation is a bit artificial and does not reflect the real-world scenario, where we do not have the gold syntax. we also suggested injecting syntactic information into the model. unfortunately, we did not manage to get any improvement with this approach. our system ended up in 3 rd place. moreover, we reached the best performance on two datasets out of 13.