Neural End-to-End Coreference Resolution using Morphological Information (2023)

Abstract: In morphologically rich languages, words consist of morphemes containing deeper information in morphology, and thus such languages may necessitate the use of morpheme-level representations as well as word representations. This study introduces a neural multilingual end-to-end coreference resolution system by incorporating morphological information in transformer-based word embeddings on the baseline model. This proposed model participated in the Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023). Including morphological information explicitly into the coreference resolution improves the performance, especially in morphologically rich languages (e.g., Catalan, Hungarian, and Turkish). The introduced model outperforms the baseline system by 2.57 percentage points on average by obtaining 59.53% CoNLL F-score.

## introduction coreference resolution (cr) is the task of determining coreferential relations between mentions referring to the same real-world entity in a document. cr is one of the essential components of comprehending natural language and is investigated under the semantic level of natural language processing (nlp). an end-to-end cr system consists of two stages which are trained jointly: 1) mention detection and 2) coreference linking. in the first, all possibly referential mentions are extracted in a text. then, the coreferential relations between the automatically predicted mentions are created during the linking stage. when the cr task crosses with the complex linguistic diversity of natural languages, it becomes even more difficult, and morphological richness is one of such diversity. morphologically rich languages require considering sub-word units (or morphemes) which carry deeper information at the morphology level. therefore, this study explores the impact of including morphology informa-tion explicitly in a neural multilingual end-to-end cr system. moreover, even if cr is an actively studied topic for quite a long time, the multilingual studies are currently in the process of development. most studies propose cr datasets in their own data format and report their performances in one language only. the lack of quality and standardized datasets makes building multilingual cr systems harder. corefud initiative fills this gap in the cr literature by proposing a universal coreference representation scheme which was built on top of the universal dependencies @xcite @xcite initiative. in this paper, we propose a neural, multilingual, end-to-end cr model trained with the data convened in corefud v1.1 @xcite ; we extend the baseline model @xcite by enhancing the transformer-based word embeddings with dense and sparse (i.e., one/multi-hot encoding) vector representations of morphological information (i.e., pos tags and morphological features). the corefud v1.1 contains 17 different datasets for twelve languages in a harmonized, universal scheme. the proposed cr model employing sparse vector representations of morphological information achieves 59.53% conll score on the test set (average across all languages), which means a 2.57 percentage points improvement over the baseline. the results show that the impact of explicitly incorporated morphological information is particularly high in the cr performance of morphologically rich languages. the paper is structured as follows: section 2 gives the related work, section 3 introduces the proposed neural model in detail, section 4 presents the experimental setup and results, and section 5 gives the conclusion. ## related work deep learning-based cr approaches conforming to the end-to-end fashion have begun to be studied extensively in the last few years. @xcite proposes the first end-to-end neural cr system, which creates a base for later studies. this study is enhanced with transformer-based embeddings via bert @xcite and span-bert @xcite with the higher-order inference (hoi) mechanism on top which are featured by @xcite . moreover, @xcite proposes a neural cr system employing entity-based features which were obtained by graph neural networks. in parallel, @xcite introduce bert-sru-based pointer networks with the integration of morpheme boundaries as features for korean. there are many studies proposed in the previous multilingual cr shared task (crac 2022) @xcite . the winning team of the shared task was úfal corpipe with 2 of their 3 submissions being on the leaderboard. the best model, straka, is trained jointly on all training data in all languages, and provides 70.72% conll fscore by primary metric. ondfa @xcite is a baseline-based model using pre-trained xlm-robertalarge @xcite and also containing mention-head prediction. with the power of the mention-head prediction component, the model ends up getting a higher head-match score. k-sap @xcite is introduced for only polish. in addition to neural cr systems, rule-based models (berulasek, simple-rule-based, moravec) also exist in the crac 2022. available annotated cr datasets in the literature are in a lack of standardization, which makes the development of multilingual cr systems complicated. by means of the corefud scheme @xcite , a multilingual coreference dataset collection is established and the task is shaped into a more generalized form. in parallel, crac organizations encourage researchers to develop and submit their own systems utilizing corefud dataset under the shared representation. crac 2022 was organized with the corefud v1.0 @xcite containing 13 datasets for 10 languages. crac 23 is organized with corefud v1.1 release. this version consists of 17 different datasets for 12 languages. recent contributions involve hungarian with one dataset, turkish with one dataset, and norwegian with two datasets. these made turkish and norwegian to appear in the corefud collection for the first time. ## the proposed model the introduced model is a modified version of the baseline model provided in the crac 2023 shared task @xcite , with the span rep-resentations updated. the baseline model @xcite provides a multilingual, end-to-end neural cr system which is a re-implementation of an available study @xcite . basically, the model learns the probability distribution of coreferential links in the training data by maximizing the marginalized log-likelihood of gold antecedents for each possible span. to rank automatically detected referential mentions and link them with their possible antecedents, the model estimates the combination of two types of scores: 1) individual mention score, and 2) paired antecedent score. individual mention score represents the likelihood of a span being a referential mention. antecedent score entails a span pair and ranks their possibility of being coreferent. since spans are considered as a sequence of words, they are represented by their words' embeddings obtained from a transformer, i.e., bert. this study introduces an enhanced span representation by incorporating morphological information explicitly in addition to contextual embeddings obtained by bert. each span embedding consists of three main sub-parts foot_0 : the embeddings of its first and last tokens, and the head attended embeddings of all tokens, as formularized in equation ( 1 this study extends the first and last tokens' embeddings by incorporating one/multi-hot encoded morphological information explicitly. there are two types of morphological information utilized: universal part-of-speech (upos) and universal morphological features (feats). the output sample of this procedure is shown for the first token's embedding in the equation ( 2 ). the operation annotated by ⊕ is concatenation. therefore the size of e(s i ) is extended by the total unique number of universal pos tags and morphological features in the corefud collection. the same procedure is also applied to the last token's embedding. ## experimental setup & results this section introduces the performance of the proposed model and also intermediate results. ## conclusion this study proposed a neural, end-to-end, multilingual cr model which is an improved version of the baseline model incorporating morphological information into transformer-based span embeddings. the results show that extending word representations with morphological information helps cr systems on average but especially for languages with high morphological complexity and agglutinative characteristics (e.g., catalan, hungarian, norwegian, and turkish). the proposed model completed the crac 2023 shared task at 7 th place on average. besides, on individual dataset scores, our highest rank is on catalan (ca_ancora), which is the 5 th place. then it is followed by 6 th place on turkish (tr_itcc), hungarian (hu_korkor), german (de_potsdamcc), and english (en_parcorfull).