Silver Data for Coreference Resolution in U krainian: Translation, Alignment, and Projection (2023)

Abstract: Low-resource languages continue to present challenges for current NLP methods, and multilingual NLP is gaining attention in the research community. One of the main issues is the lack of sufficient high-quality annotated data for low-resource languages. In this paper, we show how labeled data for high-resource languages such as English can be used in low-resource NLP. We present two silver datasets for coreference resolution in Ukrainian, adapted from existing English data by manual translation and machine translation in combination with automatic alignment and annotation projection. The code is made publicly available.

## introduction coreference resolution is a task that requires clustering the mentions in text that refer to the same underlying entity @xcite . for example, in a sentence "john asked dan to drive him to work because his car was broken.", "john", "him", and "his" belong to the same cluster. in the past few years, significant progress has been achieved for english coreference resolution. to compare, in 2017 the average f 1 score of the model with the best performance in the conll-2012 shared task @xcite was 67.2 @xcite , and in 2021 the score of the best model increased to 81.0 @xcite . while the task of entity coreference resolution is relatively well-researched in english, it remains uncharted territory for many other languages. models developed with english in mind often fail to perform on the same level if used for a different language @xcite . one of the prevalent issues when working with low-resource languages is the lack of annotated data. it is often complicated to find or compile high-quality datasets even for english, and labeled data for complex tasks in other languages is much rarer. such data scarcity hinders nlp progress, as many state-of-the-art models require large amounts of labeled texts for training which are not available for low-resource languages @xcite . building new high-quality datasets is essential for expanding nlp research to the lowresource setting. one way of mitigating the data scarcity issue is adapting the data collected for high-resource languages. in this paper, we present two silver datasets annotated for coreference resolution in ukrainian, both of which are built using existing english labeled data as a basis. first, we manually translated the winograd schema challenge dataset @xcite into the ukrainian language. this dataset contains pairs of sentences with an ambiguous anaphor that can only be resolved using world knowledge and reasoning. second, we used a machine translation model to translate texts from ontonotes 5.0 @xcite into ukrainian, followed by automatically aligning and projecting annotations based on the cross-attention layer of an encoder-decoder model. our approach allows efficiently creating silver datasets based on existing high-quality data, which can then be used to extend language model training to low-resource languages. ## related work in this section, we analyze different techniques commonly utilized to help with data scarcity issues in multilingual nlp. ## ukrainian coreference resolution in this section, we present a survey of existing research for coreference resolution in ukrainian. hlybovets (2018) focuses on building complex information processing systems using a concept of agent-based modeling. a coreference resolution module is presented as part of the bigger system. for detecting mentions, the author uses a rule-based ner system based on a generalized left-to-right (glr) parser; the system can detect per, org and loc entities. a manually annotated news corpus is used for ner testing; the system achieves 0.48 f 1 score. to determine if the mentions are coreferent, methods from @xcite and @xcite are adapted: the resulting system uses a multi-pass filtering sieve together with a decision tree classifier. the author does not report accuracy scores for this part of the system. @xcite attempt to create a coreference resolution system for ukrainian using a convolutional neural network. following @xcite , coreference resolution is presented as a clustering task. every entity in the text is considered a separate cluster at the initialization step. the task of the model is then to go over pairs of clusters and merge the ones that refer to the same entity. for creating the clusters, the system proposed in @xcite uses a rule-based filtering sieves module and a multichannel cnn module. the rules are mostly based on direct string comparison with regular expressions, although some of them incorporate dictionaries of entity names scraped from wikipedia. then, pairs of clusters are given as an input to a convolutional neural network. clusters are represented by averaging the word2vec embeddings of the corresponding entity words. the cnn module works as a binary classifier; to train it, the authors use the searn method adapted from @xcite . a dataset of ukrainian news articles is used for training, testing, and evaluation. the model achieves 92.11 f 1 score for the b 3 coreference evaluation metric. in @xcite the authors continue the work presented previously, now making some important changes. first, bilstm is trained instead of a cnn. second, they perform feature analysis and conclude that word embeddings used for mention representation contribute a lot to the result, so they turn to elmo embeddings instead of word2vec used previously. as for the results, the proposed model achieves 92.21 f 1 score for the b 3 coreference evaluation metric. another important contribution is that both the pre-trained model and the dataset are made available to the public, which allows using them as a baseline for continuing research in this direction @xcite . the model can be used for different tasks; three endpoints are available to extract mentions, estimate coherence of the text and extract coreferent pairs. it also attempts to perform pos tagging and extract other grammatical features described in the previous checkpoint of the evaluated texts. the coreferent clusters dataset presented by @xcite is, to our knowledge, the only publicly available dataset for coreference resolution in ukrainian. it is distributed via mendeley as a mysql database. the database contains a single table word with the texts and relevant information about the tokens: their parts of speech, case, animacy, gender, number, aspect, and mood. all mentions are labeled, including singletons, which are potentially coreferent but appear only once in a document. each non-singleton mention belongs to a coreferent cluster. it is unclear whether the dataset was annotated manually or automatically, as the authors provide no specifics about the corpus creation process. overall, the task of coreference resolution in ukrainian remains underresearched. high-quality annotated datasets are needed to appropriately evaluate the performance of existing models as well as train new ones using state-of-the-art nlp methods. ## data in this section, we describe the base english datasets used to form the silver ukrainian datasets. we explore a total of two source datasets: a smaller test dataset that allows for testing coreference resolution systems ability to deal with complicated anaphora ambiguity cases, as well as a large dataset commonly used for training coreference resolution models. ## silver data creation in this section, we describe the methods and the process of creating silver ukrainian coreference resolution datasets on the basis of high-quality english data. ## error analysis in this section, we provide an analysis of the formed silver ukrainian datasets. ## conclusion creating new datasets is crucial in order to extend nlp research to the low-resource setting. labeled data for languages such as english can be effectively utilized for this task. we present an approach to efficiently create silver data corpora for lowresource languages based on existing annotated data for high-resource languages such a english with machine translation, alignment, and annotation projection. we demonstrate how the suggested methods can be used to create two corpora for training and testing coreference resolution models for ukrainian. the scripts for automatic translation, alignment, and projection, as well as the ukrainian wsc dataset are made publicly available foot_1 . future work will involve training and evaluating a baseline model using the created silver ukrainian datasets. the suggested approaches may be improved by using different machine translation models or trying out better alignment methods.