Named Entity Recognition for Low-Resource Languages - Profiting from Language Families (2023)

Abstract: Machine learning drives forward the development in many areas of Natural Language Processing (NLP). Until now, many NLP systems and research are focusing on high-resource languages, i.e. languages for which many data resources exist. Recently, so-called low-resource languages increasingly come into focus. In this context, multi-lingual language models, which are trained on related languages to a target low-resource language, may enable NLP tasks on this low-resource language. In this work, we investigate the use of multi-lingual models for Named Entity Recognition (NER) for low-resource languages. We consider the West Slavic language family and the low-resource languages Upper Sorbian and Kashubian. Three RoBERTa models were trained from scratch, two mono-lingual models for Czech and Polish, and one bi-lingual model for Czech and Polish. These models were evaluated on the NER downstream task for Czech, Polish, Upper Sorbian, and Kashubian, and compared to existing state-of-the-art models such as RobeCzech, HerBERT, and XLM-R. The results indicate that the mono-lingual models perform better on the language they were trained on, and both the mono-lingual and language family models outperform the large multi-lingual model in downstream tasks. Overall, the study shows that low-resource West Slavic languages can benefit from closely related languages and their models.

## introduction the success of recent large language models such as the gptx-family @xcite is due to a vast amount of training data and the availability of appropriate compute resources which allow to train these models. however, the availability of training data varies extremely between the languages of the world. high-resource languages such as english allow to train language models, performing impressively well on a variety of nlp tasks @xcite , whereas for the majority of the languages these large corpora are not available and thus, the same training concept does not necessarily yield well performing language models. this imbalance is addressed by multi-lingual models and transfer learning approaches. large multi-lingual language models such as xlm-r @xcite are trained on text data in 100 different languages and show good results on a variety of nlp downstream tasks in different languages, like e.g. named entity recognition (ner) in german. however, there are many low-resource languages in the world, which are still not covered by these commonly available language models due to the lack of a reasonable amount of training data. this problem is addressed by different transfer learning approaches. one approach considers language families and the transfer based on the similarities of the languages of the same family @xcite . in this case, the small amount of training data can partly be compensated by the similarities of the languages within the family. while training multi-lingual language models on languages from the same family, the training process profits from a larger amount of training data and from structural similarities of the languages at the same time. in contrast to the training of multi-lingual language models, @xcite consider the transfer from large language models for high-resource languages to large language models for lower-resource languages based on the overlapping vocabulary. in this approach, a large language model for a high-resource language (hrl) is used together with a small language model for a lowerresource language (lrl) in order to initiate the training of a large language model for lrl. this approach yields promising results and extensions to other language pairs which need to be investi-1 gated. however, as this approach is based on an overlapping vocabulary, language families are of special interest. in this paper, we present investigations on the west slavic language family. the aim of this work was to assess the possibilities for low-resource languages like upper sorbian @xcite and kashubian @xcite to profit from language models from the same language family. for this reason, we trained mono-lingual and multilingual language models from the same language familiy and evaluated them on the downstream task ner. since there are several publicly available mono-lingual language models for slavic languages @xcite , for comparison we evaluated some of them on the same downstream task. our contributions are as follows. we consider the languages czech (cs), polish (pl), upper sorbian (hsb), and kashubian (csb), all being members of the west slavic language family @xcite . we trained three roberta models @xcite from scratch, two monolingual models for czech and polish respectively and one bi-lingual model for czech and polish, based on the czech and polish subset of the os-car data set @xcite . for model evaluation, we used the downstream task, named entity recognition (ner), as described in @xcite and the corresponding wikiann dataset. we evaluated the three roberta models on czech and polish ner and on upper sorbian and kashubian ner. for comparison, we also considered existing sota mono-and multi-lingual models, namely the czech roberta model robeczech foot_0 @xcite , the polish bert model herbert foot_1 @xcite , and the multi-lingual roberta model xlm-r foot_2 @xcite , and evaluated them on czech and polish ner and on upper sorbian and kashubian ner. ## related work in de @xcite , the impact of language families on low-resource languages was investigated. the authors used mono-lingual bert models (source languages english, german, dutch) and the multi-lingual mbert to show, that linguistic structure can be transferred for the low-resource languages gronings and west frisian, which are closely related to the source languages. a different approach is taken in @xcite , where a transformer-based model is trained on 11 low-resource african languages belonging to a single language family. this expands the training data corpus by utilizing data within one language family. in contrast, we are interested in detecting those language combinations, which best support dedicated low-resource languages. there is a variety of czech and polish language models available, as shown in @xcite . in straka et al. (2021) robeczech, a czech roberta model is presented and evaluated on several downstream tasks, including ner using two datasets @xcite . a polish roberta model is described in @xcite and evaluated on ner, using the nkjp dataset @xcite . in @xcite herbert, a polish bert model is presented, trained on six different polish datasets and evaluated on the nkjp dataset. for several reasons we decided to train models from scratch as baseline models. first, we wanted to compare mono-lingual and multi-lingual language models, which are trained on a subset of the languages of a language family, based on the same training corpora. our particular focus was on the sorbian language, which is spoken in a region of germany adjacent to both poland and the czech republic. as in practice geographic distances between countries, syntactic similarity and syntactic overlap play an important role for transfer learning (de vries and nissim, 2021), we wanted to train a czech-polish model. however, for comparison, we considered existing czech and polish language models in addition. secondly, we were interested in performance analysis of distributed model training on our hpc infrastructure. these results are beyond the scope of this paper. evaluating language models on ner is very common. especially for balto-slavic languages there is a series of work, addressing the shared tasks of the balto-slavic nlp workshop series, e.g. @xcite . in piskorski et al. (2021) results of the last workshop are presented. as a starting point however, we restricted our investigations to ner for only three entites, namely person, organisation, and location. n_d low_lbp_d red_d meta_s pl 443 209 607 10,121 cs 127 98 339 6,689 ## training of baseline models this investigation considers publicly available pretrained language models such as robeczech, her-bert and xlm-roberta as well as models trained from scratch. in this section, the setup for training language models from scratch is described, which comprises training data, model architecture, tokenizer and the concrete training process. ## model evaluation results throughout the experiments, the main goal is to investigate whether the low-resource languages, upper sorbian and kashubian, can benefit from language models that are trained for closely related higher-resource languages (here: czech and polish). thereby, the experiment is twofold: first, the self-trained (from scratch) mono-lingual models and the multi-lingual language model are evaluated on the downstream task ner wrt the lowresource-languages of interest. second, there is a further comparison with publicly available pretrained mono-and multi-lingual language models such as e.g. robeczech or xlm-roberta. @xcite . ## conclusion and future work in our paper, we investigated the west slavic language family to evaluate the potential of language models for low-resource languages like upper sorbian and kashubian. we trained three roberta models from scratch, two mono-lingual models for both czech and polish respectively, and one multilingual model for czech and polish. these models were evaluated on the ner task for czech, polish, upper sorbian, and kashubian. we also compared the performance of our models with existing sota mono-and multi-lingual models, namely robeczech, herbert, and xlm-r. it can be seen that both mono-lingual models show better accuracy on the language they were trained on in comparison with the czech-polish model. the czech and czech-polish models show a better f1-score than robeczech and xlm-r in the czech downstream task. for both downstream tasks, the mono-lingual model for the respective language and the language family model (czech-polish) perform better than a large multi-lingual model. the adaptation of the language models for the upper sorbian and the kashubian language was investigated. the herbert model shows the best f1-score for ner in upper sorbian. our own czech model performs the best for accuracy in upper sorbian. our own polish-czech model shows the best f1-score for ner in kashubian, while the herbert model shows similar accuracy. overall, the contribution has shown, that lowresource west slavic languages such as upper sorbian or kashubian can profit from closely related languages and their belonging models. but the crucial point seems to be the fundamental understanding of relatedness between low-resource languages and potentially promising high-resource languages. this requires a close collaboration with linguists, to successfully infer, where to profit from common training data and/or models. there is still a lot of potential to investigate more languages within a family and compare them with larger highquality data sets (e.g. cnec @xcite , nkjp @xcite ) and evaluate the models on modified ner tasks as described in @xcite . furthermore, an interesting approach could be a cross-lingual and progressive transfer learning approach @xcite , where training of language models for low-resource languages starts with a large language model for a high-resource language and includes overlapping vocabulary. this method has yielded promising results for creating large models, but it refers to language families and not single languages. another development direction could be in building large corpora from existing parallel corpora. this would allow for the creation of high-quality training data for multi-lingual models and enable the training of models for low-resource languages that may not have sufficient training data available. dresden. the authors thank alexander fraser for helpful suggestions at the very beginning of this project, and the anonymous reviewers for supportive comments.