Analysis of Transfer Learning for Named Entity Recognition in S outh- S lavic Languages (2023)

Abstract: This paper analyzes a Named Entity Recognition task for South-Slavic languages using the pre-trained multilingual neural network models. We investigate whether the performance of the models for a target language can be improved by using data from closely related languages. We have shown that the model performance is not influenced substantially when trained with other than a target language. While for Slovene, the monolingual setting generally performs better, for Croatian and Serbian the results are slightly better in selected cross-lingual settings, but the improvements are not large. The most significant performance improvement is shown for the Serbian language, which has the smallest corpora. Therefore, fine-tuning with other closely related languages may benefit only the “low resource” languages.

## introduction named entity recognition (ner) is one of the cornerstones of the nlp tasks and is widely used in many real-life applications, including in the news industry. in our study, we focus on south-slavic languages and investigate whether the performance of the models for a target language can be improved by using data from closely related languages. the research on ner has a long history. already in the 90s, the research was performed by @xcite , followed by @xcite ; @xcite , to mention a few of the early works. early literature focused on rule-based models @xcite , which were based on a set of pre-defined patterns, and hand-crafted rules (e.g., ltg, netowl). these approaches were followed by the unsupervised methods @xcite , where no annotated data were required. the advent of machine learning algorithms opened a novel direction for ner tasks where feature engineering gained more traction @xcite @xcite . with recent advances in neural networks, ner was formulated as a sequence-labelling task and took advantage of the neural systems, especially trans-formers, to minimize the effort of feature engineering @xcite . ensemble systems that combine different machine learning @xcite and neural representation @xcite or architectures @xcite were also under consideration. besides richresourced languages (e.g., english), there is a shift to several less-resourced ones, including the slavic family (see several organized shared tasks @xcite @xcite ). the availability of multilingual large language models and transfer learning strategies @xcite have simplified the cross-lingual transfer for a variety of nlp tasks. this opened new opportunities in the development of multilingual applications, especially in settings with limited resources. cross-lingual learning allows for overcoming the problems with the lack of data, including in zero-and few-shot learning, where no or very small number of data for the target language is available. moreover, getting the performance of a multilingual neural model as close as possible to the performance of a monolingual one can be very beneficial also in terms of simplicity and scalability, as a single model can be used instead of many monolingual ones. last but not least, even if data for the target language is available, adding data in other languages can lead to an improvement in results. multilingual models have been used in a large number of tasks, including cross-lingual hatespeech detection @xcite , zero-shot sentiment analysis @xcite as well as for ner @xcite . it was shown that the multilingual bert transformer model outperforms the bilstm-crf model for the ner task. the performance can be even further improved with a wordlevel crf layer @xcite . nevertheless, it is also evident that xlm-roberta outperforms bert @xcite in such tasks. the closest to our paper is the work by prelevikj and zitnik (2021), who showed that the monolingual ner model performance for the slovene language is practically equal to that of a multilingual one. in our paper, we focus on ner in slovene, croatian and serbian and aim to answer the following question: does fine-tuning with related languages influence the performance of a multilingual model compared to fine-tuning only in the target language? the rest of the paper is structured as follows. first, we present the corpora we used and how we preprocessed them, followed by their analysis. next, we continue with presenting the methodology, where we first introduce the measures, models, hyper-parameters, and software used. finally, we continue by evaluating the results and by presenting conclusions. ## data description in this section, we first present all the corpora used. then, we continue with the description of the conversion of these datasets to the expected format and conclude with the corpora structure analysis. we used the most common and established ner corpora for selected languages (see we used ner tags in iob2 @xcite format from the conll-2003 shared task @xcite as a common denominator for all corpora and experiments. each corpus was first combined if split, then converted to a common format, reshuffled, and split to train/validation/test set in an 80/10/10 ratio. we produced combined corpora by concatenating the sets without further reshuffling so that the experiments could be repeated. our study uses slovene, croatian, and serbian as target languages. however, in addition to those, also bosnian and macedonian are considered as the source languages, as they are closely related. corpora used are presented in ## methodology in the following section, we present the methodology used in our experiments to test our hypothesis that the ner classification f1-score increases when we fine-tune the pre-trained multilingual model with an additional, related language. ## evaluation in the following section, we define the f1-score we used for evaluation. then we present the experiment results: the evaluation of the pre-trained multilingual model, followed by the evaluation of fine-tuning for each language. for all classification measurements, the seqeval library @xcite was used. although the library uses conll evaluation by default, we chose "strict" mode evaluation. when calculating measurements, the strict mode also considers the iob2 tag's "beginning" and "inside" parts. therefore the ner tags must match exactly. ## conclusion we have shown that model performance can be influenced substantially by cross-lingual training with other language corpora, but that improvements only seem to occur if the target language has relatively small corpora. while for slovene, the monolingual setting generally performs better, for croatian and serbian, the results are slightly better in selected cross-lingual settings. the most significant performance improvement is shown for the serbian language, which has the smallest corpora. this indicates that fine-tuning with other closely related languages may benefit only the "low resource" languages. our initial hypothesis has not been fully upheld, but the result is still beneficial. first, when considering less-resourced settings, leveraging closely related languages is beneficial. second, the performance does not degrade much if we fine-tune the model with additional language corpora from the same family. this is an important finding, as using a multilingual model in an application is a simpler solution than having several monolingual models. in future work, we propose further investigating how performance changes when distantly related languages are used for fine-tuning the models. this will further benefit the usage in an industrial setting if the performance is not degraded, as having a single model that supports more languages with similar performance to monolingual training is more scalable and practical.