Too Many Cooks Spoil the Model: Are Bilingual Models for S lovene Better than a Large Multilingual Model? (2023)

Abstract: This paper investigates whether adding data of typologically closer languages improves the performance of transformer-based models for three different downstream tasks, namely Part-of-Speech tagging, Named Entity Recognition, and Sentiment Analysis, compared to a monolingual and plain multilingual language model. For the presented pilot study, we performed experiments for the use case of Slovene, a low(er)-resourced language belonging to the Slavic language family. The experiments were carried out in a controlled setting, where a monolingual model for Slovene was compared to combined language models containing Slovene, trained with the same amount of Slovene data. The experimental results show that adding typologically closer languages indeed improves the performance of the Slovene language model, and even succeeds in outperforming the large multilingual XLM-RoBERTa model for NER and PoS-tagging. We also reveal that, contrary to intuition, distantly or unrelated languages also combine admirably with Slovene, often out-performing XLM-R as well. All the bilingual models used in the experiments are publicly available at https://github.com/pranaydeeps/BLAIR

## introduction the last decade has witnessed the increasing popularity of large language models, such as bert @xcite and xlm-roberta @xcite . these transformer-based models have not only pushed the state-of-the-art for a wide range of nlp tasks, but have also shown to perform well in a multilingual setting. despite their success, these models are also confronted with a number of challenges. first, questions arise regarding their sustainability, given the exponential rise in parameters, and deployability in practical applications. second, although these models have been shown to achieve good performances for multilingual setups, research has shown that the performance of low(er)-resourced languages, when considering the amount of available wikipedia data, is below baseline @xcite . in this research, we want to investigate (1) whether a low(er)-resourced language benefits more from adding data from a typologically closer language, than from more distant languages, and (2) how the performance of such a small dedicated "close family" language model relates to the performance obtained with a purely monolingual model, trained with the same amount of data, on the one hand, and a plain multilingual xlm-roberta model trained with 105 different languages and a huge data set, on the other hand. for these pilot experiments, we opted to train various transformer-based language models for slovene, a low(er)-resourced slavic language. the motivation to perform these "family language model" experiments for a slavic language originates from the fact that the slavic languages show a high structural similarity, with a similar inflectional system, and also share a common core vocabulary. as a result, we hypothesize that adding other slavic languages will boost the performance of the slovene language model. for each model, we evaluate the performance on three different nlp tasks, namely part-of-speech tagging, named entity recognition and sentiment analysis. ## related research deep contextualised multilingual language models, such as mbert and xlm-r, have shown to perform well for many nlp tasks and for a variety of languages, including low(er)-resourced languages. nevertheless, previous research has revealed that more similar languages are more helpful for boosting the performance for low(er)-resourced languages. @xcite have investigated the degree to which the representations in multilingual bert @xcite generalise across languages, by fine-tuning the multilingual model on task-specific data from one language, and evaluating it on another language. although the authors show that mbert is able to perform cross-lingual generalization very well, the transfer works best for typologically similar languages, even suggesting that the model works best for languages with similar word orders @xcite . de vries et al. (2022) performed an extensive transfer learning evaluation with 65 different source languages and 105 target languages, and have shown that, amongst other factors, matching language families, writing systems, word order systems, and lexical-phonetic distance significantly impact the cross-lingual performance. multilingual models, such as mbert and xlm-roberta, use a wide variety of languages from different genera, including the slavic languages, as part of the same multilingual model. in contrast, other multilingual models have been trained on a smaller selection of languages, with a stronger focus on slavic languages. the researchers of the deeppavlov initiative, for example, developed a model for bulgarian, czech, polish and russian @xcite . while this model was initialised from the multilingual bert model and then fine-tuned on the task-specific data in the different languages, the crosloengual model was pre-trained from scratch for croatian, slovene and english and fine-tuned for taskspecific data for all languages @xcite . this model was built with the intention to apply it for multi-and cross-lingual training, making use of existing data sets for the same task in multiple languages. by doing so, the amount of task-specific data significantly increases, resulting in increased performance of the tasks of ner, pos-tagging and dependency parsing. although this shows that multilingual training causes an increase in performance, the main motivation for the multilingual aspect of the model is the data-hungry nature of the transformer architecture. this same motivation has also led to a transformer model that exclusively uses languages of the slavic genus. the bertić language model @xcite was trained from scratch for bosnian, croatian, montenegrin and serbian. whereas the crosloengual model uses more distant languages, bertić selected these languages because they are very closely related, are mutually intelligible and because they are considered part of the same serbo-croatian macro language (according to the iso 639-3 macrolanguage mappings). as such, bertić could be considered not a monolingual or multilingual but rather a macrolingual model. while these languages are exceptionally closely related, this setup does invite the following questions: "how important is the similarity of languages in a combined multilingual language model?" and "is it preferable to include more closely related languages over distant languages when building a multilingual model?". to compare language model performance for similar languages, researchers have often used the world atlas of language structures (wals) to group typologically similar languages @xcite . wals @xcite is a large database of structural (phonological, grammatical, lexical) properties of languages gathered from descriptive sources such as reference grammars. these linguistic features allow for comparison through qualitative features. this means that they can show in what ways languages are similar and in what ways they differ. however, beside counting the number of shared features, this does not allow for a quantitative comparison. one metric that does allow for a quantitative comparison, i.e. measure how similar the languages are, is ldnd (levenshtein distance normalized divided) @xcite . this metric was also used by @xcite in the context of cross-lingual training (training on data from other languages for the same task). their work has shown that "languages with low ldnd distances between source and target language (i.e. when two languages share cognates) are indeed associated with high accuracy, whereas high ldnd distances (very dissimilar languages) seem less informative". end, we performed experiments for the following roberta-based language models including slovene: (1) a slovene monolingual model, (2) a slovene combined with serbo-croatian model, (3) a slovene combined with slovak model, and (4) a slovene combined with czech model. we also performed experiments with two typologically distant languages, dutch and basque, for comparison. the motivation for combining specifically these languages with slovene originates from the ldnd measures but can also be linguistically supported. as shown in the serbian-croatian-bosnian cluster, as well as the czech-slovak cluster, are quite close to the slovene cluster. dutch and basque are distantly clustered, with basque being the farthest of all the visualised languages. ## evaluation and discussion we evaluated the various versions of the roberta language model on three different downstream tasks: one semantic task, being sentiment analysis, one syntactic task, being part-of-speech (pos) tagging, and one task requiring both syntactic and semantic understanding, namely named entity recognition (ner). for sentiment analysis, we use the sentinews dataset @xcite , which consists of news documents annotated with three sentiment labels (neutral, positive, and negative). we use the sentence-level sentiment setup with approximately 169,000 sentences, distributed into 80:10:10 for training, validation, and testing, respectively. for ner we, use the wikiann @xcite dataset with 15,000 train samples, and 10,000 samples each for validation and testing. finally, for pos tagging, the ssj treebank part of the universal dependencies foot_5 project is used, consisting of 13,000 annotated sentences, split into an 80:10:10 setup for training, validation and testing as well. for all downstream tasks, the respective roberta models were fine-tuned for 10 epochs, with a learning rate of 5e -5 with 500 warmup steps followed by a linear weight decay of 0.01. the results are summarised in firstly, the monolingual slovene model seems to perform comparably to xlm-roberta on all tasks, while only performing slightly worse than the upper-bound (ub) sloberta foot_6 model, which was trained on significantly (21 times) more data. this indicates that the presence of the additional 99 languages does not have a significant impact on slovene performance. the bilingual model with slovene+serbo-croatian seems to perform the best for ner, even outperforming the state-ofthe-art sloberta (ub), while the slovene+czech model seems to be the best for pos tagging, and only 0.06% worse than the ub, while the slovene-slovak model works best for sentiment analysis. for all three tasks, the best models come from the typologically closely related languages, however, the models with distant languages, slovene+dutch and slovene+basque, do not perform as badly as hypothesized. both models outperform the monolingual baseline, while sometimes also competing with the closely related languages in some settings. this is an interesting and rather counter-intuitive finding, since it suggests that the addition of data, irrespective of the language, is helpful for a given target language. even basque, with an ldnd distance of more than 100, is able to influence the slovene performance in a positive sense. this incites the following question: if all languages are indeed useful, irrespective of their differences, why is xlm-roberta the worst performing model then, with the highest amount of combined data? a logical inference would then be that after a certain amount of languages, the representation power of the roberta-base setup is not sufficient to model all 104 languages simultaneously, resulting in degradation for the poorly represented languages in the data, as would be the case for slovene. these observations might still indicate, however, that a multilingual model with 3 or more languages might show further improvements to our bilingual setup. in general, one can observe a downward trend as we move further away from slovene in terms of typological similarity or ldnd distance. this trend can be seen more clearly in figure 2 for pos and ner, while the trend is not as explicit for sentiment analysis, with a few anomalies. we dive further into the potential reasons for the inconsistencies with sentiment analysis performance in the next section. ## manual evaluation for sentiment analysis as the results for the sentiment analysis task do not align with our hypothesis and do not follow the tendencies we noticed for the other tasks, we decided to have a look at the predicted labels to find an explanation for these deviant results. a closer look at the evaluation data revealed a couple of reasons for the unexpected results. the evaluation data was selected from curated economic and political news corpora, characterised by a more neutral writing style. as a result, the sentiment is often implicit or ambiguous and requires world knowledge and human experience to be interpreted correctly. this is also confirmed by the modest inter-annotator agreement reported by @xcite , with f1-scores below 65% for their 3-way classification models. as shown in example 1, a rather neutral statement can also carry an implicit (negative) sentiment although it was annotated as neutral in the data set. ## conclusion this paper presents a pilot study to investigate whether adding data from typologically close languages improves the performance of a monolingual model for a low-resourced language, slovene in this case. to summarise the results, our experiments showed that adding data from a second language always helps, even if this language is more distant. in addition, the trained bilingual models outperform the very large multilingual model in almost all cases. finally, the bilingual slavic models outperform the bilingual models with more distant languages for the task of named entity recognition and pos tagging barring a few anomalies, whereas this is not confirmed for the task of sentiment analysis. as the results for sentiment analysis were somewhat counter-intuitive and not in line with the findings of the other tasks, we decided to also perform a small manual analysis where we outlined a number of issues with the complexity and subjectivity of the sentiment analysis task, including modest inter-annotator agreement and a number of ambiguous instances. in future research, we will perform validation experiments for additional combinations and downstream tasks, especially because the deviant scores for sentiment analysis might be partly due to the nature of the evaluation set used. additionally, it would also be worthwhile to check whether adding additional data for a second language (like croatian) would have a stronger positive impact on the evaluation of slovene compared to adding the same amount of data for a third language (czech). finally, we will also investigate simultaneously adding more than two languages to the training setup, to find the optimal inflection point for multilingual setups, after which some performance degradation is likely.