Comparing domain-specific and domain-general BERT variants for inferred real-world knowledge through rare grammatical features in S erbian (2023)

Abstract: Transfer learning is one of the prevailing approaches towards training language-specific BERT models. However, some languages have uncommon features that may prove to be challenging to more domain-general models but not domain-specific models. Comparing the performance of BERTić, a Bosnian-Croatian-Montenegrin-Serbian model, and Multilingual BERT on a Named-Entity Recognition (NER) task and Masked Language Modelling (MLM) task based around a rare phenomenon of indeclinable female foreign names in Serbian reveals how the different training approaches impacts their performance. Multilingual BERT is shown to perform better than BERTić in the NER task, but BERTić greatly exceeds in the MLM task. Thus, there are applications both for domain-general training and domain-specific training depending on the tasks at hand.

## introduction the recent introduction of transformer models @xcite has precipitated a dramatic shift in the landscape of natural language processing, bringing unprecedented gains in performance and accuracy. of particular note is bidirectional encoder representations for transformers (bert), a model which has become a baseline for nlp tasks @xcite . as with other deep neural network models, however, it is largely unknown how bert is able to achieve its performance. the bulk of nlp research focused on bert, a sub-field that has come to be known as bertology, has centred around probing underlying embeddings through various aptitude tests, comparing the performance of different bert variants to each other as well as to human performance metrics. these tasks may consist of more traditional tasks such as the cloze task, or more nlp-specific tasks such as named entity recognition or sentiment analysis. as english is the language of the original bert models, these efforts have usually focused on investigating linguistic phenomena that exist in english. this leaves us with a knowledge gap of bert's representation of linguistic typological features that are not shared with english but occur in other language families, such as the slavic languages. due to the success of transformer models, many domain-specific derivatives of bert have been produced. this includes topic-based domains such as scientific text model scibert @xcite , as well as domains of related languages such as the finnish-estonian model finestbert @xcite and bertić @xcite , a model for bosnian, croatian, montenegrin and serbian (bcms). the introduction of domain-specific derivatives has sparked a debate within bertology on how specific the dataset of the fine-tuning task should be. should derivatives be trained on a more general dataset within a domain or should they be fine-tuned on a much more domain-restricted dataset? we contribute to this debate by focusing on the case of closely related and under-resourced languages. we compare two variants of bert: the domain-general multilingual bert (mbert), and the language-specific model bertić, trained from scratch on the bcms languages. in particular, we explore how the two training approaches affect performance on rare grammatical phenomena in serbian. our case study is indeclinable nouns, a phenomenon typical of fusional languages where the same morphological form is used for all grammatical functions of a noun. this is a challenging phenomenon to model as it is typically infrequent and the usual morphological cues of the language aren't expressed. we create adapted versions of the common probing tasks of masked language modeling and named entity recognition specifically targeting this phenomenon. we contend that such phenomena that do not occur in english pose unique challenges to language modelling, particularly in under-resourced languages, and can reveal some of the overlooked underlying representations learned by bert derivatives. we aim to show areas in which transformerbased language model training can improve, as well as emphasize the importance of analysing the linguistic capabilities of non-english bert variants. ## background typically, bert makes use of mixed-domain transfer learning. the first stage of training uses generaldomain data, such as base bert's training on wikipedia and bookcorpus, followed by a finetuning domain-specific stage. domain-specific pretraining has been proposed to be more effective. @xcite compare the results of a more general scientific domain bert variant scibert with that of biomedical-specific biobert @xcite . scibert outperformed biobert in biomedical text tasks. @xcite contend that scibert's higher performance stems from its from-scratch training on scientific domain text. non-english language modelling provides distinct challenges compared to domain-specific training. human languages differ in ways that exceed that of domains of the same language. while related languages may share vocabulary and grammatical features, they often differ vastly in information structure and syntax. languages may also have varying amounts of quality data available. high resource languages such as english or german can be trained on monolingual text, while under-resourced languages may have no options but transfer learning. transfer learning is the predominant approach to building language-specific variants of bert. on top of base bert, multilingual bert (mbert) is additionally trained on the text of 104 languagespecific wikipedias without any cross-lingual alignment. mbert achieved impressive cross-lingual performance and itself is used as a base for countless language-specific bert derivatives, taking a mixed-domain training approach @xcite . however, several studies have shown that language-specific bert models trained on a dataset consisting of only one language still perform better than mbert-based models, especially in the case of under-resourced languages @xcite . @xcite show that a bangla-specific variant, banglabert, outperforms both mbert and a bangla-english bilingual variant. @xcite similarly show that an estonian-specific bert outperforms multilingual variants in five out of seven tasks. likewise, @xcite find that a bert variant trained ground-up on a swahili dataset outperforms multilingual models. bertić, a variant trained on bosnian, croatian, montenegrin and serbian, also outperforms both mbert and a trilingual croatian, slovene and english bert in nearly every task @xcite . ## methodology we compare bertić and mbert on two tasks: a feminine named entity recognition (ner) task, targeting the name domain in which the indeclinable noun phenomenon occurs, and masked langauge modelling (mlm), a more intrinsic evaluation task. bertić is pretrained with the elec-tra training objective, where instead of masking tokens, tokens are corrupted and a detection task is performed @xcite . mbert uses the standard bert mlm training objective. other multilingual bert-based models are available such as xlm-r @xcite and mt5 @xcite , but these all use the standard mlm objective rather than electra. out of these, we chose to compare to mbert as this comparison was also made by @xcite . both models use wordpiece subword tokenization @xcite . for the feminine ner task, ner-fine-tuned variants of both bertić and mbert are used. the bertić variant we use is the bcms-bertic-ner variant, which has been fine-tuned on the croatian hr500k dataset, serbian setimes.sr dataset, and the reldi-hr and reldi-sr internet (twitter) datasets in croatian and serbian respectively. in total, the dataset consists of 768k tokens. since a ner variant is not readily available for mbert at the time of testing, we use bert-base-multilingual-cased-ner-hrl instead. this model is fine-tuned on arabic, german, english, spanish, french, italian, latvian, dutch, portuguese and chinese ner. the training process is not well documented, but appears to consist mainly of newspapers from the early-to mid-2000s. this datedness ensures that mbert does not have an extra advantage from being exposed to a wider selection of modern names. spacy foot_0 is used as a baseline for comparison. for the mlm task we only use the base bertić and mbert models. all tasks are performed using a dell optiplex 7010 with an intel i7 processor and 12gb of ram. ## conclusion we evaluated the performance of two bert variants, multilingual bert (mbert) and bertić, on serbian indeclinable nouns, using a ner task and a mlm task. while in a general ner task, bertić and mbert show similar performance on serbian @xcite , mbert outperforms bertić in our feminine ner task. in the mlm task, bertić vastly outperforms mbert and both models performed significantly worse on indeclinable names. bertić produces a larger diversity of pragmatically correct responses overall. these results indicate that bertić may encode information about gender and names, but whether the encoding can be considered a morphological feature of nouns or is specific to a semantic domain of names remains unclear. we only see that bertić's performance is sensitive to name frequency. mbert on the other hand produces feminine forms significantly less often, and produces responses from related languages such as slovene and czech. the results from the ner task suggest that multilingual models perform better when the names are not native to the text language. on the other hand, language-specific tasks such as sentence completion will produce significantly more relevant results from models trained specifically for the language, as the embeddings contain a significantly larger amount of vocabulary for the target language. potential future directions include research on other typologically rare grammatical features, the behaviour of bert with other kinds of fusional languages and probing how contextual real-world knowledge inferred from them may be encoded. the representation of bi-alphabetical languages in language modelling could be explored further, as well as the ways language-specific training compares to more general training when dealing with closely related variants. more broadly, we claim that research on closely related languages contributes to our knowledge of the conditions and factors that affect the choice between using a transfer learning or in-domain learning approach.