BERT ić - The Transformer Language Model for B osnian, C roatian, M ontenegrin and S erbian (2021)

Abstract: In this paper we describe a transformer model pre-trained on 8 billion tokens of crawled text from the Croatian, Bosnian, Serbian and Montenegrin web domains. We evaluate the transformer model on the tasks of part-of-speech tagging, named-entity-recognition, geo-location prediction and commonsense causal reasoning, showing improvements on all tasks over state-of-the-art models. For commonsense reasoning evaluation we introduce COPA-HR - a translation of the Choice of Plausible Alternatives (COPA) dataset into Croatian. The BERTić model is made available for free usage and further task-specific fine-tuning through HuggingFace.

## introduction in recent years, pre-trained transformer models have taken the nlp world by storm @xcite @xcite , yielding new state-of-the-art results in various tasks and settings. while such models, requiring significant computing power and data quantity, started to emerge for non-english languages @xcite , as well as in multilingual flavours @xcite , there is a significant number of languages for which better models can be obtained with the available pre-training techniques. this paper describes such an effort -training a transformer language model on more than 8 billion tokens of text written in the bosnian, croatian, montenegrin or serbian language, all these languages being very closely related, mutually intelligible, and classified under the same hbs (serbo-croatian) macro-language by the iso-693-3 standard. 1 the name of the model -bertić -points at two facts: (1) the language model was trained in zagreb, croatia, in whose vernacular diminutives ending in ić are frequently used (fotić eng. photo camera, smajlić eng. smiley, hengić eng. hanging together), and (2) in all the countries / languages of this model the patronymic surnames end to a great part with the suffix ić, with likely diminutive etymology. the paper is structured as follows: in the following section we describe the data the model is based on, in the third section we give a short description of the modelling performed, and in the fourth section we present a detailed evaluation of the model. ## data as our data basis we use already existing datasets, namely (1) the hrwac corpus of the croatian top-level domain, crawled in 2011 @xcite and again in 2014 @xcite , (2) the srwac corpus of the serbian top-level domain, crawled in 2014 @xcite , (3) the bswac corpus of the bosnian top-level domain, crawled in 2014 @xcite , (4) the cn-rwac corpus of the montenegrin top-level domain, crawled in 2019, and (5) the riznica corpus consisting of croatian literary works and newspapers @xcite . given that most of the crawls contain data only up to year 2014, we performed new crawls of the bosnian, croatian and serbian top-level domains. we brand these corpora as classla web corpora given that classla is the clarin knowledge centre for south slavic languages foot_1 under which we perform most of the described activities. we deduplicate the classla corpora by removing the resulting sizes of the datasets used for training the bertić model are presented in ## model training for training this model we selected the electra approach to training transformer models @xcite . these models are based on training a smaller generator model and the main, larger, discriminator model whose task is to discriminate whether a specific word is the original word from the text, or a word generated by the generator model. the authors claim that the electra approach is computationally more efficient than the bert models @xcite based on masked language modelling. as in bert and similar transformers models, we constructed a wordpiece vocabulary with a vocabulary size of 32 thousand tokens. a word-piece model was trained using the huggingface tokenizers library foot_2 on the random sample of 10 ## evaluation in this section we present an exhaustive evaluation of the newly trained bertić model on two token classification tasks -morphosyntactic tagging and named entity recognition, and two sequence classification tasks -geolocation prediction and commonsense causative reasoning. the reference points in each task are the state-of-the art transformer models covering the macro-language -multilingual bert @xcite and crosloengual bert @xcite . while multilingual bert (mbert onwards) was trained on wikipedia corpora, crosloengual bert (csebert onwards) was trained on a similar amount of croatian data used to train bertić, but without the data from the remaining languages. ## conclusion in this paper we have presented a newly published electra transformer language model, bertić, trained on more than 8 billion tokens of previously and newly collected web text written in bosnian, croatian, montenegrin or serbian. we have applied a very thorough evaluation of the model, comparing it primarily to other state-of-the-art transformer models that support the languages in question. we have obtained significant improvements on all four tasks, with no difference obtained only on one single-source-dataset with little text variation and high training and testing data similarity. the main conclusions we can draw from our results are the following. (1) although csebert and bertić use a different approach to building transformer language models, our assumption is that the performance difference between these two lies primarily in the larger amount of data presented to the bertić model. (2) the improvements on the four tasks with the bertić model seem to be smaller on the morphosyntactic tagging task than the remaining three tasks that require more world and commonsense reasoning knowledge. (3) except for the named entity recognition task on the serbian non-standard dataset, we fail to observe greater improvements on serbian tasks than on croatian ones between csebert and bertić, regardless the fact that the former has seen none and the latter has seen huge quantities of serbian text, showing the irrelevance of minor language differences for performance of large transformer models. (4) while bilstm models are still closeto-competitive on the morphosyntactic tagging task, they cannot hold up on the named entity recognition task as it requires more common knowledge. such knowledge transformer models manage to absorb to a much higher level than pre-trained static embeddings used by bilstms. the bertić model is available from the hug-gingface repository at https://huggingface. co/classla/bcms-bertic .