S lovak BERT : S lovak Masked Language Model (2022)

Abstract: We introduce a new Slovak masked language model called SlovakBERT . This is to our best knowledge the first paper discussing Slovak transformers-based language models. We evaluate our model on several NLP tasks and achieve state-of-the-art results. This evaluation is likewise the first attempt to establish a benchmark for Slovak language models. We publish the masked language model, as well as the fine-tuned models for part-of-speech tagging, sentiment analysis and semantic textual similarity.

## introduction fine-tuning pre-trained large-scale language models (lms) is the dominant paradigm of current nlp. lms proved to be a versatile technology that can help to improve performance for an array of nlp tasks, such as parsing, machine translation, text summarization, sentiment analysis, semantic similarity etc. the state-of-the-art performance makes lms attractive for any language community that wants to develop its nlp capabilities. in this paper, we concern ourselves with the slovak language and address the lack of language models, as well as the lack of established evaluation standards for this language. in this paper, we introduce a new slovak-only transformers-based language model called slovak-bert foot_0 . although several multilingual models already support slovak, we believe that developing slovak-only models is still important, as it can lead to better results and more compute and memorywise efficient processing of the slovak language. slovakbert has roberta architecture @xcite and it was trained with a web-crawled corpus. since no standard evaluation benchmark for slovak exists, we created our own set of tests mainly from pre-existing datasets. we believe that our evaluation methodology might serve as a standard benchmark for the slovak language in the future. we evaluate slovakbert with this benchmark and we also compare it to other available (mainly multilingual) lms and other existing approaches. the tasks we use for evaluation are: part-of-speech tagging, semantic textual similarity, sentiment analysis and document classification. we also publish the best-performing models for selected tasks. these might be used by other slovak researchers or nlp practitioners in the future as strong baselines. our main contributions in this paper are: the rest of this paper is structured as follows: in section 2 we discuss related work about language models and their language mutations. in section 3 we describe the corpus crawling efforts and how we train slovakbert with the resulting corpus. in section 4 we evaluate the model with four nlp tasks. ## training in this section, we describe our own slovak masked language model -slovakbert, the data that were used for training, the architecture of the model and how it was trained. ## evaluation in this section, we describe the evaluation methodology and results for slovakbert and other lms. we conducted the evaluation on four different tasks: part-of-speech tagging, semantic textual similarity, sentiment analysis and document classification. for each task, we introduce the dataset that is used, various baseline solutions, the lm-based approach we took and the final results for the task. for some tasks (part-of-speech tagging and semantic textual similarity) we also performed layer-wise model analysis. ## conclusions we have trained and published slovakbert -a new large-scale transformers-based slovak masked language model using 19.35gb of web-crawled slovak text. we proposed an evaluation benchmark with multiple tasks for slovak language and evaluated several models. we conclude, that slo-vakbert achieves state-of-the-art results on this benchmark, but multilingual language models are still competitive, especially larger but computationally less efficient models such as xlm-r-large. we also release fine-tuned models for the slovak community. the lack of evaluation benchmarks is still an issue for many mid-resource languages, i.e. languages that have a sizeable corpus of text available on the web, but do not have annotated natural language understanding datasets available. our work was limited by this as well, as we were forced to use datasets created by machine translation (in case of sts), noisy datasets (in case of sentiment analysis), or datasets with almost saturated performance (in case of document classification). creating new high-quality datasets for the evaluation of slovak is our future work.