Czert – C zech BERT -like Model for Language Representation (2021)

Abstract: This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.

## introduction transfer learning and pre-trained word embeddings became a crucial component for most natural language processing (nlp) models. contextualized methods @xcite @xcite overcame the initial context insensitive word embeddings approaches @xcite @xcite . @xcite . the word representations generated by the named methods are usually used as input features for other task-specific models that are further trained. starting with the bert @xcite , the bert-like models @xcite @xcite based on transformer architecture @xcite , achieved a significant performance improvement in many nlp tasks @xcite . these recent models are trained on a language model task or tasks that are closely related to it. such pre-training allows them to capture the general representation of language and text. the pre-trained models are then directly fine-tuned with specific data for a selected downstream task. the performance improvement of these models is paid by the vastly increased requirements (i.e., data and computational resources) for their training. the mentioned models are primarily trained for english. recently, models for other, mostly larger, languages have been released, e.g., french @xcite , polish @xcite , turkish @xcite , russian foot_0 , italian foot_1 , german 2 , arabic @xcite , but also for languages that are spoken by a relatively small number of people, i.e., romanian @xcite , @xcite or finish @xcite . there were also introduced multilingual models @xcite , that can be used for multiple languages at once but usually at the cost of lower performance in comparison to solely monolingual models @xcite @xcite as we show in this paper. our main motivation is to train and provide publicly available models foot_2 for the czech language that performs better than available multilingual models. in this paper, we describe a process of training of two bert-like models for czech language and their evaluation on six tasks along with a comparison to two multilingual models, i.e. mbert @xcite and slavicbert @xcite . more concretely, the architectures of our models are based on the albert @xcite ) model (czert-a) and the original bert @xcite . both of our models are trained on a text corpus of the approximate size of 36 gb of plain text consisting of czech wikipedia articles, crawled czech news and czech national corpus @xcite . we train the models from scratch (i.e., with random initialization) using masked language model (mlm) and next sentence prediction (nsp) tasks as training objectives with a slight modification of the nsp task, see section 3. we evaluate our models on six tasks foot_3 : semantic text similarity (sts), named entity recognition (ner), morphological tagging (mot), semantic role labeling (srl), sentiment classification (sc) and multi-label document classification (mlc). our main contributions are the following ones: 1) we release a pre-trained and ready to use bert model (czert-b) for the czech language that outperforms the compared models on all evaluated sentence-level tasks and it performs comparably on semantic role labeling task. along with the pre-trained model, we also release the fine-tuned models for each task. 2) we achieve new stateof-the-art results on seven datasets. moreover we outperform the multilingual models with our newly trained czert-b model on 7 out of 10 datasets. ## evaluation the following section summarizes the performance of czert on various tasks and compares our model with similar available models. we also add experiments without the pre-training phase to highlight the impact of additional unsupervised data in the czech language. we also compare czert with the following baselines: ## discussion we summarize the overall results of all evaluated tasks in we also outperformed other bert-like models with czert-b in mot, and surpass the sota. we accomplish outstanding performance and increase the sota in two other tasks: sentiment classification (sc) and semantic text similarity (sts). the increase is of ∼5% and ∼3% in both sentiment datasets and of ∼5% in one of the semantic similarity datasets. we also overcame sota in mlc. ## conclusion in this work, we present two monolingual bertlike models (bert and albert) for the czech language. we train the models with the original mlm task and with a slightly modified nsp task. we thoroughly evaluate our models on six common tasks, and we compare them with other multilingual models. we include task-specific state-of-the-art models in our comparison. we outperform multilingual models with our newly trained czert-b model on 9 out of 11 datasets. in addition, we establish the new state-of-the-art results on 9 datasets 12 . the results show the strong performance of the czert-b 12 the results in our models are publicly available for research purposes at our website and in the hugging face repository 13 .