H er BERT : Efficiently Pretrained Transformer-based Language Model for P olish (2021)

Abstract: BERT-based models are currently used for solving nearly all Natural Language Processing (NLP) tasks and most often achieve state-of-the-art results. Therefore, the NLP community conducts extensive research on understanding these models, but above all on designing effective and efficient training procedures. Several ablation studies investigating how to train BERT-like models have been carried out, but the vast majority of them concerned only the English language. A training procedure designed for English does not have to be universal and applicable to other especially typologically different languages. Therefore, this paper presents the first ablation study focused on Polish, which, unlike the isolating English language, is a fusional language. We design and thoroughly evaluate a pretraining procedure of transferring knowledge from multilingual to monolingual BERT-based models. In addition to multilingual model initialization, other factors that possibly influence pretraining are also explored, i.e. training objective, corpus size, BPE-Dropout, and pretraining length. Based on the proposed procedure, a Polish BERT-based language model – HerBERT – is trained. This model achieves state-of-the-art results on multiple downstream tasks.

## introduction recent advancements in self-supervised pretraining techniques drastically changed the way we design natural language processing (nlp) systems. even though, pretraining has been present in nlp for many years @xcite @xcite , only recently we observed a shift from task-specific to generalpurpose models. in particular, the bert model @xcite proved to be a dominant architecture and obtained state-of-the-art results for a variety of nlp tasks. while most of the research related to analyzing and improving bert-based models focuses on english, there is an increasing body of work aimed at training and evaluation of models for other languages, including polish. thus far, a handful of models specific for polish has been released, e.g. polbert foot_0 , first version of herbert @xcite , and polish roberta @xcite . aforementioned works lack ablation studies, making it difficult to attribute hyperparameters choices to models performance. in this work, we fill this gap by conducting an extensive set of experiments and developing an efficient bert training procedure. as a result, we were able to train and release a new bert-based model for polish language understanding. our model establishes a new state-of-the-art on the variety of downstream tasks including semantic relatedness, question answering, sentiment analysis and part-of-speech tagging. to summarize, our contributions are: the rest of the paper is organized as follows. in section 2, we provide an overview of related work. after that, section 3 introduces the bert-based language model and experimental setup used in this work. in section 4, we conduct a thorough ablation study to investigate the impact of several design choices on the performance of downstream tasks. next, in section 5 we apply drawn conclusions and describe the training of herbert model. in section 6, we evaluate herbert on a set of eleven tasks and compare its performance to other stateof-the-art models. finally, we conclude our work in section 7. ## related work the first significant ablation study of bert-based language pretraining was described by @xcite . authors demonstrated the ineffectiveness of next sentence prediction (nsp) objective, the importance of dynamic token masking, and gains from using both large batch size and large training dataset. further large-scale studies analyzed the relation between the model and the training dataset sizes @xcite , the amount of compute used for training @xcite and training strategies and objectives @xcite . other work focused on studying and improving bert training objectives. as mentioned before, the nsp objective was either removed @xcite or enhanced either by predicting the correct order of sentences (sentence order prediction (sop), @xcite or discriminating between previous, next and random sentence (sentence structural objective (sso), @xcite . similarly, the masked language modelling (mlm) objective was extended to either predict spans of tokens @xcite , re-order shuffled tokens (word structural objective (wso), @xcite or replaced altogether with a binary classification problem using mask generation @xcite . for tokenization, the byte pair encoding algorithm (bpe, @xcite is commonly used. the original bert model used wordpiece implementation @xcite , which was later replaced by sentencepiece @xcite . @xcite discovered that rare words lack semantic meaning. ## experimental setup in this section, we describe the experimental setup used in the ablation study. first, we introduce the corpora we used to train models. then, we give an overview of the language model architecture and training procedure. in particular, we describe the method of transferring knowledge from multilingual to monolingual bert-based models. finally, we present the evaluation tasks. ## ablation study in this section, we analyze the impact of several design choices on downstream task performance of polish bert-based models. in particular, we focus on initialization, corpus size, training objective, bpe-dropout, and the length of pretraining. ## herbert in this section, we apply conclusions drawn from the ablation study (see section 4) and describe the final pretraining procedure used to train herbert model. pretraining procedure herbert was trained on the large corpus. we used dropout-bpe in tokenizer with a probability of a drop equals to 10%. finally, herbert models were initialized with weights from xlm-roberta and were trained with the objective defined in equation 1 with sso weight equal to 0.1. ## conclusion in this work, we conducted a thorough ablation study regarding training bert-based models for polish language. we evaluated several design choices for pretraining bert outside of english language. contrary to @xcite , our experiments demonstrated that sso is not beneficial for the downstream task performance. it also turned out that bpe-dropout does not increase the quality of a pretrained language model. as a result of our studies we developed and evaluated an efficient pretraining procedure for transferring knowledge from multilingual to monolingual bert-based models. we used it to train and release herbert -a transformer-based language model for polish. it was trained on a diverse multi-source corpus. the conducted experiments confirmed its high performance on a set of eleven diverse linguistic tasks, as herbert turned out to be the best on eight of them. in particular, it is the best model for polish language understanding according to the klej benchmark. it is worth emphasizing that the quality of the obtained language model was even more impressive considering its short training time. due to multilingual initialization, herbert base outperformed polish roberta base even though it was trained with a smaller batch size (2560 vs 8000) for a fewer number of steps (50k vs 125k). the same behaviour is also visible for herbert large . additionally, we conducted a separate ablation study to confirm that the success of herbert is caused by the described initialization scheme. it showed that in fact, it was the most important factor to improved the quality of herbert. we believe that the proposed training procedure and detailed experiments will encourage nlp researchers to cost-effectively train language models for other languages.