E st BERT : A Pretrained Language-Specific BERT for E stonian (2021)

Abstract: This paper presents EstBERT, a large pretrained transformer-based language-specific BERT model for Estonian. Recent work has evaluated multilingual BERT models on Estonian tasks and found them to outperform the baselines. Still, based on existing studies on other languages, a language-specific BERT model is expected to improve over the multilingual ones. We first describe the EstBERT pretraining process and then present the models’ results based on the finetuned EstBERT for multiple NLP tasks, including POS and morphological tagging, dependency parsing, named entity recognition and text classification. The evaluation results show that the models based on EstBERT outperform multilingual BERT models on five tasks out of seven, providing further evidence towards a view that training language-specific BERT models are still useful, even when multilingual models are available.

## introduction pretrained language models, such as bert @xcite or elmo @xcite , have become the essential building block for many nlp systems. these models are trained on large amounts of unannotated textual data, enabling them to capture the general regularities in the language and thus can be used as a basis for training the subsequent models for more specific nlp tasks. bootstrapping nlp systems with pretraining is particularly relevant and holds the greatest promise for improvements in the setting of limited resources, either when working with tasks of limited annotated training data or less-resourced languages like estonian. since the first publication and release of the large pretrained language models on english, considerable effort has been made to develop support for other languages. in this regard, multilingual bert models, simultaneously trained on the text of many different languages, have been published, several of which also in1 the model is available via huggingface transformers library: https://huggingface.co/tartunlp/estbert clude the estonian language @xcite @xcite . these multilingual models' performance was recently evaluated on several estonian nlp tasks, including pos and morphological tagging, named entity recognition, and text classification @xcite . the overall conclusions drawn from these experiments are in line with previously reported results on other languages, i.e., for many or even most tasks, multilingual bert models help improve performance over baselines that do not use language model pretraining. besides multilingual models, language-specific bert models have been trained for an increasing number of languages, including for instance camem-bert @xcite and flaubert @xcite for french, finbert for finnish @xcite , robbert @xcite and bertje @xcite for dutch, chinese bert @xcite , beto for spanish @xcite , rubert for russian @xcite and others. for a recent overview about these efforts refer to @xcite . aggregating the results over different language-specific models and comparing them to those obtained with multilingual models shows that depending on the task, the average improvement of the language-specific bert over the multilingual bert varies from 0.70 accuracy points in paraphrase identification up to 6.37 in sentiment classification @xcite . the overall conclusion one can draw from these results is that while existing multilingual bert models can bring along improvements over language-specific baselines, using language-specific bert models can further considerably improve the performance of various nlp tasks. following the line of reasoning presented above, we set forth to train estbert, a language-specific bert model for estonian. in the following sections, we first give details about the data used for bert pretraining and then describe the pretraining process. finally, we will provide evaluation results on the same tasks as presented by @xcite on multilingual bert models, which include pos and morphological tagging, named entity recognition and text classification. additionally, we also train a dependency parser based on the spacy system. compared to multilingual models, the estbert model achieves better results on five tasks out of seven, providing further evidence for the usefulness of pretraining language-specific bert models. additionally, we also compare with the estonian wikibert, a recently published estonian-specific bert model trained on a relatively small wikipedia data @xcite . compared to the estonian wikibert model, the estbert achieves better results on six tasks out of seven, demonstrating the positive effect of the amount of pretraining data on the generalisability of the model. ## data preparation the first step for training the estbert model involves preparing a suitable unlabeled text corpus. this section describes both the steps we took to clean and filter the data and the process of generating the vocabulary and the pretraining examples. ## evaluation tasks before describing the estbert model pretraining process itself, we first describe the tasks used to both validate and evaluate our model. these tasks include the pos and morphological tagging, named entity recognition, and text classification. in the following subsection, we describe the available estonian datasets for these tasks. ## pretraining estbert the estbert model was pretrained on the architecture identical to the bert base model with 12 transformer blocks with 768 hidden units each and 110m trainable parameters. it was pretrained on the masked language modeling (mlm) and the next sentence prediction (nsp) tasks as described by @xcite . in mlm, the probability of correctly predicting the randomly masked tokens is maximised. because in the ## results the next subsections present the results obtained with the final estbert models with both sequence lengths on the tasks described in section 3. we follow the same setup of @xcite to enable direct comparison with the multilingual models. some additional steps were taken to prepare the estonian valence corpus. first, all duplicate items, 17 in total, were removed. also, all items with the ambiguous label were removed as retaining them has been shown to lower the the classification accuracy @xcite . the same preprocessing was also applied in evaluating the multilingual bert models for estonian @xcite . for finetuning, we used the same scripts from the huggingface transformers repository that were used for the pretraining validation in section 4.2. the same scripts were also used to evaluate the multilingual models by @xcite . for each task, the learning rate of the adamw optimiser and the batch size was tuned on the development set. the learning rate grid values were (5e-5, 3e-5, 1e-5, 5e-6, 3e-6) and the batch size grid values were (8, 16). the best model was found on the development set by using early stopping with the patience of 10 epochs. we compare the results of estbert with the multilingual bert models' results from @xcite and the wikibert model trained on the estonian wikipedia @xcite . wikibert-et model was finetuned using the same setup described above. ## discussion this objective of this paper was to describe the process of pretraining the language-specific bert model for estonian and to compare its performance with the multilingual bert models as well as with the smaller estonian wikibert model on several nlp tasks. overall, the pretrained estbert was better than the best multilingual xlm-roberta model on five tasks out of seven: upos, xpos, and morphological tagging, rubric classification, and ner. only in the sentiment classification and dependency parsing tasks, the xlm-roberta was better. compared to wikibert-et, the estbert model was better on six tasks out of seventhe wikibert-et model was superior only in the ner task, predicting org entities considerably better than any other model. we did not observe any consistent difference between the models of different sequence lengths, although the model with the sequence length 512 was trained longer. it is possible that the shorter model was already trained long enough, and the subsequent training of the longer model did not add any effect in that respect, aside from the fact that it can accept longer input sequences. one crucial aspect of this work was obtaining a large-enough corpus for pretraining the model. we used the estonian national corpus 2017, which was the largest corpus available at the time. a newer and larger version of this corpus-the estonian national corpus 2019 @xcite -has become available meanwhile. there are also few other resources, such as the estonian part of the conll 2017 raw data @xcite and the oscar crawl, which probably partially overlap with each other and with the estonian national corpus. still, these corpora would potentially provide additional data that was currently not used. another challenge was related to finding annotated datasets for downstream tasks. while the estonian ud dataset provides annotations to the common dependency parsing pipeline tasks, datasets for other, especially semantic nlp tasks, are scarce. we adopted the estonian valence corpus for two-way text classification. however, the labels of this corpus are semiautomatically derived from user ratings, and thus the quality of these annotations cannot be guaranteed. an estonian coreference dataset with some simple baseline results in nominal coreference resolution has recently become available @xcite , which gives further opportunities to test out the estbert model in future work. when preprocessing the data and pretraining the model, we mostly followed the process of training the finbert model for finnish @xcite . we also decided to truecase our corpus to reduce the number of capitalised words in the vocabulary. the est-bert model itself was also pretrained on the truecased corpus. however, when training the task-based models for evaluation, the estbert was finetuned on the cased datasets. thus, truecasing the datasets before finetuning might have a positive effect on the results. in order to verify this, the estbert-based task-specific models should be finetuned using the truecased annotated datasets as input, and compared with the results reported in this paper. although we did see some improvements with es-tbert compared to xlm-roberta on the smaller model for the ner task, the differences in scores were generally relatively small. however, we have observed that the annotations of this ner dataset are occasionally erroneous, containing, for instance, label sequences (i-per, i-per) instead of (b-per, i-per). we have also observed unlabelled entities in the text. thus, the small variations in the systems' results might not be informative about the systems themselves but can instead stem from the noise in the data. although these annotation errors have been noticeable enough, the magnitude of these errors has not been quantified. the differences between the estbert and the xlm-roberta model were, in most cases, relatively small. in previous experiments with several multilingual bert models on the same estonian tasks @xcite , the xlm-roberta proved to be the best multilingual model. this suggests that one option to obtain an even better estonian language-specific model would be to train an estonian-specific roberta by initializing the model with the parameters of the xlm-roberta. considering that the multilingual roberta already performs very well on estonian tasks, finetuning it with more estonian data would hopefully bias it even more to the estonian language while also maintaining the gains obtained from multilingualism. ## conclusion we presented estbert, the largest bert model pretrained specifically on the estonian language. while several existing multilingual bert models include estonian, the only language-specific estonian bert model available until now has been trained on the relatively small wikipedia data. in order to pretrain the estbert model, we used the largest estonian text corpus available at the time. the estbert model was put to the test by finetuning it for several tasks, including pos and morphological annotations, dependency parsing, named entity recognition, and text classification. on five tasks out of seven, the classifiers based on estbert achieved better performance than the models based on multilingual bert models, although in several cases, the gap with the best-performing multilingual xlm-roberta was relatively small. these results suggest that training a roberta model for estonian, initialised with the multilingual model's parameters, might be beneficial. on six tasks out of seven, the models based on estbert achieved better results than the estonian bert model trained on wikipedia, suggesting that using more textual data for pretraining leads to a more generalisable model.