The birth of R omanian BERT (2020)

Abstract: Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus com-position and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We opensource not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.

## introduction a revolution started in natural language processing (nlp) a few years ago, when the first transformerbased model @xcite demonstrated a significant increase in state-of-the-art results compared to previous neural approaches. the bidirectional bert @xcite language model has since been widely adopted as the baseline for transformer models, and it has been successfully applied to a broad range of nlp tasks from standard language modeling to question answering, text summarization, and machine translation. a number of papers have been dedicated to studying why and how this model performs so well, including comparison to classical nlp @xcite , investigation of the newly introduced multi-head attention mechanism @xcite , and analyses of what bert learns @xcite . a good recent review is presented by @xcite . following bert, a number of variations of language models using corpus lines words size opus 55.1 m 635.0 m 3.8 gb oscar 33.6 m 1725.8 m 11 gb wikipedia 1.5 m 60.5m 0.4 gb total 90.2 m 2421.3 m 15.2 gb the transformer architecture have been introduced, including extended models such as xlm @xcite and xlnet @xcite as well as more efficient ones like distillbert @xcite , albert @xcite , and electra @xcite . however, the vast majority of these studies have focused only on english models. while google has released a multilingual bert model trained on 100+ languages, only recently have monolingual models for other languages started to appear: flaubert for french @xcite , bertje for dutch @xcite and finbert for finnish @xcite . but none for romanian, until now. while the multilingual bert can be used also for romanian, a monolingual model can bring an increase in accuracy that can be observed also in downstream task performance. thus, we here introduce romanian bert. this paper focuses on the technical and practical aspects of romanian bert, covering corpus composition and cleaning in section 2, the training process in section 3, and evaluation on romanian data sets in section 4. ## corpus the unannotated texts used to pre-train romanian bert are drawn from three publiclyavailable corpora: opus, oscar and wikipedia. opus opus @xcite is a collection of translated texts from the web. it is an open- ## pretraining the pretraining process starts with building a vocabulary on the available corpus. using byte-pairencoding (bpe), we generated cased and uncased vocabularies containing 50000 word pieces. character coverage 1 was set to 2000. pretraining was performed using the standard bert recipe. each model was trained for 1m steps, with the initial 900k trained on a sequence length of 128 and the rest with the maximum length of 512. figure 1 shows the progress for both models. the sudden increase in loss at the 900k steps mark is due to the switch to the 512 sequence length, but both models quickly recover from it. the models were trained using a batch size of 140 per gpu (for 128 sequence length), and then 20 (for 512 sequence length). the optimizer used was layerwise adaptive moments optimizer for batch training (lamb @xcite ), with warm-up over the first 1% of steps up to a learning rate of 1e-4, followed by a decay. eight nvidia volta v100 gpus with 32gb memory were used, and the pretraining process took around 2 weeks per model. ## evaluation we evaluate romanian bert on three downstream tasks: for each task we compare romanian bert with m-bert, on both cased and uncased versions. to mitigate the effect of the random seed we run each experiment 5 times and report only the mean score. all results listed here are reproducible by using the evaluation scripts provided on github. ## conclusions we have introduced the first bert model trained solely on a 15gb romanian corpus, obtained by a thorough clean of oscar, opus and wikipedia sub-corpora. we have shown that romanian bert outperforms the only available model that works on romanian, the multilingual m-bert. as the current corpus is better cleaned, more text added, tweaks to improve vocabulary coverage are performed, new versions of bert as well as future models will be released to the open source domain foot_2 .