R o BERT – A R omanian BERT Model (2020)

Abstract: Deep pre-trained language models tend to become ubiquitous in the field of Natural Language Processing (NLP). These models learn contextualized representations by using a huge amount of unlabeled text data and obtain state of the art results on a multitude of NLP tasks, by enabling efficient transfer learning. For other languages besides English, there are limited options of such models, most of which are trained only on multi-lingual corpora. In this paper we introduce a Romanian-only pre-trained BERT model – RoBERT – and compare it with different multi-lingual models on seven Romanian specific NLP tasks grouped into three categories, namely: sentiment analysis, dialect and cross-dialect topic identification, and diacritics restoration. Our model surpasses the multi-lingual models, as well as a another mono-lingual implementation of BERT, on all tasks.

## introduction in the last decade, recurrent neural networks (rnns) based on lstm @xcite and gru @xcite cells represented the basis of state of the art methods for a wide range of natural language processing (nlp) tasks @xcite @xcite . in general, rnns make great use of pre-trained word embeddings such as word2vec @xcite or glove @xcite . word embeddings are usually computed using specialized neural networks trained in an unsupervised manner, and learn for each word a single vector representation. recently, a paradigm-shift in the nlp community occurred: word embeddings were replaced by large-scale pre-trained language models that compute contextual embeddings (i.e., output embeddings depend on the entire sequence). transformer @xcite has quickly become the building block of multiple state of the art architectures such as gpt @xcite @xcite , bert @xcite , roberta @xcite , or xlnet @xcite . @xcite propose the usage of multiple-head self-attention blocks, instead of the more classical recurrent approach, to model long-range sequence interactions. replacing the sequential recurrent neural network with self-attention modules allows for easy parallelization, thus ensuring faster training on large-scale architectures. large-scale transformers have the advantage of a single computationally expensive phase (pre-training), followed by an easy and fast fine-tuning phase, specific for each task. while transformer models have quickly become the standard approach for nlp tasks, the vast majority of studies have been performed on english. for other languages, the options are rather limited: either pre-train an entire model on the preferred language, or use a multi-lingual model trained on several languages. two multi-lingual models stand out: multi-lingual bert @xcite , which is a bert-base model trained on 104 languages, and xlm-roberta @xcite , which is trained on a massive 2.5tb corpus containing samples from 100 languages. in this paper, we set out to pre-train bert-based models for romanian and perform an extensive study on its performance on a multitude of downstream tasks. three variants of robert (i.e. small, base, and large) were pre-trained and publicly released at foot_0 . both multi-lingual and mono-lingual (romanian) bert models were analyzed and compared on three downstream tasks (one with five sub-tasks), showing that robert variants consistently outperform multi-lingual models and previous approaches on all considered tasks. ## related work a study by @xcite compared mono-lingual variants of bert on english and german, with multi-lingual bert. experiments were conducted on a simple syntactic classification task, a cloze test, and full text generation. on the simple syntactic classification task, rather small differences were encountered between mono-and multi-lingual models. as the tasks increased in difficulty, the gap between model performance increased to the point where multi-lingual bert was barely usable for language generation. the study concludes that a real need exists for mono-lingual bert models, instead of relying on multi-lingual ones. mono-lingual variants of bert-based models are available for a multitude of languages. flaubert @xcite and camembert @xcite are both roberta-based @xcite french models, that outperform the multi-lingual variant of bert on a variety of nlp tasks. dutch versions of @xcite are also available, as well as models for russian @xcite , german foot_1 , finnish @xcite , italian @xcite , portuguese @xcite , spanish @xcite , vietnamese (nguyen and nguyen, 2020), japanese foot_2 , and arabic @xcite . there are only two multi-lingual bert-based models available for the romanian language at the time of writing this paper, namely mbert @xcite and the more recent xlm-roberta (xlm-r) @xcite . we only found one repository foot_3 with a model trained specifically for romanian. unfortunately, we did not find a very great level of details regarding how their model was trained. a great overlap between our and their pre-training corpora exists, although the collections are not identical (i.e., both approaches are mostly based on oscar @xcite and romanian wikipedia; more details in section 3.1). one noteworthy difference is represented by the size of the vocabulary, 38k tokens (ours) and 50k tokens (theirs). in all following experiments, we refer to this model as bert-base-ro. to our knowledge, these models are the only transformer-based options for romanian. ## evaluation in the following section, we describe the methodology, tasks, and datasets used to evaluate and compare our models with other state-of-the-art methods applicable for the romanian language. unfortunately, we are not aware of any large collection of natural language understanding tasks for romanian language. therefore, the models were tested on a total of seven downstream tasks grouped into three categories: sentiment analysis, moldavian versus romanian dialect and cross-dialect topic identification (with five sub-tasks), and automated diacritics restoration. we believe the seven tasks represent a reliable benchmark for comparing natural language understanding models because they are well balanced, having data sources originating from various informal (i.e. online product reviews), semi-formal (i.e. talk show scripts), and formal (i.e. news) sources. furthermore, we present mlm and nsp accuracies and losses computed on the evaluation corpus (see ## conclusions three romanian language models based on bert architecture were pre-trained, evaluated, and publicly released. we compare our models with strong baselines represented by both multi-lingual models and another pre-trained romanian language model. our results indicate that mono-lingual models consistently outperform multi-lingual models, in spite of the fact that the former have fewer parameters, and the latter are pre-trained on significantly more data. since our objective was to compare our models with existing multi-lingual ones, we kept a very similar architecture and very little parameter tuning across all tasks. better results could probably be obtained for a specific task by experimenting with additional hyper-parameters. nevertheless, our models set the state of the art results on almost all considered tasks. the release of pre-trained bert models is an important step for nlp progress in languages with limited resources, such as romanian. overall, pre-trained models are easily incorporated, introduce important performance improvements, and help researchers tackle new problems, when task-specific datasets are scarce, because fine-tuning a model requires fewer examples in contrast to a model trained from scratch.