R ob BERT : a D utch R o BERT a-based L anguage M odel (2020)

Abstract: Pre-trained language models have been dominating the field of natural language processing in recent years, and have led to significant performance gains for various complex natural language tasks. One of the most prominent pre-trained language models is BERT, which was released as an English as well as a multilingual version. Although multilingual BERT performs well on many tasks, recent studies show that BERT models trained on a single language significantly outperform the multilingual version. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train a Dutch version of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch language model called RobBERT. We measured its performance on various tasks as well as the importance of the fine-tuning dataset size. We also evaluated the importance of language-specific tokenizers and the model’s fairness. We found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets. These results indicate that it is a powerful pre-trained model for a large variety of Dutch language tasks. The pre-trained and fine-tuned models are publicly available to support further downstream Dutch NLP applications.

## introduction the advent of neural networks in natural language processing (nlp) has significantly improved state-of-the-art results within the field. initially, recurrent neural networks and long shortterm memory networks dominated the field. later, the transformer model caused a revolution in nlp by dropping the recurrent part and only keeping attention mechanisms @xcite . the transformer model led to other popular language models, e.g. gpt-2 @xcite . bert @xcite improved over previous models and recurrent networks by allowing the system to learn from input text in a bidirectional way, rather than only from left-to-right or the other way around. this model was later reimplemented, critically evaluated and improved in the roberta model @xcite . these large-scale attention-based models provide the advantage of being able to solve nlp tasks by having a common, expensive pre-training phase, followed by a smaller fine-tuning phase. the pre-training happens in an unsupervised way by providing large corpora of text in the desired language. the second phase only needs a relatively small annotated dataset for fine-tuning to outperform previous popular approaches in one of a large number of possible language tasks. while language models are usually trained on english data, some multilingual models also exist. these are usually trained on a large quantity of text in different languages. for example, multilingual-bert is trained on a collection of corpora in 104 different languages @xcite , and generalizes language components well across languages @xcite . however, models trained on data from one specific language usually improve the performance of multilingual models for this particular language @xcite . training a roberta model @xcite on a dutch dataset thus also potentially increases performances for many downstream dutch nlp tasks. in this paper, we introduce robbert 1 , a dutch roberta-based pre-trained language model, and critically evaluate its performance on various language tasks against other dutch languages models. we also propose several new tasks for testing the model's zeroshot ability, evaluate its performance on smaller datasets, and for measuring the importance of a language-specific tokenizer. finally, we provide an extensive fairness evaluation using recent techniques and a new translated dataset. ## related work transformer models have been successfully used for a wide range of language tasks. initially, transformers were introduced for use in machine translation, where they efficiently improved the stateof-the-art @xcite . this cornerstone was used in bert, a transformer model obtaining state-of-the-art results for eleven natural language processing tasks, such as question answering and natural language inference @xcite . bert is pre-trained with large corpora of text using two unsupervised tasks. the first task is called masked language modeling (mlm), making the model guess which word is masked in certain position in the text. the second task is next sentence prediction, in which the model has to predict if two sentences occur subsequent in the corpus, or randomly sampled from the corpus. these tasks allow the model to create internal representations about a language, which could thereafter be reused for different language tasks. this architecture has been shown to be a general language model that could be fine-tuned with little data in a relatively efficient way for a very distinct range of tasks and still outperform previous architectures @xcite . transformer models are also capable of generating contextualized word embeddings @xcite . traditional word embeddings, e.g. word2vec @xcite and glove @xcite , lack the capibility of differentiating words based on context (e.g. "a stick" versus "let's stick to"). transformer models, like bert, on the other hand automatically incorporate the context a word occurs into its embedding. the attention mechanism in transformer encoder models also allows for better resolution of coreferences between words @xcite . for example, in the sentence "the trophy doesn't fit in the suitcase because it's too big.", the word "it" would refer to the the suitcase instead of the trophy if the last word was changed to "small" @xcite . being able to resolve these coreferences is for example important for translation, as dependent words might change form, e.g. due to word gender. while bert has been shown to be a powerful language model, it also received scrutiny on its training and pre-processing. the authors of roberta @xcite showed that while the nsp pre-training task made the model perform better, it was not due to its intended reason, as it might merely predict relatedness between corpus sentences rather than subsequent sentences. that @xcite trained a better model when using nsp than without nsp is likely due to the model learning long-range dependencies that were longer than when just using single sentences. as such, the roberta model uses only the mlm task, and uses multiple full sentences in every input. other researchers later improved the nsp task by instead making the model predict for two subsequent sentences if they occur in the given or flipped order in the corpus @xcite . @xcite also presented a multilingual model (mbert) with the same architecture as bert, but trained on wikipedia corpora in 104 languages. unfortunately, the quality of these multilingual embeddings is considered worse than their monolingual counterparts, as @xcite illustrated for german and english models in a generative setting. the monolingual french camembert model @xcite also outperformed mbert on all tasks. @xcite also outperformed mbert on several dutch tasks using their dutch bert-based language model, called bert-nl, trained on the small sonar corpus @xcite . more recently, @xcite also showed similar results for dutch using their bertje model, outperforming multilingual bert in a wide range of tasks, such as sentiment analysis and part-of-speech tagging by pre-training on multiple corpora. since both these works are concurrent with ours, we compare our results with bertje and bert-nl in this paper. ## pre-training robbert we pre-trained robbert using the roberta training regime. we trained two different versions, one where only the pre-training corpus was replaced with a dutch corpus (robbert v1) and one where both the corpus and the tokenizer were replaced with dutch versions (robbert v2). these two versions allow to evaluate the importance of having a language-specific tokenizer. ## evaluation we evaluated robbert on multiple downstream dutch language tasks. for testing text classification, we evaluate on sentiment analysis and on demonstrative and relative pronoun prediction. the latter task helps evaluating the zero-shot prediction abilities, i.e. using only the pre-trained model without any fine-tuning. both classification tasks are also used to measure how well rob-bert performs on smaller datasets, by only using subsets of the data. for testing robbert's token tagging capabilities, we used both part-ofspeech (pos) tagging and named entity recognition (ner) tasks. ## robbert and fairness as language models are trained on large corpora, this poses a risk that minorities and protected groups are ill-represented, e.g. by encoding stereotypes @xcite @xcite . in word embeddings, these studies often rely on analogies @xcite or embedding analysis @xcite . these approaches are not directly transferable to bert models, since the sentence the word occurs in influences its embedding. efforts to generalize these approaches often rely on templates @xcite . these can be intentionally neutral ("<@xmath0."). one can then perform an association test between possible values for @xmath1, similar to a word embedding association test @xcite . in this section, we discuss two distinct potential origins of representational harm (blodgett et al., 2020) a language model could exhibit, and evaluate these on robbert v2. the two discussed behaviours are (i) stereotyping of gender roles in figure 2: ranking difference between gendered pronouns for various professions. three templates were used to evaluate, @xmath2. in the leftmost template, the pronoun and profession refer to different entities. ## code the training and evaluation code of this paper as well as the robbert model and the fine-tuned models are publicly available for download at ht tps://github.com/ipieter/robbert. ## limitations and future work there are several potential improvements for creating a better pre-trained robbert-like model. first, since bert-based models are still being actively researched, one could potentially improve the training regime using new unsupervised pretraining tasks when they are discovered, e.g. sentence order prediction @xcite . second, while robbert is trained on lines that contain multiple sentences, it does not put subsequent lines of the corpus after each other due to the shuffled nature of the oscar corpus @xcite . this is unlike roberta, which does put full sentences next to one another if they do not exceed the available sequence length, in order to learn the long-range dependencies between words that the original bert learned using its controversial nsp task. creating an unshuffled version of oscar might thus further improve the performance of the pre-trained model. third, there might be some benefit to modifying the tokenizer to use morpheme-based tokens, as dutch uses compound words. fourth, one could improve model's fairness during pre-training. we illustrated how representational harm in downstream tasks can affect the end user's experience, like the unequal predictive performance for the dbrd task. various methods have been proposed to mitigate unfair behaviour in ai models @xcite . while we refrained from training fair pre-trained and fine-tuned models in this research, training such models could be an interesting contribution. in addition, with the increased attention on fairness in machine learning, a broader view of the impact on other protected groups due to large pre-trained language models is also called-for. the robbert model itself can be used in new settings to help future research. first, robbert could be used in a model that uses a bert-like transformer stack for the encoder and a generative model as a decoder @xcite second, robbert can serve as the basis for a large number of dutch language tasks that we did not examine in this paper. given rob-bert's state-of-the-art performance on small as well as on large datasets, it could help advance results when fine-tuned on new datasets. ## conclusion we introduced a new language model for dutch based on roberta, called robbert, and showed that it outperforms earlier approaches as well as other bert-based language models for a several different dutch language tasks. more specifically, we found that robbert significantly outperformed other bert-like models when dealing with smaller datasets, making it a useful resource for a large range of application domains. we expect this model to serve as a base for fine-tuning on other tasks, and thus help foster new models that can advance results for dutch language tasks.