G erman’s Next Language Model (2020)

Abstract: In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. All trained models will be made publicly available to the research community.

## introduction deep transformer based language models have shown state-of-the-art results for various natural language processing tasks like text classification, ner and question answering @xcite . they are pretrained, first by feeding in large unlabeled text corpora before being fine-tuned on the downstream task. in this work we present a set of german bert and electra models, the best of which, gelectra large , significantly improves upon state of the art performance on the germeval18 hate speech detection task by about +4% / +2.5% for the coarse and fine variants of the task respectively. this model also reaches sota on the germeval14 ner task, outperforming the previous best by over +4%. while performant, such models are prohibitively large for many and so we also present a new gbert model which matches deepset bert, the previous best german bert, in size but outperforms it by +2.23% f1 averaged over three tasks. in the process of pretraining the language models, we also a) quantify the effect of increasing the training data by an order of magnitude and b) verify that whole word masking has a positive effect on bert models. because of the computational expense of training large language models from scratch, we adopt a downstream-oriented evaluation approach to ensure that we get the best performance from a limited number of runs. this involves regularly checkpointing the model over the course of pretraining, evaluating these on a set of classification and ner tasks and selecting as final the checkpoint which shows the best performance. this stands in contrast to approaches where the final model is simply saved after a fixed number of steps. our method is also an important tool in diagnosing pretraining and we hope that it will be of use to other teams looking to train effective language models on a budget. ## related work modern language model architectures are trained to build word representations that take into consideration the context around a given word. first versions such as elmo @xcite , ulmfit (howard and ruder, 2018) and flair @xcite are lstm based and these were able to set new performance benchmarks on downstream tasks like text classification, pos tagging and ner. more recent approaches use transformer-based @xcite architectures and examples include gpt-2 @xcite , bert @xcite , roberta @xcite , albert @xcite and electra @xcite . in this work we focus on bert and electra models. bert uses a masked language modeling (mlm) strategy to corrupt an input sentence by replacing some tokens with a [mask] symbol. the model is then trained to re-construct the original token. however, this method of training is somewhat restricted in that the model only learns from the masked out tokens which typically make up about 15% of the input tokens. electra addresses this problem by introducing a new pretraining task called replaced token detection. instead of masking out tokens, a subset of the input tokens are substituted by a synthetically generated token. the model is then trained to classify whether each input token is original or substituted, thus allowing for gradient updates at every input position. practically speaking, this is achieved by having a discriminator that performs the replaced token detection and a generator which provides plausible token substitutes. these two components are trained jointly and are both transformer based. the bert model received an update when the original authors added whole word masking foot_0 whereby masking one subword token requires that all other tokens in the word are also masked out. the authors report that this method improves the training signal by removing the easiest cases and show that it improves performance in their tasks. there is also a line of work that looks into bringing language modeling techniques that were first developed on english to other languages. these include but are not limited to monolingual models such as camembert @xcite and flaubert @xcite for french, finnish bert @xcite and german berts by dbmdz foot_1 and deepset foot_2 . for a more comprehensive list, see @xcite . some models are also capable of supporting multiple languages such as multilingual bert (mbert base ) and xlm-roberta @xcite . multilingual bert is a multilingual model for 104 different languages foot_3 trained on wikipedia dumps. the xlm-roberta model is trained on 2.5tb of data from a cleaned common crawl corpus @xcite for 100 different languages. it is worth emphasizing here that systems trained on naturally occurring data will learn pre-existing cultural biases around gender @xcite , race and religion @xcite . critical evaluation of machine learning methods is more important than ever as nlp is gaining broader adoption. researchers have been advocating for better documentation of decisions made during the construction of a dataset @xcite , explicit statements of a dataset's "ingredients" @xcite and recognition of the dataset characteristics that may lead to exclusion, overgeneralisation and underexposure @xcite . these topics will be addressed in section 3.1. ## evaluation in our approach, models are evaluated continuously during pretraining. model checkpoints are saved at regular intervals and converted into pytorch models using hugging face's transformers library @xcite . using the farm framework foot_9 , we evaluate the performance of each checkpoint on ger-meval18 (coarse) and germeval18 (fine) which are both hate speech classification tasks @xcite . using hugging face's transformers we also evaluate on germeval14 @xcite which is a ner task. in bert, the vector corresponding to the [cls] token serves as a representation of the whole input sequence, while in electra, all word vectors are combined through a feed forward layer. in both cases, this input sequence representation is passed through a single layer neural network in order to perform prediction. in the ner task, each vector corresponding to the first token in a word is passed through a single layer neural network and the resulting prediction is applied to the whole word. each checkpoint is evaluated 3 times on each document classification task since we observed significant variance across different runs. each of these runs is performed with early stopping and a different seed each time. for ner, the model is evaluated just once without early stopping. the reported performance is the average of the single best run for germeval18 (coarse), germeval18 (fine) and germeval14. for comparison, we also run this evaluation pipeline on the two publicly available german bert models (deepset german bert base and dbmdz german bert base ) as well as multilingual models such as mbert base and xlm-roberta large . ## results the downstream performance graphs in figure 1 show that the models are capable of learning with most of the gains being made in the first phase of training and more incremental gains coming later. the best checkpoints come at different points for different models as can be seen in in ## gelectra large 1000 figure 1 : the f1 performance of each model averaged over the three downstream tasks over the course of language model pretraining. in germeval18 (coarse), gbert data + wwm , xlm-roberta large , gbert large and gelectra large all improve upon the previous sota. gelectra large does so with the largest margin reaching a score that is +3.93% better. in germeval18 (fine), xlm-roberta large beats the previous best by +1.39% and gelectra large sets a new sota that is better than the previous by +2.45%. in germeval14, all 7 trained models exceed the previous sota, with gelectra large showing a +4.3% improvement over the previous best. these results indicate that adding extra data gives a consistent but modest performance boost to our language models. gbert data outperforms dbmdz bert base by +0.25%, gbert data + wwm outperforms gbert wwm by +0.93% and gelectra data outperforms gelectra by +1.59%. for the bert models, whole word masking also shows a consistent positive impact with gbert wwm outperforming dbmdz bert base by +1.70% and gbert data + wwm outperforming gbert data by +2.38%. ## conclusion the set of german models which we trained vary in terms of training regime and model architecture. we hope that the results that we present here will serve as important data points to other nlp practitioners who are looking to train language models from scratch but are limited by compute. our experiments should give other teams a sense of the batch sizes and training lengths that make for efficient model training. on top of this, we also present a set of gelectra and gbert models which, according to our evaluations, set new sota performance for both large and base sized models on germeval18 and germeval14.