G ott BERT : a pure G erman Language Model (2024)

Abstract: Pre-trained language models have significantly advanced natural language processing (NLP), especially with the introduction of BERT and its optimized version, RoBERTa. While initial research focused on English, single-language models can be advantageous compared to multilingual ones in terms of pre-training effort, overall resource efficiency or downstream task performance. Despite the growing popularity of prompt-based LLMs, more compute-efficient BERT-like models remain highly relevant. In this work, we present the first German single-language RoBERTa model, GottBERT, pre-trained exclusively on the German portion of the OSCAR dataset. Additionally, we investigated the impact of filtering the OSCAR corpus. GottBERT was pre-trained using fairseq and standard hyperparameters. We evaluated its performance on two Named Entity Recognition (NER) tasks (Conll 2003 and GermEval 2014) and three text classification tasks (GermEval 2018 fine and coarse, and 10kGNAD) against existing German BERT models and two multilingual models. Performance was measured using the F1 score and accuracy. The GottBERT base and large models showed competitive performance, with GottBERT leading among the base models in 4 of 6 tasks. Contrary to our expectation, the applied filtering did not significantly affect the results. To support the German NLP research community, we are releasing the GottBERT models under the MIT license.

## introduction the computation of contextual pre-trained word representations is the foundation of neural language modeling (lm) in natural language process-ing (nlp). the field of nlp experienced remarkable progress by the use of transformer-based approaches @xcite . especially bidirectional encoder representations from transformers (bert) @xcite impacted the field which subsequently was robustly optimized to roberta @xcite . these transformerbased approaches rely on large-scale pre-trained language models, which are subsequently finetuned through supervised training on specific downstream tasks, leveraging the context representations learned from the generic domain to achieve superior performance compared to training from scratch, a process known as transfer learning. on the other hand, the computation of the language model is performed self-supervised. large text blobs are required for training and strong hardware such as hundreds of graphics processing units (gpu) @xcite or tensor processing units (tpu) @xcite . initially, most of the research took place in english followed by multilingual approaches @xcite . although, multilingual approaches were trained on large texts of many languages, they can be outperformed by single language models @xcite @xcite . additionally, a single language model requires fewer computational resources and a smaller dataset compared to the vast and varied data needed for multilingual models. single language models trained with the open super-large crawled almanach corpus (os-car) @xcite showed good performance due to the size and variance of the os-car corpus @xcite . the focus of pre-training language models has shifted towards scaling up transformer-based large language models (llms) @xcite @xcite . these models, like llama 3 1 , are vastly larger than the aforementioned models @xcite . despite the advantages of llms, such as gradientfree prompting, smaller models remain valuable for their efficiency and practical deployment. for these reasons, we pre-trained the first german roberta single language models with the german portion of the first published deduplicated version of os-car -the german oscar text trained bert (gottbert). inspired by flaubert @xcite , we also trained models with a filtered version of oscar. in an evaluation we compared the performance of all models on the two named entity recognition tasks conll 2003 and germeval 2014, nli as well as on the text classification tasks germeval 2018 and gnad with existing german single language bert and models and two multilingual models. our contributions can be summarized as follows: ## related work most recently, transformer-based models widely impacted the field of nlp. from neural translation @xcite to generative language models starting with gpt-2 @xcite , remarkable performance gains were achieved. with bert, an approach to facilitate pre-trained transformer-based models was introduced. fine-tuned on downstream tasks, bertbased approaches improved the performance of several nlp tasks @xcite ., 1 https://github.com/meta-llama/llama3 2019 ). however, bert models were first released as single-language models in english based on 16gb of raw text and as the multilingual model mbert based on wikipedia in about 100 languages @xcite . these models were followed by single-language models for several languages: bertje (de vries et al., 2019) for dutch, finbert @xcite for finish, ger-manbert foot_0 and a german bert from the mdz digital library team at the bavarian state library to which we refer to as dbmz bert in this paper foot_1 . germanbert was trained using 12gb of raw text data basing on the german wikipedia (6gb), the openlegaldata dump (2.4gb) and news articles (3.6gb). dbmz bert used as source data a german wikipedia dump, eu bookshop corpus, open subtitles, commoncrawl, paracrawl and news crawl which sums up to a dataset of 16gb. with the release of roberta a new standard for raw text size was set as it was trained on 160gb of raw english text. further, roberta enhances the original bert approach by removing segment embeddings, next sentence prediction and improved hyperparameters. additionally, instead of using wordpiece @xcite tokenization, roberta utilizes gpt2's byte pair encoding (bpe) @xcite with the benefit that language-specific tokenizers are not required. other than mbert, the multilingual xlm-roberta @xcite was trained on 2.5tb of filtered commoncrawl data. camem-bert is a french roberta model that was trained on the oscar and uses sentencepiece @xcite bpe. further, they pre-trained a model with 4gb of the french oscar portion and another model with 4gb of the french wikipedia. the comparison of these models using downstream tasks shows that high text variance leads to better results. umberto foot_2 is an italian roberta model, similarly designed as camembert. robbert, the dutch single language roberta, was trained on 39gb of the dutch portion of the oscar and outperformed bertje. a more recent version of rob-bert showed the performance gains of language specific bpe compared to the english based gpt2 bpe in downstream tasks. susequently, flaubert @xcite for french was released trained on 71gb data. they cleaned a 270gb corpus of mixed sources by filtering out meaningless con-tent and unicode-normalization. data was pretokenized by moses @xcite and encoded by fastbpe foot_3 which is an implementation of @xcite . after the publication of roberta, google released electra @xcite which denoted an improvement to the bert architecture. based on these developments, further german language models were then published: gbert and gelectra @xcite . these models were trained on the german portion on oscar (145gb) besides three other datasets (18.4gb) and outperformed previously released german single language models. ## methodology following the approach of utilizing the oscar, we computed the german oscar text trained bert (gottbert). however, the drawback of bert approaches is the computational power requirement. multiple gpus or tpus were used for pre-training. all previously listed robertabased models were computed on gpus whereas gottbert is the first published roberta model pre-trained on tpus. ## results as gottbert large was the same checkpoint for last and best, we ended up with 7 gottbert checkpoints. gottbert base was saved after 91848 training steps (12 epochs). the filtered models, both f gottbert large and f gottbert base , were saved also saved after they trained 94530 steps (15 epochs). for these models, this was approximately 1 epoch earlier then the full training steps. the dirty models trained up to 13.07 epochs and the filtered ones up to 15.87 epochs. the results of all the downstream tasks are listed in nli among the large models, gottbert large achieved an accuracy of 82.46%, while f gottbert large and f gottbert large slightly improved on this with accuracies of 83.31% and 82.79%, respectively. these results position the gottbert models as strong contenders, though they were outperformed by gelectra large , which achieved the highest accuracy of 86.33%. gbert large also performed well with an accuracy of 84.21%, followed closely by xlm-r large with 84.07%. for the base models, gottbert base and gottbert base achieved accuracies of 80.82% and 81.04%, respectively, demonstrating competitive performance. overall, the results indicate that while the gottbert models exhibit strong performance in the nli task, gelectra models generally achieved the highest accuracies in both the base and large categories. named entity recognition for the ner tasks, the base versions of the gottbert models showed competitive performance with f1 scores around 87.50% on the germeval 2014 dataset and around 86.10% on the conll dataset. the large versions of these models improved upon these scores, with f gottbert last achieving an f1 score of 88.27% on germeval 2014 and 86.78% on conll. however, among the large models, xlm-r achieved the highest f1 score of 88.83 on the germeval 2014 dataset, whereas gbert large performed the best on the conll dataset with an f1 score of 87.19%. overall, the large gottbert models demonstrated robust performance across both datasets, validating their effectiveness for the tasks at hand. among the base architecture the gottbert models took the lead. text classification for germeval 2018, the large gottbert models showed again competitive performance. the gottbert large and f gottbert large models achieved overall f1 scores of around 79.3 for coarse-grained predictions, with minimal differences in fine-grained scores around 54.7. f gottbert large had slightly lower performance in coarse predictions but was consistent in fine-grained predictions. in comparison, gelectra large outperformed all large models in coarse-grained predictions with an f1 score of 81.28, and also showed strong fine-grained performance with an f1 score of 56.17. gbert large followed closely with 80.84 in coarse-grained predictions and led in fine-grained predictions with 57.37. xlm-r large scored slightly lower than the gottbert models, with 79.05 and 55.06 in coarse and fine-grained predictions, respectively. among the base models, gottbert base scored 78.17 for coarse-grained predictions and 53.30 for fine-grained predictions. gottbert base showed similar performance in coarse-grained prediction with an f1 score of 78.18 and a fine-grained f1 the evaluation of gottbert models on the 10kgnad dataset demonstrated their strong performance in german news classification tasks. for the large models, gottbert large achieved an accuracy of 90.24, while f gottbert large and f gottbert large scored 90.31 and 90.17, respectively. among the competing models, gelectra large outperformed all with an accuracy of 90.97, followed by gbert large at 90.74, and xlm-r large matching f gottbert large at 90.17. for the base models, gottbert base excelled with an accuracy of 90.27, while gottbert base and f gottbert base achieved 89.64 and 89.79, respectively. f gottbert base performed well with an accuracy of 89.92. notably, dbmdzbert scored the highest among the base models with 90.34, followed closely by gbert base at 90.30. german-bert, xlm-r base , and mbert also showed competitive accuracies ranging from 88.90 to 90.18. overall, gottbert models demonstrate strong and consistent performance across the classification tasks, highlighting their robustness and effectiveness. ## discussion in this study, we successfully trained and evaluated gottbert models on two versions of the oscar corpus. noteworthily, @xcite published gottbert base as preliminary work. since its release, the model has been utilized for various purposes in several related works showing its relevance. it has served as a baseline model in research studies @xcite @xcite . beyond that, in the field of neural machine translation (nmt), researchers have used contextualized embeddings from pre-trained models including gottbert @xcite . additionally, the model has been applied to named entity recognition (ner) tasks in the medical field, using both translated @xcite and synthetic @xcite ner data annotated with medical entities through a fine-tuned version of gottbert. furthermore, a specialized version of the model known as bio-gottbert has been developed specifically for the medical domain @xcite . tpu training generally does not permit dynamic memory allocation, as tpus are designed for ef-ficient, high-throughput computation with fixed memory allocation. as a result, the corpus was processed as a single stream rather without considering document boundaries, unlike roberta training on gpus. additionally, due to limitations in the fairseq implementation we used, we conducted the computations in 32-bit mode since 16-bit was neither properly implemented nor tested, leading to increased memory usage and hence more computation time required. also, we used more conservative learning rates than the ones recommended by the fairseq documentation for pre-training on gpu. dataset annotation is expensive, as it is usually performed by multiple annotators. as the one-class svd is trained unsupervised and annotations were only used to estimate its performance and to find a good nu, the dataset was only annotated by one person of our team. the mcc of 0.5663 suggests the model has a moderate ability to make accurate binary predictions overall, balancing true and false positives and negatives. the f1-score of 0.8578 indicates the model is performing well in terms of precision and recall for the positive class. these metrics together imply that while the model is quite good at correctly identifying positive cases and maintaining a balance between precision and recall, there is still room for improvement in making more accurate overall predictions as reflected in the moderate mcc score. possibly a better approach would have been possible with a more complex model, the computational cost as well as the annotation efforts would have been much more expensive. however, the use of language properties on a syntactical level denoted an efficient and creative approach that could be carried out with fair effort, including in terms of computational costs. our results do not provide a clear conclusion regarding the impact of data cleaning on the resulting model. a ranking of all base models by their position, taking into account only the number of first and second place models, shows that f gottbert base is on top. however, when considering the gottbert models in isolation as a subgroup, the gottbert base model stands out as the top performer among the base models. conversely, when evaluating the large models all gottbert models were outperformed by the competitors. in this global comparison, the unfiltered model gottbert large emerges as the best performer of all the gottbert models winning one second place in conll03, while f gottbert large emerges as the superior performer in the iso-lated comparison. we anticipated a more definitive outcome, particularly since the filtered models were pre-trained for an additional epoch due to the smaller corpus size. the importance of hyperparameters in model performance is welldocumented, even considering random seeds as shown by @xcite . this suggests that our chosen hyperparameters could be extended even more to find better ones. nevertheless, clear differences should have been already pointed out within our experimental setup. however, we did not see any great benefit, especially considering the high cost of cleaning a data set in this way. potentially, the data cleaning process might have inadvertently removed important variance from the corpus. according to the @xcite , a corpus with greater variance generally leads to better performance compared to a homogeneous one. therefore, we suggest creating a corpus with more variance. in our case, incorporating additional corpora such as opus, wikipedia, and openlegal-data could have been beneficial. moreover, whole word masking (wwm) leads to better models @xcite . finally, for roberta models, the size of the vocabulary also impacts performance, as investigated by @xcite . according to their findings, our vocabulary size wasn't a bad choice. finally, potential risks include bias and fairness issues, leading to unfair outcomes. data privacy concerns exist, with the model potentially revealing sensitive information. this affects especially the corpus used and the filtered version of it, as the filtering did not operate on a semantical but on a syntactical level. misuse could result in harmful content, like misinformation or spam. overreliance without human oversight might cause critical errors, especially in healthcare or finance. the environmental impact of training such models is considerable due to high energy consumption. it is also vulnerable to adversarial attacks. ## conclusion in this work we present the german single language roberta based model gottbert in two versions computed on a corpus with 145gb and a filtered version with 121gb plain text with both base and large roberta architecture. gottbert is the first german single language roberta based model. in our experiments, we were able to show that the base models lead 4 of 6 tasks. however, this did not apply to the large models. the comparison of the pre-training with filtered and raw corpus did not show a clear result as anticipated. we therefore suggest considering other measures, such as increasing variance by using many corpora and using wwm. we release all gottbert models in huggingface and fairseq format to the community under the mit license.