S uper GLEB er: G erman Language Understanding Evaluation Benchmark (2024)

Abstract: We assemble a broad Natural Language Understanding benchmark suite for the German language and consequently evaluate a wide array of existing German-capable models in order to create a better understanding of the current state of German LLMs. Our benchmark consists of 29 different tasks ranging over different types such as document classification, sequence tagging, sentence similarity, and question answering, on which we evaluate 10 different German-pretrained models, thereby charting the landscape of German LLMs. In our comprehensive evaluation we find that encoder models are a good choice for most tasks, but also that the largest encoder model does not necessarily perform best for all tasks. We make our benchmark suite and a leaderboard publically available at https://supergleber.professor-x.de and encourage the community to contribute new tasks and evaluate more models on it (https://github.com/LSX-UniWue/SuperGLEBer).

## introduction fueled by the release of @xcite , the development of very capable, large language models (llms) has been accelerating, which also results in the release of more and more powerful models capable of the german language @xcite . from an nlp point of view, german is a language that apart from smaller, commonly bert-based models traditionally has seen little attention when it comes to publicly available, explicitly for german pretrained foundational models. this now led to the situation that an increasing number of presumably very capable, german-pretrained llms are being released, but no established, diverse and systematic german evaluation suite for these models is available. to illustrate this point, we emphasize that, newly introduced german bert-based models have historically only been evaluated on two 1 github.com/lsx-uniwue/supergleber tasks each @xcite , which is not enough to get a comprehensive understanding of the models capabilities. hence a german evaluation suite is desireable to properly compare and assess the abilities of widely used, but also newly developed models, like there is e.g. for english with glue @xcite , super-glue @xcite or even more recently opencompass @xcite . commonly researchers turned to these english evaluation suites to assess their german models and -for lack of a better solution -had to help themselves by translating very hard benchmark datasets from english to german using e.g. chatgpt @xcite . this arguably leads to unreliable results, as the models are evaluated on a task that has been machine-translated sometimes by the very same model these benchmarks were created to be hard to solve and understand for @xcite . our benchmark evaluation suite thus aims for both: 1. aggregating a diverse set of available german natural language understanding (nlu) tasks, 2. identifying commonly used german-pretrained llms and evaluating the models on this benchmark. to this end, we select a wide range of different task types to make sure to properly assess the models' capabilities, such that our benchmark includes document classification, sequence tagging, sentence similarity and question answering tasks ( our contributions are as follows: 1. assembling a diverse benchmark for german nlu consisting of 29 different tasks across four task types, 2. comprehensively evaluating 10 different german-pretrained llms across various architectures on this benchmark, 3. providing this open-source evaluation framework to the community, allowing for easy extension in the future. ## german evaluation tasks in order to create a challenging and diverse benchmark for german nlu we select a wide range of different tasks from various different domains for our evaluation suite. we also list the included tasks as well as statistics for each dataset in the appendix in ## evaluated models in our evaluation we aim to cover a large number of different models and model types available for the german language ( we evaluate three different bert models, one being "bert-base-german-cased", pretrained on 12 gb of wikipedia, legal documents and news. the other two berts have been pretrained by @xcite and only differ in size: "deepset/gbertbase" and "deepset/gbert-large". both models have been pretrained on 163.4 gb of german text, mostly consisting of oscar, enriched with opus, wikipedia and legal documents. we also evaluate "uklfr/gottbert-base" @xcite , which is a roberta model pretrained on 145 gb of os-car, wikipedia and a book corpus. for decoder models we evaluate "dbmdz/german-gpt2" @xcite , which is a gpt2 model pretrained on about 16 gb of german text, consisting of subtitles, and a diverse set of web crawls like commoncrawl and news. "leolm/leo-hessianai-7b" is a very recent, comparably large language model, finetuned from a llama2 checkpoint using german text @xcite mostly sourced from oscar and has only been evaluated on a machine-translated version of the english openllm dataset. furthermore, we consider the multilingual-trained "bigscience/bloomz-560m" model @xcite . it was trained in two steps: first on a 1.5 tb multilingual corpus of 45 languages and 12 programming languages using causal language modeling (workshop et al., 2023), then further multilingual, multi-task pretraining using supervised tasks @xcite . we also evaluate the encoder-decoder multilingual-trained "bigscience/mt0-small" model @xcite , which was finetuned analogously to the previously introduced bloomz model, but is instead finetuned from the "google/mt5-small" checkpoint. this model in turn was trained on 101 languages, including german, using the "span-corruption" objective @xcite on the c4 corpus @xcite and is also included in our evaluation. lastly we evaluate the multilingual-trained "facebook/mbart-large-50" model, trained on 50 languages, including german, using the translation objective @xcite . in contrast to bart, the mbart model was only trained on the translation objective between any pair of languages and not additionally on the denoising objective, thus never saw german text as input and target at the same time. nevertheless @xcite showed that mbart is able to perform well on german-only tasks. ## evaluation we extensively evaluate the models from section 4 on the tasks introduced in section 2 resulting in the second row gives the type of metric used for the respective task type. here "mixed" means that -like in other benchmarks (2019; 2020; 2023) -at least two kind of metrics have been averaged together. the results marked with † have been averaged over tasks for which a "cuda oom" error occured on an a100 80gb gpu (only mbart). the results marked with ‡ have been averaged over tasks where a "shapeerror" occured (only bloomz). both symbols have been placed at all averages this affects transitively. all missing values have been treated as a 0.0 when calculating the average. types. we also list the results for the individual tasks in the appendix in appendix d. in the following we will discuss our results under various different aspects. ## conclusion we introduce the first large and diverse german language understanding benchmark for language models, consisting of 29 different tasks and covering four different task types: text classification, sequence tagging, sentence similarity and question answering. the text classification and sequence tagging tasks themselves contain a wide range of different language understanding tasks, covering various different domains and prediction targets. we evaluate 10 different models, including four encoder-only, three decoder-only and three encoder-decoder models on our newly introduced benchmark. in our comprehensive evaluation we find, that on average the encoder models perform best and are usually close to each other in performance on the classification and sequence tagging tasks. despite not being encoder models, the two largest evaluated models mbart and leo-7b are performing comparably well across all tasks, mostly being competitive with the encoder models. in contrast, we did not find a clear advantage for the larger encoder model, as the gbert-large model is not able to consistently profit from its larger size, often being outperformed or matched by its smaller counterparts. we make the benchmark and leaderboard publicly available and encourage the community to contribute tasks as well as models to the benchmark.