On the Impact of Cross-Domain Data on G erman Language Models (2023)

Abstract: Traditionally, large language models have been either trained on general web crawls or domain-specific data. However, recent successes of generative large language models, have shed light on the benefits of cross-domain datasets. To examine the significance of prioritizing data diversity over quality, we present a German dataset comprising texts from five domains, along with another dataset aimed at containing high-quality data. Through training a series of models ranging between 122M and 750M parameters on both datasets, we conduct a comprehensive benchmark on multiple downstream tasks. Our findings demonstrate that the models trained on the cross-domain dataset outperform those trained on quality data alone, leading to improvements up to 4.45% over the previous state-of-the-art.

## introduction with established scaling laws @xcite , increasing the model size and datasets has become a consistent scheme in the rapidly evolving field of large language models (llms) @xcite @xcite . this pattern exists for any recently proposed architecture @xcite , independent of decoderonly @xcite @xcite , encoder-only @xcite @xcite , or encoder-decoder models @xcite . as these have shown impressive performance for natural language understanding, many works built upon them for specific tasks such as medical language understanding @xcite or human interaction @xcite , or advance them through intelligent design choices. for instance, @xcite introduced the disentangled attention mechanism that encodes the token position and information separately, enabling it to surpass human performance on the superglue benchmark @xcite . however, as the data requirements for llms can become difficult to acquire for smaller institutions, it poses the questions whether strictly increasing the data amount is the only solution or if it is possible to enable the training of llms through a more elaborate data selection process. the pile @xcite has played a crucial role in emphasizing the advantages of augmenting web-crawl datasets with a wide array of domainspecific data. this realization has prompted the training of large language models (llms) on the pile and similar datasets @xcite . @xcite presented a 1.3b parameter model trained on a dataset of 7b carefully curated tokens that is comparable to gpt3.5 on humaneval indicating that data quality apart from general scaling laws @xcite play an important role as well. however, a notable disparity arises when considering the german language, as there is currently a lack of comparable variety-focused datasets. to address this gap, we propose various methods for curating a diverse german dataset, even for domains with limited resources, such as the medical field. our methods are applicable to a wide range of languages, as approximately 74% of this dataset is acquired through automatic translation and webcrawl filtering techniques. we demonstrate the advantages of such datasets over pure web-crawlbased datasets with a set of encoder-only models we call geberta. in 2020, chan et al. published a set of german transformer @xcite models based on bert and electra, achieving state-of-theart results on two downstream datasets. around the same time, gottbert was released, a german roberta model @xcite . since the two publications were only evaluated on two common downstream datasets and both dispensed with hyper-parameter tuning, it is unclear how they compare. in this work, we train new german lms with the deberta @xcite architecture and assemble a comprehensive benchmark of eight downstream datasets. by fine-tuning the hyperparameters of our models and the previously released ones, we showcase the enhanced model performance achieved through the deberta architecture and our cross-domain dataset. @xcite and @xcite were based on the web-based oscar dataset. moreover, @xcite explored the impact of data quantity on model performance. the aim of this work is to quantify the effects of data quality and the inclusion of cross-domain data on encoder-only models. our contributions can be summarized as follows: ## data collection cc100 prioritizes quality in its construction. while this emphasis on quality might be beneficial, it also has certain limitations when it comes to the dataset. specifically, domains that encompass informal or noisy texts, such as social media posts or medical documents, may be at a disadvantage due to the focus on quality. to address this concern and explore the trade-off between quality and variety, we have developed two distinct datasets. the first dataset d quality is centered around ensuring high quality, featuring carefully curated and reliable content. however, recognizing the importance of incorporating diverse texts, we have taken steps to extend the quality-focused dataset to include a broader range of material, thereby increasing its variety. by creating these two datasets with different emphases, we aim to examine whether the advantages gained from a variety-focused approach outweigh any potential drawbacks associated with reduced data quality. ## discussion our extensive experiments provide clear evidence that the models that we have published have achieved a significant step forward in the processing of natural language in the german language (nlp). it is important to note that certain improvements can undoubtedly be attributed to the de-berta architecture. this can be seen in improved performance on tasks such as question answering and named entity recognition with larger entity spans. however, a direct comparison between models trained on web crawl data alone and those trained on cross-domain data shows a clear improvement, which can be attributed to including diverse pre-training data. we support this claim with a significance test between geberta q base and geberta v base which is shown in an interesting finding from our research is that our base model achieved a comparable level of performance to the previous large state-of-the-art models. this finding suggests that the required com- ## conclusion in conclusion, this paper presents german language models that achieve state-of-the-art results on various natural language processing tasks by pre-training on more diverse datasets. the experiments show that training language models on crossdomain datasets leads to improved performance compared to models trained on web crawl data alone. the results suggest that the computational resources required for training can be significantly reduced while still achieving high performance through the use of diverse datasets and efficient model architectures. overall, the research provides evidence that incorporating diverse, multilingual data is crucial for building high-quality language models.