G erman Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data (2024)

Abstract: This study pioneers the use of synthetically generated data for training generative models in document-level text simplification of German texts. We demonstrate the effectiveness of our approach with real-world online texts. Addressing the challenge of data scarcity in language simplification, we crawled professionally simplified German texts and synthesized a corpus using GPT-4. We finetune Large Language Models with up to 13 billion parameters on this data and evaluate their performance. This paper employs various methodologies for evaluation and demonstrates the limitations of currently used rule-based metrics. Both automatic and manual evaluations reveal that our models can significantly simplify real-world online texts, indicating the potential of synthetic data in improving text simplification.

## introduction in our modern and digitalized societies, access to information is essential for active participation. however, certain groups, such as individuals with intellectual disabilities or non-native speakers, often struggle to understand the local language, which can impede their social and civic engagement. each group faces unique challenges in text comprehension. integrating automatic text simplification tools can significantly benefit these groups by providing accessible information, thereby providing a pivotal means for greater inclusion. various linguistic initiatives, like (netzwerk-leichte-sprache, 2022), have been established in german-speaking regions to address this, specifically designed for different target groups. such efforts align with international legal frameworks like article 9 of the un convention on the rights of persons with disabilities foot_0 , which advocates for the right to accessible communication. figure 1 : illustration of synthetic data generation. data is crawled from websites specializing in language simplification. gpt-4 generates texts in everyday language, ensuring the original content remains unaltered. we construct a simplification dataset where these texts serve as input while the crawled simplifications act as reference simplifications. creating simplified content manually is a laborintensive and time-consuming process, significantly hindering its broad availability and accessibility. in contrast, large language models (llms), especially smaller ones fine-tuned for text simplification, offer a viable and efficient alternative @xcite . these smaller models require fewer resources and are simpler to operate than larger llms, making them ideal for scaling up the process of automatic language simplification. a key challenge in finetuning llms for text simplification lies in the limited availability of parallel data @xcite . our approach, as illustrated in figure 1 , tackles the challenge of data scarcity in language simplification by creating semi-synthetic data. this involves crawling various sources for already simplified web content and then utilizing gpt-4 to generate hypothetical original texts corresponding to these simplifications. our dataset thus comprises gpt-4's outputs as the inputs and the crawled con-tent as the simplified outputs, forming a text simplification dataset. we apply this dataset as the basis for finetuning large language models (llms) for automatic text simplification. we publish all necessary resources to reproduce this paper's results on a public github repository foot_1 . our scientific contributions can be summarized as follows: various approaches and methodologies have been developed for automatic german text simplification. @xcite proposed a two-step approach utilizing pretrained language models finetuned on german simplifications to diminish the requirement for parallel data. in contrast to our approach, the parallel data contains a mixture of summarization and simplification and targets only newspaper articles. @xcite train german text simplification models by using labels to target specific language levels, ensuring model adaptations are level-appropriate and control copying behavior. their dataset focuses on newspaper articles and sentence-level simplification. diverging from neural network-based methods, @xcite introduced methodology based on parse trees. similarly, @xcite offered a pattern-based syntactic simplification framework. @xcite presented methods for text simplification in languages with limited simplified corpora, including lexical substitution and monolingual translation, focusing on resource-scarce languages like japanese. regarding evaluation metrics, @xcite investigate the limitations of bleu as a widespread evaluation metric for text generation tasks. @xcite explored the correlation between existing metrics and human judgments in multi-operation text simplifications, providing insights into the appropriateness of automatic metrics for assessing text simplification. @xcite introduced lens, a learnable evaluation metric for text simplification trained on modern language models, showing a better correlation with human judgment. regarding parallel corpora and resources, @xcite aggregated corpora for the automatic processing of simplified german, providing resources for training and evaluation. @xcite ) create so-called pseudo parallel sentence pairs of simple and complex sentences from given sentence collections. @xcite introduced sniml, a multilingual corpus of news articles in simplified language, and @xcite showcased a dataset for documentlevel text simplification in german, including articles paired with simplified summaries. @xcite addressed the challenge of concurrently summarizing and simplifying longer texts, introducing a new dataset for joint text simplification and summarization. @xcite introduces a dataset with parallel sentence-level simplications and additional information about the document's rhetorical structure. text simplification corpora exist for various languages. for example, @xcite introduced a dataset pairing english wikipedia with simple english wikipedia, enabling the analysis of various simplification operations, including rewording, reordering, insertion, and deletion. in easy language generation, @xcite investigated the feasibility of using chat-gpt to translate administrative texts into german leichte sprache (easy language), a highly regulated language variety with a focus on text simplification. ## task definition in this section, we give a task definition for language simplification. this definition outlines the requirements for the trained models and motivates our methodology for dataset creation. the inputs space consists of editorially created german texts. based on the selection of web sources, we assume predominantly grammatically complete sentences and quality-assured content. in the context of this work, we exclude user-created input and social media texts. these web documents contain multiple paragraphs and sentences. the output space consists of simplifications of the input. the style and level of simplification correspond to the contents currently available in the german language, precisely as they are presently accessible. we aim to avoid modifications of the content, like summarization. however, understandability may require additional explanations of certain concepts in the simplified texts. two concepts for language simplification have been established for the german language: einfache sprache (simple language) and leichte sprache (easy language). we seek to explain how our methodologies intersect and align with these well-established frameworks. simple language covers text simplification in general. possible target groups include readers unfamiliar with the domain or used language, for example, in legal or medical texts or language learners. the target group can fundamentally understand the concepts. linguistic complexity, however, makes understanding more difficult. moreover, these simplifications can aid in language acquisition. in contrast, easy language is aimed at people with severely limited text comprehension, such as those with intellectual disabilities. fixed sets of simplification rules have been established (netzwerk-leichte-sprache, 2022). these rules cover areas like syntactical, lexical, or typographical simplifications. in our approach, we scrape texts from various sources, each characterized by its distinct language style. our research hints that these target texts generally do not conform to the rules of easy language. rather, many crawled texts may align more with the domain of simple language. ## methodology this study aims to perform task-specific fine-tuning of llms. the extensive volume of publications in this domain makes it impractical to evaluate all available models and training configurations against one another. instead, we aim to justify our core design choices in this chapter and provide further justification through targeted evaluation in the subsequent chapter. ## evaluation in this chapter, we provide a comprehensive evaluation of the finetuned models. our analysis is twofold: firstly, we assess the performance of various model configurations on the semi-synthetic dataset. this evaluation will delve into how different configurations perform in terms of efficacy, which will be measured using a range of metrics. secondly, we extend our evaluation to include an analysis of crawled web content. this is a vital step towards demonstrating the real-world applicability of our models. ## limitations our examination reveals that rule-based metrics have limited suitability for evaluating state-of-theart models in document-level simplification. while our chosen evaluation methodology yields promising results, it lacks a targeted analysis of the endusers for whom the simplification is intended, a scope beyond the ambit of this study. further-more, alternative methods to simplify language using llms, such as in few-shot learning, merit a comparative analysis against our approach. ## conclusion this study represents a significant stride in tackling the challenge of data scarcity in automatic text simplification. we have crafted a semi-synthetic dataset that has proven effective for training models, which are capable of simplifying complex texts. notably, our models trained on this synthetic data have demonstrated proficiency in simplifying real web content, validating the practicality of our approach. semi-synthetic data offers the opportunity to efficiently integrate large amounts of existing simplifications into supervised training without manual effort. this is an efficient and promising alternative to alignments or the manual creation of parallel data. a vital contribution of this work is the open availability of both the dataset and the models, which serve as a foundational resource for researchers in the text simplification field. integrating state-ofthe-art llms with supervised learning has shown to be an efficient method for german text simplification. the limitations of current automated and rule-based metrics, such as bleu, meteor, and sari, are increasingly apparent, particularly for our document-level simplification dataset. this suggests that state-of-the-art llms may advance to a point where more nuanced evaluation methodologies are required to accurately measure performance differences and select superior models. looking ahead, there are promising directions for future research. one crucial area is adapting generation to adhere more closely to the specific simplification styles, for example, following the rules of easy language and thereby tailoring simplifications more effectively to target audiences. this could involve exploring ways to influence the style of simplification, which would enhance the applicability of these models in real-world applications. by fine-tuning the models to align with the nuanced requirements of different user groups, we can make significant strides toward more inclusive and accessible digital content.