Language Models for G erman Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training (2023)

Abstract: Automatic text simplification systems help to reduce textual information barriers on the internet. However, for languages other than English, only few parallel data to train these systems exists. We propose a two-step approach to overcome this data scarcity issue. First, we fine-tuned language models on a corpus of German Easy Language, a specific style of German. Then, we used these models as decoders in a sequence-to-sequence simplification task. We show that the language models adapt to the style characteristics of Easy Language and output more accessible texts. Moreover, with the style-specific pre-training, we reduced the number of trainable parameters in text simplification models. Hence, less parallel data is sufficient for training. Our results indicate that pre-training on unaligned data can reduce the required parallel data while improving the performance on downstream tasks.

## introduction automatic text simplification (ats) is the task of simplifying a text's lexical and structural complexity while preserving its original meaning. easyto-read texts can help people with learning deficiencies or non-native speakers gain access to texts that they could not understand otherwise. on the one hand, ats can be used to create assisting tools for people with reading disabilities or professional translators @xcite . on the other hand, ats can be applied as a preprocessing step for other natural language processing tasks such as machine translation or information retrieval to improve their performances @xcite , making it an important field of study. in german, there exist multiple levels of simplified language. in contrast to the underspecified simple language, the so-called leichte sprache (easy language) enforces a very strong simplification level and follows predefined structural rules @xcite . these rules include conveying only one message per sentence (structural simplification), restriction to common words (lexical simplification), and usage of simplified grammar (syntactical simplification). this simplified grammar breaks with standard german grammar, for example, by using dative instead of genitive to indicate possession. we consider easy language as a standalone language style. therefore, we refer to easy language data as monolingual data in the further course of the paper, even though it is german as well. this work shows the benefits of fine-tuning language models for specific styles and characteristics. we publish and discuss a collection of causal language models fine-tuned for german easy language. as shown in previous work @xcite , pre-training language models for specific domains can benefit the performances of downstream tasks in the respective domain. we extend this analysis to the language style of easy language. in addition, the fine-tuned models can be used to generate text with the specificities of easy language, for example, in data augmentation applications. finally, we present how these models can serve as plug-in-decoders in bartlike architectures @xcite to speed up and improve the training on sequence-to-sequence (seq2seq) tasks. therefore, our contributions are the following: with the reduction of trainable parameters, less aligned data is needed to train an ats system. especially for languages other than english, where aligned data is sparse, pre-trained causal language models can improve ats performance. we publish our code and results for further research and application foot_0 . ## related work causal language models can complete text based on a prompt. in contrast to masked language models, where the models know about the context before and after a specific token, these causal language models rely only on the input and the previously outputted tokens. therefore, they are called autoregressive models. the generative pre-trained transformer (gpt) @xcite is a prominent example of such an autoregressive language model. it was trained on a collection of web data and, thus, outputs text for general purposes. previous work has fine-tuned gpt for multiple domains and tasks, such as the task of quest generation in games @xcite or the medical domain @xcite . in addition to domain adaption, gpt was tailored to specific text styles and characteristics. these style transfer approaches include fine-tuning for poem generation @xcite or the reduction of non-normative clauses @xcite . @xcite trained a gpt model to mimic the language of people with dementia. by calculating the perplexities of texts with the finetuned and original version, they could distinguish samples from healthy and diseased people. @xcite adapted a language model for simple language by only masking easy-tounderstand words in training. however, this model is a masked language model that can only fill in blanks and not generate text from scratch. most similar to our work is the transformerlm by @xcite trained for japanese text simplification. the authors used a parallel corpus to directly fine-tune a gpt model for simplification. in contrast, our models are finetuned on monolingual easy language data. therefore, they do not require alignments and can be used for a broader range of tasks. ## datasets several sources are available in easy language; however, they mostly encompass news websites, and only a few are aligned with articles in standard german. in the following sections, we detail the information on the data used in our training, including the easy language monolingual corpus utilized for fine-tuning german language models and the parallel corpus for the downstream task of text simplification. the dataset utilized for the downstream task of text complexity prediction is publicly available as a part of the germeval 2022 shared task @xcite (refer to subsection 5.4). we published scrapers to recreate our sources for the use of the academic community foot_1 . we also provide an overview of available monolingual and parallel data sources for simplified german beyond our training data in appendix a. ## methodology our approach is divided into two parts. first, we fine-tuned generative language models for german easy language. then, we used these models as plug-in decoders in a bart-based simplification task. ## evaluation this section describes four experiments to compare our fine-tuned (ft) models with their original (o) versions. first, we measured the models' perplexities on easy and normal texts and analyzed the readability of their outputs. in addition, the models were evaluated on two downstream tasks; text complexity prediction and automatic text simplification. ## conclusion with this paper, we have published a collection of causal language models for german easy language. these models mimic the style of easy language and favor short and precise sentences. in addition, they adapt to the conventions of only conveying one thought per sentence and putting a line break after every sentence. we exploited these pre-trained models in a sequence-to-sequence text simplification task. as the models were already fine-tuned to the desired output style, we only had to train the encoder-decoder cross attention and, thus, reduced the number of trainable parameters by 93%. with this, training a style-transfer system becomes feasible for settings with few aligned data or a lack of computational power.