A New Dataset and Efficient Baselines for Document-level Text Simplification in G erman (2021)

Abstract: The task of document-level text simplification is very similar to summarization with the additional difficulty of reducing complexity. We introduce a newly collected data set of German texts, collected from the Swiss news magazine 20 Minuten (‘20 Minutes’) that consists of full articles paired with simplified summaries. Furthermore, we present experiments on automatic text simplification with the pretrained multilingual mBART and a modified version thereof that is more memory-friendly, using both our new data set and existing simplification corpora. Our modifications of mBART let us train at a lower memory cost without much loss in performance, in fact, the smaller mBART even improves over the standard model in a setting with multiple simplification levels.

## introduction text simplification is the process of reducing the complexity of a text to make it more easily understandable and improve its accessibility for a wider audience. depending on the use case, target groups of simplified texts may include low-proficiency readers such as persons with intellectual disabilities, prelingually deaf persons, or non-native readers. automatic text simplification (ats) employs natural language processing methods for generating a simplified version of a given text in standard language. in general, simplification often results in a reduction of content similar to summarization, but with additional syntactic and lexical changes. considering only a compression ratio in terms of sentence length or word count can be somewhat misleading since the simplified documents often elaborate on concepts and split complex sentences into smaller units. research on text simplification for german is still sparse but has gained momentum in recent years due to a number of legal and political developments in german-speaking countries, such as the introduction of a set of regulations for accessible information technology (barrierefreie-informationstechnik-verordnung, bitv 2.0) in germany, the approval of rules for accessible information and communication (barrierefreie information und kommunikation, bik) in austria, and the ratification of the united nations convention on the rights of persons with disabilities (un crpd) in germany, austria, and switzerland. in this work, we report on two contributions regarding ats for german: ## related work traditionally, ats has relied on rule-based approaches in separate steps, e.g. lexical substitutions followed by syntactic modifications. in the case of lexical simplification (i.e. the identification of difficult words and the substitution with simpler synonyms), most modern approaches include features based on semantics, context, and language models @xcite . syntactic simplification (the identification and simplification of difficult syntactic structures) is mostly done using manually written rules applied to a syntax tree @xcite . such systems are still among the most successful for languages with little simplification data such as basque @xcite , bulgarian @xcite , or french @xcite . for languages with enough parallel data (i.e. mainly english), data-driven approaches that rely on machine learning have emerged, where ats is most often framed as a monolingual machine translation task. statistical machine translation has been applied to learn complex-simple phrase correspondences from parallel sentence-aligned corpora @xcite , sometimes in conjunction with rule-based simplification @xcite or via integration of syntactic information through syntax-based smt @xcite . more recently, neural machine translation (nmt) has been used to train models to directly map complex to simple sentences. supervised learning with recurrent or transformer architectures dominate current state-of-the-art research, some with additional simplification-specific adaptations such as lexical constraints, rule-based preprocessing, or parametrization mechanisms @xcite @xcite @xcite . some unsupervised or semi-supervised neural models, which reduce the need for parallel data, have reached similar performances @xcite @xcite . finally, experiments with multi-task learning have shown promising results @xcite , with the possibility of zero-shot translations for languages without any parallel data @xcite . these approaches represent the current state of the art, but are largely limited to english (al-thanyyan and azmi, 2021) due to a lack of training data in other languages. initial experiments with german are ongoing @xcite . when simplifying text, operations often occur across sentence borders, affecting the structure of a text as a whole. this complicates the use of sentence alignment and limits the effectiveness of sentence-level simplification models. initial experiments exist that use document-level data to avoid these problems @xcite . in this paper, we treat text simplification as a document-level task similar to summarization: the model needs to identify the most relevant information from the original text and generate a condensed version thereof. on top of that, the model should ideally learn to modify syntactic structures (e.g. split long sentences) and replace complex words (e.g. compound nouns) with simpler alternatives. ## data we introduce a new data set collected from the swiss news magazine 20 minuten that consists of full articles paired with shortened, simplified summaries that serve as a quick "tl;dr" for the reader. in contrast to other data used in our work, this data set does not distinguish different simplification levels. the corpus contains a total of 18,305 articles published since 2020. for each article we collect the title, the lead, the full news text, and the summary. we also keep track of paragraph formatting, even though this information is not used in the models presented in this paper. additionally, we use a combination of two existing corpora for german ats that explicitly label the difficulty level of the target documents according to the common european framework of reference for languages (cefr) @xcite . for some documents, we have multiple levels of simplification available. 3 the levels available to us are a1, a2 and b1 (from most simplified to close to standard german). the three corpora we use for our experiments have the following characteristics: apa is an extended version of the austrian press agency corpus described in @xcite . this data set contains news articles professionally simplified to levels a2 and b1. for both the apa and the 20m data set, the compression ratio is comparable to summaries, as the simplified documents are generally much shorter than the original text. for the capito data, this is not always the case, at least in terms of word count; the simplified texts often elaborate on concepts or processes, which leads to a similar word count between the standard and the simplified documents. however, regarding content, the simplified texts usually do condense the original information to the most salient facts. for this reason, we argue that even on this data set, the task is very similar to summarization. ## model and training initial experiments showed that fine-tuning the standard pretrained mbart model @xcite from huggingface @xcite , performs relatively well with our data, however, training is very memory-intensive, requiring a 32gb gpu even with a small batch size. for this reason, we modify the original model to allow us to train on devices with less memory. 4 our modifications are based on the code for bart with longformer attention by the allen institute for ai (beltagy et al., 2020). 5 as in the bart model with longformer attention, we swap the standard attention in the mbart encoder for longformer's windowed attention. 6 this allows for increasing the maximum input positions and avoids having to truncate long source documents to a predefined length. we use a maximum input length of 4096 to cover most of the documents in our data. the new positional embeddings are initialized with a copy of the original pretrained embeddings of size 1024, as described in @xcite . the decoder remains unchanged with a maximum sequence length of 1024. furthermore, we reduce the original mbart vocabulary from 250k to 20k, keeping only those subwords and their embeddings that are most relevant for german. 7 we apply the pretrained multilingual sentencepiece model to ∼4.5 million german sentences 8 and use the most frequent 20k subwords to filter the original mbart vocabulary. we then extend the special language tokens with tags for the different simplification levels (e.g. "de_a1"). these are initialized with the pretrained embedding for the german language tag ("de_de") and updated during fine-tuning. and lastly, we add the option to train and translate mixed batches with multiple target language labels. we train our models with early stopping according to rougel on a held-out validation set. the models converge after training for 2 to 5 days, the exact configuration and hyperparameters can be found in appendix a.1. all models are trained on a single v100 gpu with the same accumulated batch size (60), but note that the standard mbart can only fit a batch size of 1 on the gpu, whereas our modified version can fit 4 samples in a batch and thus needs fewer accumulation steps. ## results the results in ## conclusions in this paper, we have introduced a data set of simplified news articles from the swiss magazine 20 minuten, aligned on document level. the task of document-level simplification resembles that of summarization, as models need to identify the salient parts and produce a condensed version of the original text. for simplification, models should also learn to simplify syntactic structures and lexical items. experiments based on fine-tuning the pretrained mbart model from huggingface show that the model can learn to produce not just condensed, but also simpler output. our added modifications make mbart fine-tuning significantly more memoryfriendly. since the new 20m data set does not distinguish between simplification levels, we use an existing data set annotated with cefr levels @xcite to evaluate our models according to specific simplification levels. results show that our modified mbart, while using considerably less memory, can simplify documents without much loss in performance on the 20m data and even improves over standard mbart on documents labeled with cefr tags. in future work, we will conduct ablation studies to measure the effect of our modifications individually, specifically, seeing whether using windowed attention to give the model access to the full source document instead of a clipped version is beneficial. lastly, automatic evaluation with metrics such as rougel, bleu, and sari do not provide sufficient insights. to get more accurate feedback and better understand issues specific to simplification, we plan to conduct an evaluation with professional translators. original (standard german) english "lonely planet" kürt salzburg für 2020 zur besten stadt "lonely planet" selects salzburg as the best city to visit in 2020 salzburg ist im kommenden jahr für den reisebuchverlag "lonely planet" die beste stadt zum bereisen. salzburg is the best city to travel to next year according to the travel book publisher "lonely planet". im neuen "lonely planets best in travel 2020" führt die mozartstadt das ranking in der kategorie der städte nicht zuletzt wegen des 100-jahr-jubiläums der festspiele an. in the new "lonely planet's best in travel 2020", the city of mozart leads the ranking in the category of cities, not least because of the 100th anniversary of the festival. der reiseführer "best in travel" kürt jedes jahr zehn top-städte, -länder und -regionen. the "best in travel" publication selects the top ten cities, countries and regions each year. "trommelwirbel, bitte", heißt es auf der homepage des verlages. "drum roll, please," reads the publisher's homepage. "der herzensbrecher einer alpenstadt besingt das jubiläum in vollen tönen." "the heartbreaker of an alpine city celebrates the anniversary in full tones." salzburg führt das ranking 2020 vor den städten washington dc, kairo, dem irischen galway und der beethoven-stadt bonn an. salzburg leads the 2020 ranking ahead of washington dc, cairo, galway, ireland, and bonn, the city of beethoven. in der länderkategorie liegt buthan voran, als top-region wurde die seidenstraße in zentralasien angegeben. buthan leads in the country category, with the silk road in central asia given as the top region. österreich kommt im ranking 2020 kein zweites mal vor. austria does not appear a second time in the 2020 ranking.