SIMSUM : Document-level Text Simplification via Simultaneous Summarization (2023)

Abstract: Document-level text simplification is a specific type of simplification which involves simplifying documents consisting of several sentences by rewriting them into fewer or more sentences. In this paper, we propose a new two-stage framework SIMSUM for automated document-level text simplification. Our model is designed with explicit summarization and simplification models and guides the generation using the main keywords of a source text. In order to evaluate our new model, we use two existing benchmark datasets for simplification, namely D-Wikipedia and Wiki-Doc. We compare our model’s performance with state of the art and show that SIMSUM achieves top results on the D-Wikipedia dataset SARI (+1.20), D-SARI (+1.64), and FKGL (-0.35) scores, improving over the best baseline models. In order to evaluate the quality of the generated text, we analyze the outputs from different models qualitatively and demonstrate the merit of our new model. Our code and datasets are available.

## introduction text simplification is an important technique that aims to simplify a document to make it more understandable and accessible for people at different education and reading levels while still retaining the content of the original text @xcite . it concentrates on lexical simplification (i.e., using simpler vocabulary and including definitions that provide explanations in simple terms) as well as syntactic simplification (i.e., using less complicated sentence structures and grammar) @xcite . simplification is considered as a sequenceto-sequence text generation problem, closely resembling other nlp generation tasks such as text summarization @xcite @xcite and paraphrasing @xcite . the applications of text simplification are broad. it can be an important tool for assisting children @xcite and non-native speakers @xcite to understand advanced texts with ease. additionally, it is helpful for enabling people suffering from aphasia @xcite , autism @xcite , or dyslexia @xcite . moreover, text simplification can be applied as a pre-processing step in other downstream nlp tasks such as parsing @xcite , information extraction @xcite , text summarization @xcite and machine translation @xcite . two types of simplification can be defined based on the source text: sentence simplification and document simplification @xcite . sentence simplification can be applied to texts with several sentences, one at a time, meaning that the number of sentences in the input and output would be the same. conversely, document simplification can reduce the number of sentences in the output text. currently, text simplification research has been more focused on sentence simplification @xcite . the most commonly used datasets for text simplification such as wikilarge @xcite , turk-corpus @xcite , and newsela @xcite are originally designed for sentence simplification. however, various applications in the real world require document-level simplification rather than sentence-level processing. this is due to the need to understand the main points of several sentences at once and rewrite them in a simplified vocabulary and grammar structure without respecting a number of sentences. thus, document-level text simplification may have more applications than text simplification at the sentence level. in this paper, we concentrate on document-level text simplification. the main contributions of our work include: the remainder of the paper is structured as follows: section 2 presents related work on text simplification, text summarization as well as multistage generation, which all highlight the principles of our model. in section 3, we present our novel architecture. then, section 4 presents our dataset descriptions and preprocessing steps. section 5 includes the set of experiments. in section 5.3, we combine insights from our study to obtain state-ofthe-art results on two document-level benchmarks. finally, we provide an ablation study on keyword prompt and embedding similarity loss in section 6 and the human evaluation of different models' generations in section 7. ## related work since our proposed model combines a summarizer and a simplifier modules (as we describe in section 3), we present the related work by focusing on both text simplification and text summarization tasks. ## method we introduce a new model for document-level text simplification consisting of two main components: a summarizer transformer and a simplifier transformer, which jointly aim to address the documentlevel simplification task trained in an end-to-end fashion. the motivation behind such a framework is that the document-level simplification @xcite task requires retaining the primary information from the original text (where a summarization model can be useful) while making text comprehension easier (where a simplification model can help). figure 1 demonstrates the workflow of our model. the first stage is the pre-trained summarizer. then, the output from the summarizer without tokenizer's decoding feeds into the second stage -a pre-trained sentence-to-sentence simplification transformer. this enables end-to-end training for our model. if we retokenize the decoded sentence after the summarizer step, gradients are restricted from flowing through both models during the training. first summarizing text and then simplifying it makes intuitive sense due to the fact that using a summarizer at the second stage may result in rewriting a simplified text in a complex language. we also observed this issue experimentally and therefore only proceed with the current order. furthermore, existing datasets on both the text summarization task and sentence-level simplification task indicate that we can fine-tune each module on the corresponding task. ## datasets most of the widely used datasets, such as wiki-large @xcite , turkcorpus @xcite and newsela @xcite , are designed for sentence-level text simplification and are not applicable to our document-level text simplification task. fortunately, @xcite has already adjusted the pre-processed dataset d-wikipedia for the document-to-document simplification task. however, the dataset requires additional preprocessing since there exist noisy samples with a lot of mismatches in the information presented in the source and target pairs (see section 4.1 for a more detailed discussion). the second dataset for document-level simplification is text articles from wikipedia created by @xcite , which we refer to as wiki-doc. it contains 59,749 samples which we split 8:1:1 into training (47,799 samples), validation (5,975 samples), and test (5,975 samples) sets. the wiki-doc dataset contains unaligned pairs, texts with a length greater than 3,000 tokens, and pairs where the simple text is longer than a complex one. these observations motivate several pre-processing steps described in the next section. ## ablation study given that t5 is pre-trained on a mixture of supervised and unsupervised tasks, as well as making relatively lower computational demands than bart, in the following experiments with keyword prompt and embedding similarity, we only demonstrate the performance of the t5-based variant of simsum. ## human evaluation in addition to the automatic evaluation, we performed a human evaluation of the outputs from different models. we run the assessment on 50 randomly selected samples from each dataset, thus 100 in total. we recruited two expert human evaluators to independently evaluate the generated texts from seven models. following (sheang and saggion, 2021), we select three aspects to define our evaluation criteria: (1) simplicity (s): is the output simpler than the original document?, (2) correctness (c): does the output have factual errors compared to the original document?, and (3) fluency (f): is the output grammatically correct and wellformed? we chose the binary evaluation system to decrease the bias in the 5-point evaluation system. d-wikipedia dataset. simsum with keyword prompt shows the highest values on simplicity and fluency. although bart presents a better capacity to preserve the information of original texts, its simplification performance is much worse (-0.38) than simsum. wiki-doc dataset. simsum (with bartbackbone) shows the best results in correctness and fluency. after adding the keywords, simsum's simplification power improves. at the same time, bart outperforms other baselines in terms of all three criteria. ## discussion in this section, we discuss three main points that we observed as a result of our experiments: ## conclusions and future work in this paper, we propose simsum, a new model for document-level text simplification. we demonstrate that simsum sets a new state of the art on document simplification outperforming the previously competitive muss baseline in terms of sari and d-sari scores. we also release cleaned versions of two existing large-scale datasets for text simplification. through extensive experiments, we show that keyword prompt and emebedding similarity are beneficial and have an impact on en-hancing the model's performance. finally, we conducted a human evaluation showing that simsum's quantitative performance advantage translates into better output simplicity, correctness, and fluency. in the future, we plan to investigate guiding the generation process by various prompting techniques, including extensions of the keybert model, entities, dynamic prompts, and methods such as chain of thought for simplification.