Document-Level Planning for Text Simplification (2023)

Abstract: Most existing work on text simplification is limited to sentence-level inputs, with attempts to iteratively apply these approaches to document-level simplification failing to coherently preserve the discourse structure of the document. We hypothesise that by providing a high-level view of the target document, a simplification plan might help to guide generation. Building upon previous work on controlled, sentence-level simplification, we view a plan as a sequence of labels, each describing one of four sentence-level simplification operations (copy, rephrase, split, or delete). We propose a planning model that labels each sentence in the input document while considering both its context (a window of surrounding sentences) and its internal structure (a token-level representation). Experiments on two simplification benchmarks (Newsela-auto and Wiki-auto) show that our model outperforms strong baselines both on the planning task and when used to guide document-level simplification models.

## introduction text simplification aims to transform a given text into a simpler version of itself that preserves the core meaning such that it can be better understood by a wider audience @xcite . simplification has also been shown to be a useful preprocessing step for downstream nlp tasks such as relation extraction @xcite and machine translation @xcite @xcite . previous research has mostly considered the simplification of isolated sentences. much work has focused on training a statistical or a neural model on pairs of complex and simplified sentences assuming that such models will learn to perform simplification operations (e.g. sentence splitting, lexical simplification or syntactic rephrasing) implic-itly from the inductive bias present in the training data @xcite @xcite . however, because the training data is obtained using distant supervision techniques and is often imbalanced in terms of simplification operations (many of which occur infrequently @xcite ), system outputs have been found to be overly conservative, often making no changes or being limited to the paraphrasing of short word sequences @xcite . in addition, these systems provide limited capacity for controllability and are unable to express alternative variants of the simplified text @xcite . in response, controllable simplification systems have been proposed which either constrain attributes of the output (length, amount of paraphrasing, lexical and syntactic complexity) @xcite or explicitly specify which simplification operation to perform @xcite @xcite @xcite . to guide the simplification of full documents, we combine the power of data-driven neural generative models with strategies from controllable simplification. our hypothesis is that document-level simplification can be facilitated by a plan specifying how each complex input sentence should be transformed to yield a simplified version of that document -should it be copied, deleted, split or rewritten? we make the following contributions: we present a model for predicting document simplification plans which leverages both the context of sentences and their internal structure (the words they consists of). we create the data necessary to train this model by labelling complex sentences in simplification corpora with the simplification operation that relates it to the corresponding simplified sentence. we compare our planning model with several alternative neural architectures and we briefly examine the impact of planning on document simplification. experiments on two simplification benchmarks (newsela-auto and wiki-auto) show that our model outperforms strong baselines both on the planning task and when used to guide document-level simplification models. foot_0 2 related work document-level simplification. there is limited existing work on document-level text simplification. early attempts largely applied sentencelevel techniques iteratively over a document @xcite . however, this is generally viewed as insufficient for certain operations and maintaining the discourse coherence of the document @xcite . there are several works that address subproblems of simplification that only consider a limited set of operations, like paraphrasing and sentence re-ordering @xcite , insertion @xcite or deletion @xcite . others fully address simplification but only extend inputs to the level of paragraphs without clearly differentiating the problem from the sentence-level @xcite . recently, @xcite proposed a sentencelevel model (suc) that uses an encoding of surrounding sentences as context information to influence the simplification. they use two extra encoders to build a representation of the two preceding and two following sentences, which are attended over in their encoder-decoder generative model. however, when applied to the documentlevel task, their system was unable to outperform any baseline systems @xcite . ## problem formulation let c denote an english language document. the aim of document-level simplification is to produce a text s that simplifies the input document c. as a plan can provide a high-level view of a document, we hypothesize that a document-level simplification model that is based on a plan specifying a simplification operation for each input sentence should fare better than a simplification model that directly simplifies an entire document. we therefore decompose simplification into a two-stage generation process: we consider three simplification operations proposed in previous work on sentence simplification (copy, rephrase, and split) to which we add delete, an operation that is needed to account for the fact that, contrary to sentence simplification, document-level simplification can require for a sentence present in the input document to be excluded from the resulting simplified document. given the input document c, the first-stage model aims to predict the sequence of simplification operations p that should be applied to each individual sentence in that document. the secondstage model generates the output simplified document s conditioned on the input document c and its accompanying simplification plan p . in this work, we focus on the planning stage, comparing different architectures and demonstrating the impact of planning on three possible document-level simplification models. we leave the exploration of alternative, more complex architectures for the simplification stage to future work. ## data in this section we introduce the datasets used, explain how annotation is performed for each complex sentence and describe other preprocessing steps. dataset. for all experiments, we utilise wikiauto and newsela-auto @xcite , two datasets of english documents paired with their simplification. these datasets were derived from wikilarge @xcite and newsela @xcite by aligning the input document with the output simplification at both the sentence and the paragraph level. wikilarge gathers three simplification datasets which were automatically-collated from english wikipedia and wikipedia simple @xcite ; woodsend and lapata, 2011b; @xcite . newsela consists of news articles, each manually rewritten at five different levels of simplification, corresponding to discrete reading levels (0-4) of increasingly simplicity. aligned pairs are created by pairing every article version with each other version corresponding to a higher reading level. because of this, there can be up to four aligned document pairs that contain the same document as either the input or the output. the types of operations present in different reading level pairings differs significantly, with adjacent level transitions being extremely conservative (no instances of deletion throughout entire dataset). to mitigate any issues arising from this, all models we train with newsela-auto receive a control-token at the start of the input which specifies the target reading level. we do not use the d-wikipedia dataset from @xcite as it does not contain sentence/paragraph alignments and is poorly formatted. in particular, all text is lower-cased and pretokenized in a way that makes it difficult to accurately parse sentences. there are also regular formatting issues at points where references exist in the source article. annotating complex sentences. using the pairs (c i , s j ) of complex and simplified sentences available in wiki-auto and newsela-auto, we heuristically assign a silver simplification opera-tion label to each complex sentence c i in these two datasets as follows: delete: c i is not aligned to any s j . copy: c i is aligned to a single s j with a levenshtein similarity above 0.92. rephrase: c i is aligned to a single s j with a levenshtein similarity below 0.92. split: c i is aligned to multiple s j s. preprocessing. wiki-auto contains many document pairs with wildly different sizes. we therefore clip all complex documents after the last aligned paragraph. many simple articles resemble a summarization, rather than a simplification of the complex article (lots of deletion, often consisting of about one sentence from each paragraph in c). because of this, we also remove documents where more than 50% of aligned sentences are labelled as delete. finally, we remove all articles that exceed 1024 tokens (so that we can fit them into a bart baseline generative model). for newsela-auto, article pairs are much more even in length as they are manually created to be gradual, direct simplifications of each other. we perform the same length-based filtering to exclude documents that will not fit into a baseline generative model. train/dev/test split. for both datasets we use a train/validation/test split of 92.5/2.5/5. this is applied at the document-level so that sentences from the same document will not exist across different sets. for newsela, this means that all reading level versions of a single article will exist within the same set. ## planning we present our model and four alternative models we explored for comparison. training details are given in appendix a. ## simplification to assess whether document plans can help improve simplification models, we experiment with two simple document-level simplification models and compare their performance with and without a preceding planning step. ## conclusion in this paper we present an approach to document simplification that decomposes the task into a twostage process of planning and generation. we propose a planning system that is able to take document context and structure into account to produce a coherent high-level simplification plan. by using this plan to guide a sentence-level simplification model, we are able to outperform end-to-end systems in terms of both meaning preservation and simplicity. we leave for future work the development of dedicated simplification models that can leverage a document-level plan while also considering contextual information directly during generation.