Context-Aware Document Simplification (2023)

Abstract: To date, most work on text simplification has focused on sentence-level inputs. Early attempts at document simplification merely applied these approaches iteratively over the sentences of a document. However, this fails to coherently preserve the discourse structure, leading to suboptimal output quality. Recently, strategies from controllable simplification have been leveraged to achieve state-of-the-art results on document simplification by first generating a document-level plan (a sequence of sentence-level simplification operations) and using this plan to guide sentence-level simplification downstream. However, this is still limited in that the simplification model has no direct access to the local inter-sentence document context, likely having a negative impact on surface realisation. We explore various systems that use document context within the simplification process itself, either by iterating over larger text units or by extending the system architecture to attend over a high-level representation of document context. In doing so, we achieve state-of-the-art performance on the document simplification task, even when not relying on plan-guidance. Further, we investigate the performance and efficiency tradeoffs of system variants and make suggestions of when each should be preferred.

## introduction text simplification transforms a given text into a simpler version of itself that can be understood by a wider audience, while preserving the same core meaning @xcite . it has also proven useful as a preprocessing step for downstream nlp tasks such as machine translation @xcite @xcite and relation extraction @xcite . most previous work has focused on sentencelevel simplification by training neural models on complex/simple sentence pairs under the assumption that they will learn to perform required operations (e.g. sentence splitting, lexical substitution or syntactic rephrasing) implicitly from the training data @xcite @xcite . however, the imbalanced representation of simplification operations throughout popular datasets, and the overly-conservative models arising from their use, have led to attempts at controllable simplification to achieve more variation and diversity in output texts @xcite @xcite . recently, strategies from controllable simplification have been leveraged to achieve state-of-the-art results on the document simplification task @xcite . specifically, by using a planning model capable of considering the sentences surrounding a complex sentence, a sentence-level simplification model can be guided such that the structure of the resulting document remains more coherent. despite this success, the sentence simplification model still has no direct access to document context which we believe limits the extent to which it can accurately produce simplified sentences that are consistent with the larger document. as such, we propose various systems that allow access to some representation of surrounding content within the simplification module, while still allowing for the possibility of plan-guidance. we show that in doing so, we are able to achieve stateof-the-art document simplification performance on the newsela dataset, even without relying on a generated plan. further, we investigate the performance and efficiency tradeoffs of various system variants. 1 our key contributions are (i) a detailed investigation of how document context, input text and simplification plans impact document-level simplifica-tion and (ii) several state of the art models for document simplification. we show in particular that document level simplification is improved by combining a representation of the local context surrounding complex sentences with a simplification plan indicating how complex sentences should be simplified (whether they should be deleted, rephrased, split or copied). recently, the latter approach was leveraged for document simplification where it obtained state-ofthe-art performance @xcite . here, a sequence of sentence-level operations is predicted for an entire document and then used to iteratively condition a sentence-level simplification model. the system considers both local (token representation of the sentence) and global document context (sequence of sentence-level encodings) when predicting an operation for a given sentence. document-level simplification. initial attempts at document simplification simply applied sentence simplification methods iteratively over documents @xcite @xcite . however, it was noted this alone is insufficient for performing certain operations, often leading to poor discourse coherence in the output @xcite . various sub-problems of document simplification have been approached in isolation, such as sentence deletion @xcite , insertion @xcite , and reordering @xcite . @xcite took a holistic approach by iteratively applying a sentencelevel model, but with additional encoders to embed the two preceding and following sentences, which are used as additional input during generation. however, this was unable to outperform baselines. recently, @xcite achieved stateof-the-art performance by producing a document simplification framework capable of performing all of the most common operations. specifically, they use both high-level document context and sentencelevel features to generate a plan specifying which operations to be performed on each sentence in a given document, which is then used to condition a sentence simplification model. ## problem formulation the goal of text simplification is to generate a text s that simplifies an input text c. in the documentlevel case, @xmath0. . . c n is a sequence of complex sentences and @xmath1. . . s m is a sequence of simple sentences. @xcite further decompose this task into a two-stage process wherein a generated plan conditions the simplification: although the use of such plans has shown improved results, little attention has been given to how the generation stage itself can be modified to improve document-level simplification. in this work, we investigate whether further changes can be made to simplification models in order to make better use of high-level plans, or alternatively, whether it is possible to forego the planning stage entirely by incorporating high-level document context into the generative model directly. terminology and notations. we use the following terminology and notational conventions: ## data for all experiments, we use newsela-auto @xcite which is currently the highest-quality document-level simplification dataset available. it consists of 1,130 english news articles from the original newsela @xcite dataset which are each manually rewritten at five different levels of simplification, corresponding to discrete reading levels (0-4) of increasing simplicity. it also includes both sentence and paragraph alignments for each document pair. like previous works, for all our models we prepend a control-token to the input specifying the target document reading level. we use the same filtered version of newselaauto used in @xcite , along with the same train/validation/test splits to allow for model comparison. this also includes plan labels, consisting of an operation (copy, rephrase, split, or delete) assigned to each sentence pair. statistics of this data can be seen in ## models we distinguish three model categories: (i) models whose sole input is text and which simplify a document either by iterating over its sentences/paragraphs or by handling the entire document as a single input; (ii) models that take both a complex sentence and some representation of its document context as input and simplify a document by iterating over its sentences; and (iii) models that are guided by a plan via control-tokens denoting sentence-level simplification operations prepended to the input sequence. these are illustrated in ## evaluation 6.1 automatic evaluation text simplification is often evaluated on the basis of 3 criteria: adequacy (or meaning preservation), fluency, and simplicity. for automatic evaluation, we use bartscore @xcite and @xcite as analogs for both adequacy and fluency. both are reference-based metrics that have previously been used for document simplification as well as other text generation tasks. for assessing simplicity, we use both the flesch-kincaid grade level (fkgl) and sari @xcite . fkgl is a document-level metric of text readability that has the highest correlation with human judgements @xcite , while sari is a simplification metric that has become a staple in the sentence-level simplification literature. we use easse @xcite to calculate both of these. at test time we generate sequences using beam search with a beam size of 5 and a maximum length of 1024 tokens. ## results and discussion results are shown in ## model efficiency there are various others factors to consider when comparing systems, beyond their raw performance. for instance, the size of the model(s) and how much time/resources are required for each to perform inference are important practical considerations that must be made when selecting a model for realworld use. as such, we compare each system based on the time taken to simplify the test set and their total parameter counts. in our case, any system that uses a plan requires a second model, approximately doubling the number of parameters that must be loaded. these pipeline setups also naturally add to overall inference time. further, both plan pipelines and conbart make use of dynamic context, which imposes an autoregressive bottleneck on the simplification of individual documents. because of the linearly scaling attention mechanism, longformer-based models are the fastest of proposed systems. because of this and its overall high performance, we recommend led para in situations where time or computing resources are at all limited. alternatively, ô → conbart offers a good compromise that provides the high performance of a plan-guided system while mitigating further increases to inference time. this is because it uses the same autoregressive protocol as the planner and can therefore share the generated context representations. all inference processes were run on a single nvidia a40 gpu, using a batch size of 16, 32 cpu workers for data loading, and a beam size of 5 for generation. appendix f provides details on the specific algorithm used to handle dynamic context generation for appropriate models. ## conclusion we develop a range of document simplification models that are able to use different combinations of text, context, and simplification plans as input, with several models outperforming the previous state-of-the-art both on automatic metrics and according to human judgements. our results show that a high-level representation of the document can be useful for low-level surface realisation as well as global planning. further, simplification models with access to local document context, either by working at the paragraph level or handling an additional input representation, lead to better meaning preservation than those that operate on individual sentences. we conclude by evaluating the model efficiency of each system and making recommendations for their selection under different circumstances. paragraph-level human evaluation in order to reduce complexity, our human evaluation was performed on paragraphs rather than full documents. as a result, there is a potential limit to the accuracy of human judgements when certain discourse phenomena are present. for example, important information may be excluded from a specific output paragraph (therefore prompting a low adequacy rating), but this could actually be present in a different part of the true simplified document. monolinguality this study focused entirely on simplification for english-language documents. reproducing the proposed systems for use on other languages would require dedicated datasets of similar scale, along with sentence/paragraph alignments and operation labels (which likely do not currently exist). further, the nature of simplification in other languages may differ quite a lot from english with respect to the types of operations that are performed, potentially reducing the suitability of the proposed framework.