Target-Level Sentence Simplification as Controlled Paraphrasing (2022)

Abstract: Automatic text simplification aims to reduce the linguistic complexity of a text in order to make it easier to understand and more accessible. However, simplified texts are consumed by a diverse array of target audiences and what might be appropriately simplified for one group of readers may differ considerably for another. In this work we investigate a novel formulation of sentence simplification as paraphrasing with controlled decoding. This approach aims to alleviate the major burden of relying on large amounts of in-domain parallel training data, while at the same time allowing for modular and adaptive simplification. According to automatic metrics, our approach performs competitively against baselines that prove more difficult to adapt to the needs of different target audiences or require significant amounts of complex-simple parallel aligned data.

## introduction automatic text simplification (ats) aims to reduce the linguistic complexity of a text while preserving its meaning in order to make it easier to understand and more accessible to a wider array of potential readers @xcite . these readers might include children or adults with low literacy levels, cognitive impairments, or a lack of specialist knowledge in certain topics, as well as non-native language learners @xcite . however, the notion of exactly what constitutes 'simplified' text is highly subjective and can differ considerably between different types of readers. thus it is important to tailor solutions appropriately in order to accommodate the needs of specific target audiences. research on ats, or more specifically sentence simplification (ss), has been spurred on by performance gains in neural sequence-to-sequence (seq2seq) language generation methods @xcite , which aim to do away with complex hand-crafted rules @xcite and improve on earlier statistical approaches @xcite . however, fully supervised seq2seq ss approaches require a large amount of sentence-aligned parallel training data @xcite , which remains relatively scarce and difficult to attain. for this reason, much work has focused on certain aspects of ats such as lexical @xcite or structural simplification @xcite @xcite . others have aimed to make better use of limited parallel complex-simple training sentences by using more sample-efficient modelling techniques that aim to predict and execute in-place edit operations @xcite . meanwhile, despite considerable similarities between ss and paraphrasing, the task of reformulating a sentence while maintaining an equivalent meaning @xcite , relatively few works have aimed to exploit paraphrases to bootstrap seq2seq-based simplification @xcite . we investigate this last line of work and consider an alternative framing of ss as the task of controlled paraphrasing. we train a large-scale paraphrase model capable of producing high-quality and diverse paraphrases and combine it with future discriminators for generation (fudge) @xcite to control decoding and steer the generated paraphrase towards a specific target level for text simplification. our experiments show that this proves to be an effective approach for generating simplified sentences for different target audiences without requiring any parallel data in the form of complex-simple aligned sentence pairs. the code and model outputs from this work are made available at https://github.com/zurichnlp/ simplefudge . ## background & motivation perhaps the largest hurdle for seq2seq-based simplification is the collection of appropriately aligned complex-simple parallel data required for training robust and reliable systems @xcite . furthermore, as discussed by @xcite , ats systems should be developed to support a variety of target readers and would thus benefit from modular approaches that allow for easy customisation and adaption. recently, however, large pre-trained generation models have continued to demonstrate impressive performance when finetuned on conditional language generation tasks @xcite . along with this, there has been considerable work done on exploring ways to better control the outputs of large generative models in order to achieve certain communicative goals @xcite @xcite @xcite . we see a clear link between these recent developments and the challenges associated with ss and set out to investigate a modular approach suitable for simplifying text for different target audiences and that relaxes the need for complex-simple parallel training data. ## method given a complex source sentence, our goal is to transform it into a simplified target sequence 1 that preserves its meaning. under the traditional seq2seq framework, a target sequence @xmath0, ..., y t ′ } can be generated autoregressively as a series of conditional probabilities over the vocabulary, whereby each target token y i is conditioned on the source sentence @xmath1, ..., x t } and any preceding target tokens y 1:i-1 , to ensure that the generated target sequence is appropriately simplified, we employ fudge (yang and klein, 2021), which has been shown to be effective for various controlled generation tasks. fudge introduces a lightweight classifier b to control for a desired target attribute a during autoregressive generation with a model g. in essence, it modifies the conditional probability in equation 1 with the following bayesian factorisation: 1 since an appropriate simplified formulation may consist of multiple shorter sentences we refer to it as a sequence. p (y i |x, y 1:i-1 , a) ∝ p (a|y 1:i )p (y i |x, y 1:i-1 ). ## results & discussion we also note that the superior performance of the fully supervised method for level simp-4 is consistent with the findings from @xcite , where a similar approach proved most effective for simplifying ordinary german to a1-level german, despite it being the target level with the least amount of parallel data in both studies. for other target levels, however, where the differences between source and target are perhaps more subtle, this supervised model has a strong tendency to simply copy the input sentence. while the muss baseline produces decent simplifications according to sari and fkgl, the lower compression ratio and a higher proportion of deleted n-grams indicate that this model also tends to severely summarise the input. this information loss causes model outputs to diverge from the ground truth references and it is thus penalised heavily by bertscore. fudge's simplification operations are performed actively during decoding by modifying the generator's prediction logits at each timestep, with each decision being informed by the currently generated prefix and its potential continuations y 1:i . therefore, this approach does not enforce a transformation of the input text. this is an important and desirable feature for ss as oftentimes not all parts of a sentence need to be simplified @xcite . thus, assuming a well-trained classifier b, simplification operations should occur only when appropriate given the source sentence and the generation context. 5 finally, using fudge for ss also makes use of a single hyperparameter λ which controls the contribution from the classifier. in contrast, muss requires setting an appropriate value for each of the four control tokens to attain a suitable simplification. these are not only difficult to determine for each target level (see appendix e), but the way in which these tokens interact with each other is still unclear @xcite , making it difficult to set these values according to any underlying intuition. ## conclusion & future work we have explored a modular and adaptable approach to ss by reframing it as controlled paraphrasing. we used fudge @xcite to steer the generation of paraphrastic sequences toward different target levels. according to automatic metrics, this approach performs competitively when compared to state-of-the-art methods. in future work we aim to conduct a more detailed analysis of the model outputs in order to better understand the qualitative differences and potential shortcomings, as well as applying this method to larger textual units beyond sentences.