Iterative Edit-Based Unsupervised Sentence Simplification (2020)

Abstract: We present a novel iterative, edit-based approach to unsupervised sentence simplification. Our model is guided by a scoring function involving fluency, simplicity, and meaning preservation. Then, we iteratively perform word and phrase-level edits on the complex sentence. Compared with previous approaches, our model does not require a parallel training set, but is more controllable and interpretable. Experiments on Newsela and WikiLarge datasets show that our approach is nearly as effective as state-of-the-art supervised approaches.

## introduction sentence simplification is the task of rewriting text to make it easier to read, while preserving its main meaning and important information. sentence simplification is relevant in various real-world and downstream applications. for instance, it can benefit people with autism @xcite , dyslexia @xcite , and low-literacy skills @xcite . it can also serve as a preprocessing step to improve parsers @xcite and summarization systems @xcite . recent efforts in sentence simplification have been influenced by the success of machine translation. in fact, the simplification task is often treated as monolingual translation, where a complex sentence is translated to a simple one. such simplification systems are typically trained in a supervised way by either phrase-based machine translation @xcite @xcite or neural machine translation (nmt, @xcite @xcite . recently, sequence-to-sequence (seq2seq)-based nmt systems are shown to be more successful and serve as the state of the art. however, supervised seq2seq models have two shortcomings. first, they give little insight into the simplification operations, and provide little control or adaptability to different aspects of simplification (e.g., lexical vs. syntactical simplification). second, they require a large number of complexsimple aligned sentence pairs, which in turn require considerable human effort to obtain. in previous work, researchers have addressed some of the above issues. for example, @xcite and @xcite explicitly model simplification operators such as word insertion and deletion. although these approaches are more controllable and interpretable than standard seq2seq models, they still require large volumes of aligned data to learn these operations. to deal with the second issue, @xcite recently proposed an unsupervised neural text simplification approach based on the paradigm of style transfer. however, their model is hard to interpret and control, like other neural network-based models. @xcite attempted to address both issues using a pipeline of lexical substitution, sentence splitting, and word/phrase deletion. however, these operations can only be executed in a fixed order. in this paper, we propose an iterative, editbased unsupervised sentence simplification approach, motivated by the shortcomings of existing work. we first design a scoring function that measures the quality of a candidate sentence based on the key characteristics of the simplification task, namely, fluency, simplicity, and meaning preservation. then, we generate simplified candidate sentences by iteratively editing the given complex sentence using three simplification operations (lexical simplification, phrase extraction, deletion and reordering). our model seeks the best simplified candidate sentence according to the scoring function. compared with @xcite , the order of our simplification operations is not fixed and is decided by the model. figure 1 illustrates an example in which our model first chooses to delete a sentence fragment, followed by reordering the remaining fragments and replacing a word with a simpler synonym. we evaluate our approach on the newsela @xcite and wikilarge @xcite corpora. experiments show that our approach outperforms previous unsupervised methods and even performs competitively with state-ofthe-art supervised ones, in both automatic metrics and human evaluations. we also demonstrate the interpretability and controllability of our approach, even without parallel training data. ## related work early work used handcrafted rules for text simplification, at both the syntactic level @xcite and the lexicon level @xcite . later, researchers adopted machine learning methods for text simplification, modeling it as monolingual phrase-based machine translation @xcite . further, syntactic information was also considered in the pbmt framework, for example, constituency trees @xcite and dependency trees @xcite . @xcite performed probabilistic sentence splitting and deletion, followed by mt-based paraphrasing. @xcite employed neural machine translation (nmt) for text simplification, using a sequence-to-sequence (seq2seq) model @xcite . @xcite used reinforcement learning to optimize a reward based on simplicity, fluency, and relevance. @xcite integrated the transformer architecture and paraphrasing rules to guide simplification learning. @xcite produced diverse simplifications by generating and re-ranking candidates by fluency, adequacy, and simplicity. @xcite showed that simplification benefits from multi-task learning with paraphrase and entailment generation. martin et al. (2019) 2019 ) employed a similar approach but in an end-to-end trainable manner with neural networks. however, these approaches are supervised and require large volumes of parallel training data; also, their edits are only at the word level. by contrast, our method works at both word and phrase levels in an unsupervised manner. for unsupervised sentence simplification, @xcite adopted style-transfer techniques, using adversarial and denoising auxiliary losses for content reduction and lexical simplification. however, their model is based on a seq2seq network, which is less interpretable and controllable. they cannot perform syntactic simplification since syntax typically does not change in style-transfer tasks. @xcite built a pipeline-based unsupervised framework with lexical simplification, sentence splitting, and phrase deletion. however, these operations are separate components in the pipeline, and can only be executed in a fixed order. unsupervised edit-based approaches have recently been explored for natural language generation tasks, such as style transfer, paraphrasing, and sentence error correction. @xcite proposed edit-based style transfer without parallel supervision. they replaced style-specific phrases with those in the target style, which are retrieved from the training corpus. @xcite used metropolis-hastings sampling for constrained sentence generation. in this paper, we model text generation as a search algorithm, and design search objective and search actions specifically for text simplification. concurrent work further shows the success of search-based unsupervised text generation for paraphrasing @xcite and summa-rization @xcite . ## model in this section, we first provide an overview of our approach, followed by a detailed description of each component, namely, the scoring function, the edit operations, and the stopping criteria. ## conclusion we proposed an iterative, edit-based approach to text simplification. our approach works in an unsupervised manner that does not require a parallel corpus for training. in future work, we plan to add paraphrase generation to generate diverse simple sentences.