GRS : Combining Generation and Revision in Unsupervised Sentence Simplification (2022)

Abstract: We propose GRS: an unsupervised approach to sentence simplification that combines text generation and text revision. We start with an iterative framework in which an input sentence is revised using explicit edit operations, and add paraphrasing as a new edit operation. This allows us to combine the advantages of generative and revision-based approaches: paraphrasing captures complex edit operations, and the use of explicit edit operations in an iterative manner provides controllability and interpretability. We demonstrate these advantages of GRS compared to existing methods on the Newsela and ASSET datasets.

## introduction text simplification is the task of reducing the complexity and improving the readability of text while preserving its meaning. this is beneficial for persons with reading disabilities @xcite , non-native speakers, people with low literacy, and children. furthermore, other nlp tasks can use simplification as a pre-processing step, such as summarization @xcite , parsing @xcite , and machine translation @xcite . sentence simplification models can be categorized into generative and revision-based. generative approaches produce a simple sentence from a complex sentence in one step, in an auto-regressive way @xcite @xcite @xcite . revision-based methods iteratively edit a given sentence using a sequence of edit operations such as word deletion @xcite @xcite . while generative models learn complex edit operations implicitly from data, the explicit edit operations performed by revision-based approaches can provide more control and interpretability. simplification methods can also be categorized as supervised or unsupervised. supervised methods tend to have better performance, but require aligned complex-simple sentence pairs for training @xcite @xcite @xcite . unsupervised methods do not need such training data but do not perform as well @xcite @xcite . we propose grs: a new approach to bridge the gap between generative and revision-based methods for unsupervised sentence simplification. the insight is to introduce paraphrasing as an edit operation within an iterative revision-based framework. for paraphrasing, we use a fine-tuned bart model @xcite with lexicallyconstrained decoding @xcite @xcite . this decoding technique allows us to select words from the initial sentence that must be changed in the paraphrased sentence (otherwise, paraphrasing an entire sentence reduces to a pure generative model). to avoid the computational overhead of repeatedly performing constraint-based decoding using various combinations of words to paraphrase, grs includes a complex component detector to identify the most appropriate words to paraphrase. the code is available at https://github.com/imohammad12/grs . grs is unsupervised in the sense that it does not require aligned complex-simple sentence pairs, but it uses supervised models. the paraphrasing model requires paraphrasing corpora, and the complex component detector requires two unlabeled corpora, one containing more complex sentences than the other. however, collecting paraphrasing data and unaligned simplification data is simpler than collecting aligned complex-simple pairs. ## related work early work on simplification relied on rules, e.g., to split or shorten long sentences @xcite @xcite . later work treated simplification as a monolingual phrase-based machine translation (mt) task @xcite , with syntactic information added, such as constituency trees @xcite . recent work, reviewed below, leverages neural models in a generative and revision-based manner. supervised generative methods employ seq2seq models to learn simplification operations from aligned complex-simple sentence pairs @xcite . building on a seq2seq model, @xcite used reinforcement learning to optimize a reward based on simplicity, fluency and relevance. recent methods build on transformer @xcite models, by integrating external databases containing simplification rules @xcite , using an additional loss function to generate diverse outputs @xcite , combining syntactic rules @xcite , and conditioning on length and syntactic and lexical complexity features @xcite . unsupervised generative methods rely on non-aligned complex and simple corpora. zhao et al. (2020) controllable generative methods produce outputs at specified grade levels @xcite , or apply syntactic or lexical constraints on the generated sentences @xcite . however, these models do not provide any insights into the simplification process. supervised revision-based methods use complex-simple sentence pairs to learn where to apply edit operations. @xcite use keep, replace, and delete operations. some recent work used iterative non-autoregressive models to edit sentences by either predicting token-level edit-operations @xcite or using a fixed pipeline of edit operations @xcite . @xcite proposed a hybrid method with explicit edit operations in an end-toend generative model. unsupervised revision-based methods such as @xcite apply a pipeline of edit operations in a fixed order. @xcite presented an unsupervised revision-based approach by modelling text simplification as an unsupervised search problem. while grs also uses a revisionbased framework and an unsupervised search strategy, we integrate a generative paraphrasing model into the framework to leverage the strengths of both text generation and text revision approaches. ## experiments this section discusses the experimental setup (sections 4.1 through 4.4), a comparison of grs with existing approaches (section 4.5), a controllability study (section 4.6), an evaluation of the complex component detector (section 4.7), an analysis of the simplification search (section 4.8), and a human evaluation (section 4.9). ## conclusion we proposed grs, a controllable and interpretable method for unsupervised text simplification that bridges the gap between previous unsupervised generative and revision-based approaches. we combined the two approaches by incorporating an explicit paraphrasing edit operation into an iterative simplification search algorithm. empirically, we showed that grs has the advantages of both approaches. grs outperformed state-of-the-art unsupervised methods on the newsela dataset and reduced the gap between generative and revisionbased unsupervised models on the asset dataset.