T ext S implification by T agging (2021)

Abstract: Edit-based approaches have recently shown promising results on multiple monolingual sequence transduction tasks. In contrast to conventional sequence-to-sequence (Seq2Seq) models, which learn to generate text from scratch as they are trained on parallel corpora, these methods have proven to be much more effective since they are able to learn to make fast and accurate transformations while leveraging powerful pre-trained language models. Inspired by these ideas, we present TST, a simple and efficient Text Simplification system based on sequence Tagging, leveraging pre-trained Transformer-based encoders. Our system makes simplistic data augmentations and tweaks in training and inference on a pre-existing system, which makes it less reliant on large amounts of parallel training data, provides more control over the outputs and enables faster inference speeds. Our best model achieves near state-of-the-art performance on benchmark test datasets for the task. Since it is fully non-autoregressive, it achieves faster inference speeds by over 11 times than the current state-of-the-art text simplification system.

## introduction text simplification is the task of rewriting text into a form that is easier to read and understand while preserving its underlying meaning and information. it has been shown to be valuable in providing assistance in terms of readability and understandability to children @xcite , people with language disabilities like aphasia @xcite @xcite , dyslexia @xcite , or autism @xcite ; non-native english speakers @xcite @xcite , and people with low literacy skills or reading ages @xcite @xcite . moreover, it has also been successfully leveraged as a pre-processing step to improve the performance of various nlp tasks such as parsing @xcite , summarization @xcite , semantic role labeling @xcite and machine translation @xcite @xcite . evolving from the approaches ranging from building hand-crafted rules @xcite to syntactic and lexical simplification via synonyms and paraphrases @xcite @xcite , the task has gained popularity as a monolingual machine translation (mt) problem, where the system learns to "translate" a given complex sentence to its simplified form. initially, statistical phrase-based (smt) and syntactic-based machine translation (sbmt) techniques @xcite @xcite @xcite @xcite were successfully applied as a way to learn simplification rewrites implicitly from complexsimple sentence pairs, often in combination with hand-crafted rules or features. more recently, several neural machine translation-based (nmt) systems have been developed with promising results @xcite @xcite , and their successful application to text simplification, either in combination with smt or other data-driven approaches @xcite ; or strictly as neural models @xcite @xcite @xcite @xcite @xcite , has emerged as the state-of-the-art. human editors perform several rewriting transformations in order to simplify a sentence, such as lexical paraphrasing, changing the syntactic structure, or removing superfluous information from the sentence @xcite @xcite . therefore, even though nmt-based sequence-to-sequence (seq2seq) approaches offer a generic framework for modeling almost any kind of sequence transduction, target texts in these approaches are typically generated from scratch -a process which can be unnecessary for monolingual editing tasks such as text simplification, owing to these aforementioned transformations. moreover, these approaches have a few shortcomings that make them inconvenient for real-world deployment. first, they give limited insight into the simplification operations and provide little control or adaptability to different aspects of simplification (e.g., lexical vs. syntactical simplification). this inhibits interpretability and explainability, which is crucial for real-world settings. second, they are not sample-efficient and require a large number of complex-simple aligned sentence pairs for training, which requires considerable human effort to obtain. third, these models typically employ an autoregressive decoder, i.e., output texts are generated in a sequential, non-parallel fashion, and hence, are generally characterized by slow inference speeds. based on the aforementioned observations and issues, text-editing approaches have recently regained significant interest @xcite @xcite @xcite . typically, the set of edit operations in such tasks is fixed and predefined ahead of time, which on one hand limits the flexibility of the model to reconstruct arbitrary output texts from their inputs, but on the other, leads to higher sample-efficiency as the limited set of allowed operations significantly reduces the search space @xcite . this pattern is especially true for monolingual settings where input and output texts have relatively high degrees of overlap. in such cases, a natural approach is to cast the task of conditional text generation into a text-editing task, where the model learns to reconstruct target texts by applying a set of edit operations to the inputs. we leverage this insight in our work, and simplify the task from sequence generation or editing, going a step further, to formulate it as a sequence tagging task. in addition to being sample efficient, thanks to the separation of various edit operations in the form of tags, the system has better interpretability and explainability. finally, since for sequence tagging we don't need to predict tokens one-by-one as in autoregressive decoders, the inference is naturally parallelizable and therefore runs many times faster. following from the success of the aforementioned monolingual edit-tag based systems, we propose to leverage the current state-of-the-art model for grammatical error correction by @xcite and adapt it to the task of text simplification. in summary, we make the following contributions: text simplification system by adapting the gector model to text simplification, leveraging transformer-based encoders trained on large amounts of humanannotated and synthetic data. foot_0 empirical results demonstrate that our system achieves near state-of-the-art performance on benchmark test datasets in terms of readability and simplification metrics. recent text editing works have shown promising results of reformulating multiple monolingual sequence transduction tasks into sequence tagging tasks compared to the conventional seq2seq sequence generation formulation. this observation is especially true for tasks where input and output sequences have a large overlap. generally, these works try to simplify monolingual sequence transduction by explicitly modeling edit operations such as keep, add/insert and delete. alva-manchego et al. (2017) proposed the first such formulation, employing a bilstm to predict edit encoder roberta-base edit-detection feed-forward layer edit-classification feed-forward layer text simplification by tagging (tst) softmax softmax edit tags input sentence post-process output sentence repeat t times labels sequentially. our model for sentence simplification does not rely on external simplification rules nor alignment tools. @xcite proposed an approach applied only to character deletion and insertion and was based on simple patterns. lasertagger @xcite combines a bert encoder with an autoregressive transformer decoder to similarly predict the aforementioned three main edit operations for several text editing tasks. in contrast, in our system, the decoder is a softmax layer. similarly, editnts @xcite and pie @xcite predict edit labels, developed specifically for text simplification and gec, respectively. while editnts employs an autoregressive encoder-decoder based neural programmer-interpreter model, pie differs from our work because of our custom edit transformations and incorporation of a pre-trained transformer encoder for sequence tagging. levenshtein transformer @xcite , an autoregressive model that performs text editing by executing a sequence of deletion and insertion actions, is another recent work along similar lines. more recently, @xcite proposed felix -a text-editing-based system for multiple generation tasks, splitting the text-editing task into two subtasks: tagging and insertion. their tagging model employs a pointer mechanism, while the insertion model is based on a masked language model. ## system description following recent works such as @xcite ; @xcite ; @xcite , who leveraged similar frameworks for different text editing problems such as gec, sentence fusion, and abstractive summarization, we formulate the task of text simplification as a tagging problem. specifically, our system is based on gector @xcite , an iterative sequencetagging system that works by predicting token-level edit operations, originally developed for grammatical error correction (gec). we adapt the gector framework for the task of text simplification, with minimal modifications to the original architecture. our system consists of three main parts: (a) defining the custom transformations (token-level edittags), (b) performing iterative sequence tagging to convert target sequences to tag sequences, (c) fine-tuning of pre-trained transformers to predict the tag sequences. each of these components are described below. ## ablation study in this section, we present ablation experiments for each of the enhancements described in section 4, and applied to tst-base to obtain tst-final. the results of these experiments ( ## conclusion this paper introduces tst, a novel approach to text simplification, by reformulating the task into a much simpler one of sequence tagging. we build tst by adapting the gector framework for gec. we show that most of its performance gains are owed to simplistic data augmentations and tweaks in training and inference. these modifications allow us to derive maximal benefit from the already existing pre-trained transformer-based encoders on large amounts of human-annotated and synthetic data, making tst a simple, powerful, easily reusable method for monolingual editing tasks. since tst is able to progressively make simplifying edits via explicit edit-tag operations, the transformations resulting from tst are better explainable and interpretable than any other nmtbased seq2seq approaches. finally, tst is fully non-autoregressive, enabling it to perform faster inference than any other state-of-the-art text simplification methods. our empirical results demonstrate that it achieves near state-of-the-art performance on benchmark test datasets for text simplification. a major motivation in this work was to minimize changes to the original model to keep the system simple, fast, and reproducible. hence, we restricted our system to only use wikiall data and its derivatives (vs. any external data like the state-of-the-art system by @xcite ). while we did not fully beat the state-of-the-art on the task, we believe that using larger models (eg. roberta large ), ensembles, or external data will likely lead to better sari scores at the cost of speed and system complexity: ideas we plan to explore in future work. original he also completed two collections of short stories entitled the ribbajack & other curious yarns and seven strange and ghostly tales . reference he also wrote two books of short stories called , the ribbajack & other curious yarns and seven strange and ghostly tales . tst-base he also wrote two collections of short stories called the ribbajack & other curious yarns and seven strange and ghostly tales . tst-final he also wrote a series of short stories called the ribbajack & other curious yarns and seven strange and ghostly tales . original it is theoretically possible that the other editors who may have reported you , and the administrator who blocked you , are part of a conspiracy against someone half a world away they 've never met in person . reference it is theoretically possible that the other editors who may have written about you, and the officer who blocked you, are part of a bad plan against someone miles away, they 've never met face to face . tst-base it is possible that the other editors who may have reported you , and the administrator who blocked you , are part of a conspiracy against someone half a world away they ' ve never met in person . tst-final it might be that the other editors who may have sent you , and the administrator who blocked you , are part of a conspiracy against someone half a world away where they 've never met in person . original as a result , although many mosques will not enforce violations , both men and women when attending a mosque must adhere to these guidelines . reference as a result , both men and women must follow this rule when they attend a mosque , even though many mosques do not enforce these rules tst-base both men and women when going a mosque must follow these rules . tst-final both men , and women that attend mosque , must follow the law .