Controllable Sentence Simplification with a Unified Text-to-Text Transfer Transformer (2021)

Abstract: Recently, a large pre-trained language model called T5 (A Unified Text-to-Text Transfer Transformer) has achieved state-of-the-art performance in many NLP tasks. However, no study has been found using this pre-trained model on Text Simplification. Therefore in this paper, we explore the use of T5 fine-tuning on Text Simplification combining with a controllable mechanism to regulate the system outputs that can help generate adapted text for different target audiences. Our experiments show that our model achieves remarkable results with gains of between +0.69 and +1.41 over the current state-of-the-art (BART+ACCESS). We argue that using a pre-trained model such as T5, trained on several tasks with large amounts of data, can help improve Text Simplification.

## introduction text simplification (ts) can be regarded as a natural language generation task where the generated text has a reduced language complexity in both vocabulary and sentence structure while preserving its original information and meaning @xcite . its applications can be used as reading assessment tools for people with low-literacy skills such as children @xcite , and nonnative speakers @xcite , or people with cognitive disabilities such as autism @xcite , aphasia @xcite , and dyslexia @xcite . in addition, ts can also be used as a preprocessing step to improve the results of many nlp tasks, e.g., parsing @xcite , information extraction @xcite , question generation @xcite , text summarization @xcite , and machine translation @xcite . in recent years, research in ts has been mostly focused on developing models based on deep neural networks @xcite @xcite . however, and to the best of our knowledge, very few studies of transfer learning -where a model is first pre-trained on a datarich task and then fine-tuned on downstream taskshave been explored in ts. in this paper, we propose a transfer learning and controllable sentence simplification model that harnesses the power of the unified text-to-text transfer transformer (t5) pre-trained model @xcite , combining it with control tokens to provide a way to generate output that adapts to different target users. such a model can be adjusted to fit the need of different users without having to build everything from the ground up. we make the following contributions: ## model in this work, we fine-tune t5 pre-trained model with the controllable mechanism on text simplification. t5 (a unified text-to-text transfer transformer) @xcite is pre-trained on a number of supervised and unsupervised tasks such as machine translation, document summarization, question answering, classification tasks, and reading comprehension, as well as bert-style token and span masking @xcite . there are five different variants of t5 pre-trained models: t5small (5 attention modules, 60 million parameters), and t5-base (12 attention modules, 220 million parameters). due to the limited resources of colab pro, we are able to train only t5-small and t5-base. ## experiments our model is developed using the huggingface transformers library @xcite foot_1 with py-torch foot_2 and pytorch lightning foot_3 . ## ablation study in this section, we investigate the contribution of each token and different t5 pre-trained models to the performance of the system. the models in based on the results, the larger model (t5-base) performs better than the smaller one (t5-small) on both datasets (+3.06 on turkcorpus, +4.3 on as-set). it is due to the fact that larger model has more information which could generate better and more coherent text. moreover, when added control tokens, the performance increases significantly. with only one token, wordrank performs best on turkcorpus (+3.88 over t5-base) and levsim on asset (+7.43 over t5-base). using pre-trained model alone does not gain much improvement, only when combined with control tokens, the results improve by a big margin (+3.06 and +9.28 for t5-small with and without tokens), and (+5.75 and +10.89 for t5-base with and without tokens). ## conclusion in this paper, we propose a method which leverages a big pre-trained model (t5) fine-tuning it for the controllable sentence simplification task. the experiments have shown good results of 43.31 sari on turkcorpus evaluation set and of 45.04 on asset evaluation set, outperforming the current state-of-the-art model. also, we have shown that adding the control token #words is useful for generating substitutions with a shorter lengths.