Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification (2023)

Abstract: Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).

## introduction text simplification (ts) is a task in the field of natural language generation. it aims at rewriting a complex text into simple text while keeping the primary meaning intact @xcite . recently, several works have leveraged pretrained models for ts @xcite . however, problems arise when pre-trained models are applied to ts directly. in the pre-training stage, the model hardly acquires the ability to generate simple texts. the improvement of results on simplification tasks relies almost on the fine-tuning stage. it can hurt the performance of pre-trained models, especially for lowresource sub-tasks like lexical simplification. one reason for this shortcoming is the pre-training strategy. it randomly masks text spans in ordinary texts, teaching the model to generate ordinary texts rather than simple texts. we are committed to adapting the pre-trained model to ts in this paper. the pre-trained model has gained the ability to generate ordinary texts, and it is costly to start pre-training from scratch. therefore, we focus on the continued pre-training strategy @xcite . we first aim to continue pre-training on simple texts because it contains plenty of simple words. in ts, simple texts are derived almost from simplewiki @xcite and newsela @xcite . we identify simple text spans in simple texts and dynamically replace them @xmath0. then, the pre-trained model will learn by reconstructing simple words. meanwhile, we expect the pretrained model to learn from ordinary texts. we use a dictionary to replace complex words in ordinary texts with simple words. we also ensure the quality of the replaced sentences. based on bart @xcite , we continue pre-training to teach it to generate simple texts and obtain simplebart. we then conduct experiments on three main tasks of ts: sentence simplification, lexical simplification, and documentlevel simplification. simplebart achieves consistent and noticeable improvements across several datasets on all three tasks over bart and several other baselines. the results illustrate that our proposed strategy helps the pre-trained model to gain the ability to generate simple texts. ## methodology as illustrated in figure 1 , our strategy is divided into two parts: learning dynamically to reconstruct simple words from simple texts and from ordinary texts where complex words are replaced with simple ones. complexity identification a switchgear is a combina/on of electrical switches … protect or isolate electrical equipment … a switchgear is @xmath1.3 0.1 neofelis is a genus comprising two extant cat species from southeast asia neofelis is a genus including two presented cat species from southeast asia replace words from dict dynamic mask neofelis is a genus comprising two extant cat crops from southeast asian neofelis is a @xmath2: ordinary text: pre-trained model a switchgear is a combina/on of electrical switches … neofelis is a genus including two presented cat species from southeast asia ## comparing simplebart with large language models large language models (llms) have received widespread attention from researchers recently and have achieved state-of-the-art results on many natural language generation tasks. in this section, we select several representative large models to conduct experiments on text simplification and compare them with simplebart. we hope these results can serve as baselines for future research. we choose those llms that provide api or model files to ensure reproducibility. we choose gpt-3.5-turbo-0301 foot_0 , flan-t5-xl @xcite , and llama-7b @xcite as llm baselines and use zero-shot generation. then, we follow the implementation foot_1 and fine-tune flan-t5-base as another baseline. we collect the training sets of wikiauto, newsela, and d-wikipedia and conduct instruction fine-tuning. ## conclusion in this paper, we are committed to adapting the pre-trained model to text simplification. we propose a new pre-training strategy to allow the pretrained model to learn to generate simple texts. the adapted pre-trained model improves the results on various simplification tasks.