Comparing Generic and Expert Models for Genre-Specific Text Simplification (2023)

Abstract: We investigate how text genre influences the performance of models for controlled text simplification. Regarding datasets from Wikipedia and PubMed as two different genres, we compare the performance of genre-specific models trained by transfer learning and prompt-only GPT-like large language models. Our experiments showed that: (1) the performance loss of genre-specific models on general tasks can be limited to 2%, (2) transfer learning can improve performance on genre-specific datasets up to 10% in SARI score from the base model without transfer learning, (3) simplifications generated by the smaller but more customized models show similar performance in simplicity and a better meaning reservation capability to the larger generic models in both automatic and human evaluations.

## introduction controllable text simplification is a technique whereby the features of a generated simplification (e.g. its length) can be determined at inference time. control tokens prepended to the input with specific features' values can be regarded as a way of prompting text simplification systems to generate outputs with certain desired characteristics. this gives rise to flexible and controllable simplification systems that satisfy various demands from different user groups or scenarios with regulated output @xcite @xcite @xcite . a use case for such types of models is making specialised information (e.g. related to medicine) more accessible to lay users. we present genre-specific text simplification research alongside a study on the effects of different genres. we followed the idea of multilingual unsupervised sentence simplification (muss) @xcite , which is the state-of-the-art (sota) of controlled text simplification, to build the base model and expert models. different from muss, in which the authors combined the explicit control tokens with the mined paraphrase corpus, we combined the control tokens with two small expert-level genre-specific training subsets derived from simple tico-19 corpus @xcite . the base model reimplements the muss without the fine-tuning on the paraphrase corpus, while the expert models are further fined-tuned on the genre-specific training subsets. we choose the newly published simple tico-19 dataset @xcite as our training and test bench of genre-specific tasks for the expert models, because of the manual simplification from experienced annotators and expertlevel information in covid-19. based on simple tico-19, we created the two subsets with unified data source labels as two different genre-specific corpora and designed the genre-specific tasks with different permutations of each kind of subset. to verify the improvement before and after transfer learning, we tested the performance of the expert models over the base model in the abovementioned genre-specific scenarios. in addition, considering the strong competitiveness of more updated and larger language models than the base model @xcite @xcite @xcite , it is worth finding whether the large language models targeting generic content can outperform lightweight custom models that have been specialized for specific tasks. thus, we also compared the expert models with the leading generic models for generative nlp, covering gpt-3 @xcite and chatgpt. in this paper, we leveraged a newly published text simplification dataset, designed a test scenario for controlled text simplification with different genres, proved the effects of transfer learning on the genre-specific datasets, compared the performance of generic and expert models in sari score and bertscore, and discussed the cost-effectiveness between expert models and generic models. ## related work text simplification consists of reducing linguistic complexity at both syntactic and lexical levels without significant loss in the main content @xcite . in practice, this task can be treated as monolingual machine translation @xcite . research in english highly relies on wikipedia and simple wikipedia @xcite @xcite @xcite . high-quality manuallymade corpora are rare and some may come with restrictions @xcite @xcite . to alleviate this problem, we combined the large automated corpus with the small manual-made corpus. text simplification researchers have recently turned to larger pre-trained language models @xcite @xcite @xcite . from long-short term memory (lstm) to transformer-based pre-trained models @xcite @xcite , the order of magnitude of parameters used in models for text simplification has increased dramatically, the parameter count in bidirectional auto-regressive transformers (bart) is 140 million @xcite , the value in text-to-text transfer transformer (t5) reaches 220 million @xcite , the value in gpt-3 increases to 170 billion @xcite , and the value in switch transformer even reaches an astonishing 1.6 trillion @xcite . with the advent of pre-trained language models in nlp, the sota of many common tasks and leaderboard is refreshed @xcite @xcite . models with more parameters tend to perform better on downstream tasks @xcite . however, larger models require more energy to run (puvis de @xcite and are inaccessible to a typical researcher, hampering reproducibility. besides, the correlation between large models and high performance is still worth exploring and the necessity of extremely huge models is questionable. to find out the exact situation in text simplification, we leveraged the latest pre-trained large language model chatgpt. in addition to the general models, there are also researches focusing on controlled text simplification @xcite @xcite . due to the various demands of lay users in text simplification, the generic output can hardly satisfy the main user group @xcite . controlled text simplification is introduced to satisfy the various demands of different user groups or in different scenarios with explicit or implicit restraints on the output. in audience-centric sentence simplification (access), @xcite ) present the 4 control tokens used in this paper, sheang and saggion (sheang and saggion (2021) replace the bart model @xcite with t5 model @xcite , further extend the control tokens to 5 and refresh the sota. the performance and flexibility of controlled text simplification make it possible to compete with the large pre-trained language models, and they will be tested in this paper. ## methodology in this section we describe the experiments that were undertaken. a visual representation of our methodology is provided in figure 1 , which is explained in further detail throughout the following subsections. ## conclusion in this paper, we compared the performance differences between generic models and expert models on general and genre-specific simplification datasets. we showed the effect and practicality of transfer learning in genre-specific datasets with less amount of samples. the performance drop on general tasks after transfer learning is acceptable and may be further reduced in future studies. the performance, cost-effectiveness and portability of expert models prove themselves as one of the practical solutions for domains-specific or genresspecific tasks.