BLESS : Benchmarking Large Language Models on Sentence Simplification (2023)

Abstract: We present BLESS, a comprehensive performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS). We examine how well off-the-shelf LLMs can solve this challenging task, assessing a total of 44 models, differing in size, architecture, pre-training methods, and accessibility, on three test sets from different domains (Wikipedia, news, and medical) under a few-shot setting. Our analysis considers a suite of automatic metrics, as well as a large-scale quantitative investigation into the types of common edit operations performed by the different models. Furthermore, we perform a manual qualitative analysis on a subset of model outputs to better gauge the quality of the generated simplifications. Our evaluation indicates that the best LLMs, despite not being trained on TS perform comparably with state-of-the-art TS baselines. Additionally, we find that certain LLMs demonstrate a greater range and diversity of edit operations. Our performance benchmark will be available as a resource for the development of future TS methods and evaluation metrics.

## introduction large pre-trained language models (llms) have demonstrated strong performance on a wide range of nlp tasks without the need for task-specific fine-tuning, leading to a prevailing conventional wisdom that llms can solve any task. this has motivated the development of benchmarks to better understand the abilities of llms in specific domains such as healthcare @xcite , finance @xcite , education @xcite , engineering (soba1 we make our code and the generated system outputs available at https://github.com/zurichnlp/bless . however, it remains unclear how well current llms can perform on the challenging task of text simplification (ts). in this paper, we focus on sentence simplification in english, which typically involves rephrasing part or all of a sentence into language which is more accessible and easier to understand. while recent work has focused on evaluating ts abilities of select models, such as gpt-3.5-turbo @xcite and mt5 @xcite , there is currently no large-scale and detailed analysis of the simplification capabilities of different llms. in this study, we expand both the breadth and depth of the knowledge base on ts with llms, evaluating a wider variety of models on three different ts datasets: asset @xcite , newsela @xcite and med-easi @xcite . we select these datasets to cover a variety of domains (wikipedia, news, and medical) and a diverse set of ts operations (e.g. paraphrasing, splitting, and elaboration). specifically, we use in-context learning (icl) and assess llms in a few-shot setting, experimenting with three different prompts. we select 44 widely used generative models (both open and closed-weight) and evaluate their abilities from three distinct angles. first, we rely on automatic evaluation metrics commonly used in the ts literature. second, we quantify and compare the edit operations performed by the llms during simplification. finally, we perform a targeted qualitative analysis to validate our findings and to better understand the quality of the generated simplifications. our findings reveal that closed-weight models provide significant gains over open-weight alternatives under a few-shot setting, establishing them as a strong baseline for future work on ts. we summarize our contributions as follows: ## related work text simplification benchmarks most simplification work treats the task as a monolingual machine translation problem, training models on datasets containing complex-simple sentence pairs @xcite . @xcite performed a standardized evaluation of general data-driven simplification systems, using wikipedia-based datasets and newsela. at the document level, alva-manchego et al. (2019b) conducted a systematic analysis of simplification operations to demonstrate the limitations and disruptions that occur when multiple sentences are involved. benchmarks have also been established for more specific kinds of simplification: for example, both non-neural @xcite and neural @xcite approaches to lexical simplification, which aims to replace complex words with simpler alternatives. ## automatic evaluation results in this section, we present the results of our automatic evaluation of simplification outputs and summarize our main findings. first, we perform an exhaustive assessment using automatic metrics (section 3.6). for brevity, we report the results of the best-performing llms with sari and bertscore in ## qualitative analysis automatic metrics are known to have blind spots and are not always entirely reliable @xcite . to compensate for this, we perform a qualitative analysis on a total of 300 system outputs. first, we check whether or not each output is a valid simplification and highlight common failure cases such as inappropriate changes to the meaning of the original text, ungrammatical outputs, and the occurrence of hallucinations. then, we annotate occurrences of common simplification edit operations such as lexical simplification, deletion, sentence splitting, reordering, and paraphrasing. 8 for our annotations, we select model outputs from the top five systems ranked according to performance on the individual evaluation metrics of sari, bertscore, fkgl, and lens. in each ranking set, we randomly select five complexsimple pairs from all generation settings. to evaluate a unique set of models for greater diversity, if a system is repeated in the ranking (e.g. two different prompt types from the same model appear in the top five), we choose the next best system for analysis. an example of our annotated outputs is shown in by automatic metric overall, we find that simplicity and meaning preservation are fairly balanced. however, there is a clear trade-off between these two axes when we consider the top 5 models according to sari and bertscore. this agrees with earlier findings from @xcite . along with a higher degree of simplicity, the top 5 sari models exhibit more diverse edit operations than those ranked highly by bertscore. lens, however, does not trade off simplicity and meaning preservation and even achieves a higher simplicity score than sari along with its increased level of deletion. this result is in line with the previous finding that lens achieves stronger correlations with human judgments compared to existing ts metrics @xcite . the top 5 models ranked by fkgl, on the other hand, produce outputs with low simplicity and meaning preservation and an especially high amount of hallucinations. this result supports the previous finding that fkgl can be easily gamed by degenerations @xcite and is therefore an unsuitable metric for evaluating the outputs of automatic ts systems. by open-status open-weight models most frequently use the operations of lexical simplification, paraphrasing, and deletion, while structural operations such as sentence splitting and reordering are often neglected. many only achieve high meaning preservation by directly copying the input sentence. however, the closed-weight models investigated here behave very differently: they produce close to 10% more splitting, lexical simplification, and re-ordering than open-weight ones, while simultaneously performing fewer deletions. this leads to a greater degree of paraphrasing. by domain when comparing performance between different domains, we observe that all llms do significantly better on general encyclopedic texts in asset in terms of both simplicity and meaning preservation, while also exhibiting a diverse set of edit operations. although outputs from newsela contain more hallucinations, meaning preservation is still fairly high. outputs from med-easi, on the other hand, have the lowest meaning preservation by far and the least diverse set of edit operations. we find that med-easi outputs, along with others that do not preserve meaning, often contain repetitions, hallucinations, and in some cases even copy the input prompt, demonstrating a tendency to disregard the instruction and thus fail to complete the task. these failure modes are most frequently observed from the smaller t5 models, but are also exhibited by models such as llama when evaluated on med-easi. ## discussion we discuss our results around the following aspects: the access level of the simplification models (openvs. closed-weight), the training strategies (general pre-training vs. general fine-tuning strategies), and the utility of automatic metrics. access level among the openai models, we observe that all models perform particularly well on meaning preservation according to bertscore but exhibit considerable differences in their ability to simplify, as indicated by sari on 'weaker' models such as ada-001. among the evaluated open-weight models, we observe that the flan models (t5 and ul2) typically perform competitively, punching well above their weight in terms of parameter counts with much larger decoder-only models. this is a promising finding for the category of open-weight models, and we hope that this encourages future work to continue investigating different methods regardless of the model size. training strategies within model families, when comparing base models to their instruction fine-tuned counterparts, we observe that instruction-tuning typically leads to better performance in our few-shot icl setting for ts. we find this to be particularly encouraging since ts is one task often hindered by the lack of high-quality labeled training data @xcite . nevertheless, improvement is not always guaranteed, as seen when comparing bloom vs bloomz. in this case, instruction fine-tuning leads to bettermeaning preservation but a reduction in the degree of simplification, indicating that the instructiontuning method used to derive the multilingual bloomz may be less suitable for english ts. this stands in stark contrast to the flan instruction tuning method, which delivers considerable gains in both sari and bertscore despite sharing the same underlying instruction-tuning dataset as bloomz. therefore, we hypothesize that this drop in performance may be influenced by the multilingual instruction tuning setup that is unique to bloomz. ## conclusion in this paper, we provided a comprehensive assessment of how well out-of-the-box llms perform on the task of ts with few-shot in-context learning. we found that the best llms outperform state-of-the-art supervised ts baselines while also producing a more diverse set of simplification operations. we also established that closed-weight models perform better than open-weight ones and that general instruction-tuning often improves a model's abilities on ts. furthermore, we empirically validated the trade-off between simplicity and meaning preservation through automatic evaluation and a manual analysis. our analyses of multiple few-shot prompting strategies revealed that a more structured prompting format produces better results than presenting source-target examples in continuous text. our performance benchmark, bless, provides a strong foundation for future work. for example, it remains an open question as to which expressions and instructions are optimal for prompting llms to simplify texts. furthermore, this work exclusively focused on few-shot in-context learning. future work could explore the capabilities of these systems in zero-shot, fine-tuned, or retrieval-based settings.