Towards Trustworthy Lexical Simplification: Exploring Safety and Efficiency with Small LLM s (2025)

Abstract: Despite their strong performance, large language models (LLMs) face challenges in real-world application of lexical simplification (LS), particularly in privacy-sensitive and resource-constrained environments. Moreover, since vulnerable user groups (e.g., people with disabilities) are one of the key target groups of this technology, it is crucial to ensure the safety and correctness of the output of LS systems. To address these issues, we propose an efficient framework for LS systems that utilizes small LLMs deployable in local environments. Within this framework, we explore knowledge distillation with synthesized data and in-context learning as baselines. Our experiments in five languages evaluate model outputs both automatically and manually. Our manual analysis reveals that while knowledge distillation boosts automatic metric scores, it also introduces a safety trade-off by increasing harmful simplifications. Importantly, we find that the model’s output probability is a useful signal for detecting harmful simplifications. Leveraging this, we propose a filtering strategy that suppresses harmful simplifications while largely preserving beneficial ones. This work establishes a benchmark for efficient and safe LS with small LLMs. It highlights the key trade-offs between performance, efficiency, and safety, and demonstrates a promising approach for safe real-world deployment.

## introduction text simplification (ts) aims to make texts more accessible by rewriting them in simpler language. ts holds the potential to alleviate reading and understanding difficulties, particularly for individuals with dyslexia @xcite , intellectual disabilities @xcite , and deaf and hard-of-hearing adults @xcite . ts is a task strongly oriented towards real-world scenarios, aiming to promote social participation and inclusion among people who face challenges in text comprehension. recent advancements in large language models (llms) have revolutionized natural language processing and achieved state-of-the-art performance across various tasks @xcite . ts is no exception, as llms have outperformed existing ts systems @xcite @xcite . however, applying llms to ts in real-world scenarios, particularly for vulnerable user groups, faces critical challenges. first, prompts provided to llms and texts requiring simplification may contain sensitive personal information, such as data related to cognitive impairments. the use of apibased llms involves transmitting that sensitive data over the internet, raising significant privacy concerns. for instance, given that individuals with dyslexia often hesitate to disclose their condition due to concerns about stigma and negative perceptions (hamilton @xcite , it can be problematic to design prompts for ts such as "i have dyslexia; can you simplify this diagnosis result for me?". thus, ts systems capable of running locally are highly desirable. open-access llms address this privacy concern. however, high-performing open-access llms typically require substantial computational resources for inference. deploying such large models directly on resource-constrained devices, such as smartphones and tablets that are commonly used by the target users @xcite , is currently impractical. this highlights the need for developing smaller models that can perform effectively within these limited hardware environments. building on these challenges, we investigate how to develop efficient ts systems that can operate under constrained computational resources. this approach is essential for supporting information access for all while respecting user privacy. utilizing small llms is a promising approach, as ~3b models are often explicitly engineered for on-device deployment (metaai, 2024), thereby addressing privacy and efficiency issues. however, particular attention must be paid to safety when employing small llms, as their limited capacity compared to larger counterparts introduces critical considerations regarding the reliability and harmfulness of the generated simplifications. poor or inaccurate simplifications can be detrimental, as they may actively provide misinformation or cause confusion, which are more serious issues than leaving the text unchanged @xcite . therefore, in practice, it is crucial not only to simplify texts effectively, but also to minimize harmful outputs and ensure safety. as a first step towards addressing these challenges, this paper focuses specifically on lexical simplification (ls), a subtask of ts that replaces complex words in a context sentence with simpler alternatives. ls can be considered a relatively conservative and safe subtask compared to sentenceor document-level simplification, which often involves operations such as information deletion @xcite . we adopted small llms and explored two approaches: in-context learning, which requires no training, and knowledge distillation, which transfers knowledge from a larger teacher model to a smaller student model. our approach also considers extensibility to diverse languages, as supporting a broad user group requires simplification across multiple languages. to evaluate the safety of simplification outputs, particularly in suppressing harmful content, we conducted manual evaluations alongside automatic metrics. manual analysis revealed that, while knowledge distillation generally boosted automatic metric scores, it did not reduce harmful outputs and sometimes even increased them. furthermore, we observed that, especially in models trained via knowledge distillation, the output probability provided by llms may serve as a useful signal for identifying harmful simplifications. 1 our contributions are summarized as follows: ## experimental setup figure 1 illustrates the overall flow of our system development and evaluation. we used the hugging-face transformers library 3 for the development of our ls models. a single tesla t4 gpu with 16 gb of memory was used for the development. to enable high-speed inference on cpus, the modthe women and children make guernica the image of innocent, defenseless humanity victimized. the women and children make guernica the image of innocent, defenseless humanitypeople victimized. ## conclusion this study addressed the challenge of building efficient and safe ls systems using small llms, motivated by real-world needs. we proposed benchmark systems in five languages based on in-context learning and knowledge distillation, and introduced a filtering strategy using log-probability as a safety signal. experiments showed that small llms offer significant efficiency gains, but that knowledge distillation, while improving automatic metrics score, increases harmful outputs. we demonstrated that output log-probability serves as an effective signal for detecting harmful simplifications. this signal enables filtering strategy that reduce harmful outputs while retaining beneficial ones. these findings lay the foundation for lightweight ls systems that remain safe and effective across languages. future work should improve training methods to reduce harmfulness and explore real-time ls for mobile environments. ultimately, this research advances deployable, trustworthy ls tools that support inclusive information access.