O ne NRC @ TSAR 2025 Shared Task Small Models for Readability Controlled Text Simplification (2025)

Abstract: In this system description paper, we describe the team OneNRC’s experiments on readability controlled text simplification, focused on using smaller, quantized language models (<20B). We compare these with one large proprietary model and show that the smaller models offer comparable or even better results in some experimental settings. The approach primarily comprises of prompt optimization, agentic workflow, and tool calling. The best results were achieved while using a CEFR proficiency classifier as a verification tool for the language model agent. In terms of comparison with other systems, our submission that used a quantized Gemma3:12B model that ran on a laptop achieved a rank of 9.88 among the submitted systems as per the AUTORANK framework used by the organizers. We hope these results will lead into further exploration on the usefulness of smaller models for text simplification.

## introduction automatic text simplification (ats) is the task of translating a text written in a more advanced language into more accessible language. ats research in nlp is over two decades old and most of the research focused on a single complex -> simple language text generation task, considering the unavailability of large scale graded simplification corpora. @xcite and @xcite are two well-known surveys on the topic. the advent of large language models (llms) made it possible to overcome the ats dataset barrier to some extent with good zero-/few-shot performance on english text simplification @xcite and some recent research explored zero-shot text simplification targeting specific reading levels @xcite . extending this strand of research, the tsar 2025 shared task on readability-controlled text simplification @xcite aimed to compare systems that can generate simplified versions of text that conform to a target cefr reading level, while preserving the meaning. no training data was provided and the test data had 100 texts, each with targeted simplification levels of @xcite . reference simplifications were provided after the task completion. autorank @xcite was used to rank the submitted systems. approach summary: team onenrc's approach to this task focused on comparing how far can small language models go in zero-shot readability controlled ats when they are supported with an agentic workflow and tool support. we used two tools: a) a cefr prediction tool to help the language model agent verify its generated output and regenerate as needed, and b) an optional second tool to check the similarity between the generated text and the original text. for comparison, we also re-ran the same experiments with a larger proprietary model (gemini-2.5-flash), and presented an analysis of the differences in the paper. motivation: several small, open-weight language models were released with 1b-20b parameter range over the past year or two, and some of them can even be run for inference on consumer laptops, often in a quantized form. however, it is unclear how useful are they for many nlp tasks as they are not commonly compared with larger models in existing research. this gap motivates onenrc's submission to this shared task. in terms of the performance, one of the runs we submitted (using a 4-bit quantized gemma3:12b model on a laptop) achieved an autorank of 9.88, among the 40 submitted runs @xcite . in this paper, we describe our approach in detail (section 2), discuss other experiments which achieve better results than the submitted runs, share some qualitative observations (section 3) along with broad conclusions on learnings through this study. (section 4). ## approach our approach primarily concentrated on exploring the power of small language models for this task, under the following zero-shot settings: these three settings focus on incrementally enhancing an llm's built in capabilities without finetuning. the first setting relies on a hand-crafted prompt, and is potentially subject to more variation in the output depending on even minor changes in prompts. the react approach aims to interleave reasoning and actions performed by the llm with or without additional tools, and and is expected to provide more reliable outputs supported by a reasoning trace. dspy focuses on structured specification of prompts, inputs, and outputs and emphasizes on a more programmatic interface to interacting with an llm instead of handcrafted prompts, which further adds another layer of support to the base llm. thus, these three experimental settings can be viewed as a sequence of incremental adjustments to llms. compute and costs: all experiments were run on a macbook pro m1 pro laptop with a 32 gb ram, and the costs for running the gemini-2.5flash experiments amounted to 5 usd. all the implementation code including the used prompts are provided in the github for replication and reproduction. 3 3 results we report results using the three official measures (rmse, meaningbert-orig, meaningbert-ref) along with the weighted-f1 for cefr level compliance (which our approach optimized for), and the following three models: the first two models run on a consumer laptop using ollama foot_4 . the gemini-2.5-flash model was accessed through api calls with openrouter foot_5 . our final results are summarized in ## conclusions overall, our experiments lead us to three main conclusions: evaluation: evaluation in this shared task primarily relied on automated models, and what appears like an ad-hoc weighting scheme. while a smaller scale human evaluation may be needed to understand the generated texts better, explana-tory measures using llm-as-a-judge approaches or checklist based evaluation approaches @xcite ) may provide more informative evaluation for text simplification. small language models may hold a potential as judges too, providing low-cost, low-carbon footprint options. finally, manual analysis revealed the possibility of hallucination even in these constrained generation scenarios. as the use of llms for text simplification increases, the evaluation measures need to also account for hallucination as a potential dimension. future work: while a laptop environment is not conducive to conduct experiments on a larger scale, we believe future work should focus on exploring the power of smaller models in few-shot settings or fine-tuning using synthetic data generation. dspy's automatic prompt optimization capabilities in few shot settings should be better explored for this task, to understand the true potential of smaller models for text simplification. the current shared task focused only on english, but these experiments lead us to hypothesize that there may be some possibilities to replicate these results for at least some of the high resource languages, which needs to be explored. finally, using language models as support tools to aid the authors of simplified texts also seems to be a useful direction to pursue.