Archaeology at TSAR 2025 Shared Task Teaching Small Models to do CEFR Simplifications (2025)

Abstract: Large language models (LLMs) have demonstrated strong performance in text simplification tasks, but their high computational cost and proprietary nature often limit practical use, especially in education. We explore open-source LLMs for CEFR-level text simplification. By reducing model size and computational requirements, our approach enables greater accessibility and deployment in educational environments. Our results show some of the lowest error rates in producing CEFR-compliant texts at TSAR 2025, using models with 8 billion and 1 billion parameters. Such approaches have the potential to democratize NLP technologies for real-world applications.

## introduction text simplification (ts) aims to preserve the original meaning while reducing linguistic complexity, making text more accessible to a wider audience @xcite . in educational contexts, this is particularly valuable, as learners benefit from materials adapted to their proficiency levels @xcite . in this paper, we explore different prompting methods to large language models (llms) and provide a comparative analysis between large closed-source systems and smaller open-weights models. while recent llms demonstrate strong language understanding capabilities, their high computational requirements and proprietary nature still limit broad educational deployment, particularly in resource-constrained settings @xcite . one-billion parameter models fine-tuned with synthetic data show promising results in providing accessible solutions for educational use. while prior work has treated cefr-level ts as a unified task with controllable outputs (barayan * both corresponding authors. @xcite , we propose decomposing this problem into distinct sub-tasks, where each cefr level represents an independent simplification objective. we investigate two approaches: (1) level-specific prompts that are very descriptive individually for the specific target, and (2) specialized fine-tuning where models are trained exclusively for individual cefr levels. additionally, we use an iterative prompting technique inspired by minimum bayes risk decoding @xcite , where multiple attempts with slight prompt variations are used to achieve optimal simplification quality. ## results and discussion while kimi k2 achieves strong results across automatic metrics, qualitative inspection suggests that its simplifications often lack fluency and cohesion. in particular, the model tends to produce excessively fragmented and repetitive sentences, which explains the gap between metric-based performance and actual readability. this pattern also raises the possibility that kimi k2 may be exploiting weaknesses in current evaluation metrics, producing outputs that score well numerically. a detailed comparison with human references and outputs from claude and llama 3.1 is provided in appendix 2. in contrast, the results for llama 3.1 8b instruct illustrate a divergence between automatic evaluation and human judgment. according to when examining the outputs of llama 3.2 1b instruct without fine-tuning, two issues become evident. first, the model often fails to follow all prompt instructions, even when these are stated multiple times. for example, it frequently returns additional comments or explanations such as "here is the simplified text..." or even detailed notes about the transformation process, despite being explicitly instructed to output only the simplified text. second, the base model tends to produce excessively short simplifications, sometimes reducing an entire paragraph to one or two sentences. this extreme shortening is caused partly by the iterative prompting setup: since the model has no prior knowledge of how a cefr-target simplification should look (zero-shot), the repeated corrections encourage it to strip down content aggressively rather than restructure it appropriately. moreover, the extra notes and process explanations negatively bias the iterative pipeline: because the cefr classifier evaluates the entire response, the added commentary shifts the predicted level upwards or downwards in unintended ways. for instance, in one case the model returned a step-bystep justification of lexical changes ("i removed 'common' from 'common dreams', changed..."), which was treated as part of the text by the classifier and distorted the cefr label. examples of both the over-shortening and adherence failures are documented in appendix 3, demonstrating how iterative prompting without prior fine-tuning leads to unreliable outputs and noisy automatic evaluations. by contrast, the fine-tuned 1b model overcomes these limitations: it aligns much more closely with the intended structure of cefr simplifications, avoids adding unwanted commentary, and produces outputs that are considerably richer in content while still controlled in complexity. these improvements are particularly noteworthy given that the fine-tuning relied on synthetic data generated with llama 3.1 8b, itself a relatively modest model, and that the resulting dataset was not manually cleaned or verified. this suggests that the gains observed here represent only a lower bound of what could be achieved; with higher-quality, carefully curated, or state-of-the-art synthetic simplification data, fine-tuning small models would likely provide even greater improvements. for completeness, we also conducted a smallscale fine-tuning experiment on llama 3.1 8b, using approximately 1000 synthetic simplifications generated with claude sonnet 4 and the same finetune configuration as the one used for llama 3.2 1b. while the dataset was limited in size and not manually verified, it still allowed us to observe meaningful improvements. due to the small amount of data, we did not apply iterative prompting during inference and instead evaluated the model in a single-try setting. this setup affected the results in two ways: (1) rmse scores were higher compared to the iterative experiments, since the model had fewer opportunities to adjust toward the exact cefr level, and ( 2 ingbert scores improved, consistent with our hypothesis that the first simplification attempt usually yields the highest semantic similarity. even under these constrained conditions, the one-try rmse remained competitive, showing that even with a very limited number of training examples, fine-tuning substantially enhanced the model's ability to perform cefr simplification. it is also important to note that our experiments were primarily optimized towards minimizing rmse, as this metric directly reflects the alignment of the outputs with the target cefr level. consequently, aspects related to meaning preservation were not prioritized to the same extent. this explains why some systems, despite achieving lower rmse, occasionally sacrificed fidelity or omitted information from the original text. a more balanced approach that explicitly weights both accuracy of cefr control and semantic consistency could represent an important direction for future work. furthermore, our focus on minimizing rmse exposed some of the limitations and biases in the current evaluation metrics. for example, kimi k2 achieves relatively strong scores despite producing outputs that are qualitatively weak, highlighting that automatic metrics can sometimes reward undesirable behavior. this suggests that more robust, state-of-the-art evaluation methods are necessary to obtain a fair ranking of systems. under such an evaluation framework, we hypothesize that an approach built on well-balanced minimum bayes risk decoding could result in models that are both reliable for cefr control and practical enough to be deployed. in addition, as shown in ## ethical and deployment considerations our methods enable practical deployment of cefraware simplification, but operational use raises concrete risks. synthetic training data and automatic simplifiers may introduce factual inaccuracies, level misclassification, or unequal behaviour across learner groups. therefore, any real-world deployment should include human validation, continuous monitoring for meaning preservation and fairness, clear documentation of model limitations and licensing, and user-facing transparency about automated processing. furthermore, the proposed approach contributes to a more sustainable deployment paradigm: instead of relying on repeated inference from large proprietary llms, it enables the generation of synthetic data to train smaller, open models. this reduces computational and financial costs while improving accessibility for educational institutions with limited resources. another advantage of our method lies in its applicability to low-resource settings, where highquality labeled simplification data are scarce. synthetic data generation through llms can bootstrap new cefr-aligned datasets for such languages. future work could incorporate human-in-theloop feedback from language teachers to refine level control and to assess pedagogical usefulness beyond automatic metrics. ## conclusions this work demonstrates several key findings for cefr-based text simplification using smaller, open-source language models. our iterative prompting approach provides a computationally inexpensive method to significantly improve simplification quality. by allowing models up to 5 attempts per text, we achieved substantial rmse improvements across all tested models. this technique offers a practical way to enhance performance without additional training or model scaling, making it particularly valuable for resource-constrained educational environments. highly structured, cefr-specific prompts improve model understanding of proficiency level requirements. our detailed prompts, which explicitly describe vocabulary constraints, sentence complexity, and grammatical structures for a2 and b1 levels, led to more accurate simplifications compared to generic approaches. this finding suggests that explicit linguistic guidance can compensate for models' limited understanding of cefr frameworks. training separate models for each cefr level yields better results than training a single model to handle multiple target levels. this specialization allows models to develop more focused representations of the linguistic constraints and stylistic requirements specific to each proficiency level, resulting in more appropriate simplifications. our fine-tuning experiments further serve as a strong proof of concept. we observed that even when training with synthetic data generated by llms (which were not manually verified and may inherently contain too much noise) substantial improvemenets can still be achieved. this effect was visible both when using large amounts of automatically generated samples (from llama 3.1 8b) and under very constrained conditions with only 1000 examples (from claude sonnet 4 for llama 3.1 8b). these results highlight the potential of fine-tuning: with higher-quality and carefully curated datasets, the gains observed here would likely be amplified, suggesting a clear path forward for building robust simplification systems on top of smaller open models for educational use. our code is available at https://github.com/ roscanrares/emnlp-2025-tsar .