Know- AI at TSAR 2025 Shared Task Difficulty-aware Text Simplification System (2025)

Abstract: Text simplification is an active research topic with applications in multiple domains. In a simplification pipeline, assessment of text difficulty plays a crucial role as a quality control mechanism it acts as a critic and guides models to generate text at the difficulty level required by the user. This paper presents our Difficulty-aware Text Simplification System. We evaluate our pipeline using the TSAR shared task dataset and discuss challenges in constructing corpora for training models to assess text difficulty.

## introduction text simplification is a widely studied task in natural language processing (nlp), with applications in accessibility, education, and communication. it is important in many applications where the userse.g., non-native speakers-struggle to understand complex or standard language. the goal is to reduce the linguistic complexity of a text, while maintaining the original text's core meaning and coherence. increasingly, official legislation in europe (inclusion europe) requires government organizations, ngos and other public agencies to provide information to clients in clear and accessible form, including for readers who may be unable to understand standard language. we are motivated especially by applications of simplification in secondlanguage (l2) education, where personalized learning is supported by adapting text to the learner's proficiency level @xcite . our simplification pipeline, 1 shown in figure 1 , uses a critic consisting of two parts: (a) difficultyit evaluates the difficulty level of a text simplified by a large language model (llm), and (b) semantic 1 simplification.py similarity-it checks how well the simplified text preserves the semantics/meaning of the original text. this framework was introduced in @xcite , in l2 education. in this paper, we adapt the framework for simplification in english. the pipeline iteratively attempts to generate a "simplified" version of an input text. if the generated text is above the target level of difficulty, then feedback-including the generated text and its currently assessed level-is sent back to the llm to revise the output. the pipeline makes several attempts at simplification to reach the target difficulty level. we experiment with several critics in the pipeline, including an open-source transformer-based model that classifies text by difficulty level, and a regression model that we train using english-language texts labeled with difficulty levels. the paper is organized as follows: section 2 gives a brief overview of related work. section 3 describes the shared task and the evaluation methods. section 4 presents the architecture of our simplification pipeline. section 4.1 describes the experiments with controlling the behavior of a llm via the difficulty critic. section 5 presents results and analysis. section 6 concludes the paper and discusses directions for future work. ## related work prior approaches to text simplification relied on assessment of text difficulty to identify sentences requiring simplification. for example, @xcite trained a model to detect linguistically complex sentences; @xcite developed readability assessment tools to support simplifying texts for low-literacy readers. readability metrics have also been incorporated directly into rule-based simplifiers: woodsend and lapata (2011) integrate the flesch-kincaid grade formula @xcite into optimization-based simplification. more recent approaches to simplification leverage readability predictors as feedback within generation loops. alkaldi and inkpen (2023) use a readability classifier in a reinforcement learning framework to iteratively simplify text until it reaches the desired difficulty. large-scale neural systems have combined readability prediction with controllable generation techniques to produce text at the target difficulty level @xcite . ## task description the shared task on readability-controlled text simplification @xcite involves simplifying english-language paragraphs written at upper-intermediate or advanced levels. participants are required to produce simplified versions at a target readability, specified as a cefr level: common european framework of reference for languages @xcite . our experiments are based on the test dataset provided by the tsar shared task. the test set consists of english paragraphs at level b2 or higher, each associated with a target level (a1, a2, or b1). no training data, and no reference simplifications are provided. the evaluation involves measuring multiple aspects of the simplified texts: these metrics are calculated using the official evaluation scripts released by the shared task organizers with the test dataset. the semantic similarity in the evaluation scripts uses meaningbert @xcite . meaningbert is a bertbased semantic similarity model that measures how well meaning is preserved between two texts, particularly for tasks such as text simplification and paraphrase assessment. ## system overview we next describe how we use the critic model to guide in llm-based text simplification pipeline (see figure 1 ). the pipeline begins by determining the difficulty of a source text, either with a difficulty model or manual annotation. the text, together with the target cefr level and a prompt, is passed to a llm, which produces a candidate output. the critic model evaluates the candidate's difficulty; if it matches the target level, the process ends. otherwise, the llm is re-prompted with the previous output and the discrepancy from the target. this loop continues for up to n iterations-a predefined maximum, to balance between cost and quality. the system then outputs either a satisfactory simplification, or an error if the target is not reached. ## results and analysis in this section, we examine the results of simplification with different critic models. beyond exactmatch accuracy, we assess how well the predicted difficulty levels match the intended simplification direction. the direction consistency metric measures whether predictions respect the target level ordering for each input. ## discussion and future work the effectiveness of our proposed pipeline depends on the choice of difficulty assessment model used in the critic, since it guides the simplification process. in addition to the models above, we experimented with training our own difficulty assessment model. although this approach did not appear in our submissions for the shared task, it shows much promise for future work. this section summarizes the lessons learned from this attempt. first, since no training data were provided for the shared task, we construct a training, development and test set-test set 1-by taking an existing corpus foot_3 described in @xcite and translating it from finnish into english, using the opus machine translation (mt) toolkit foot_4 @xcite . it is crucial to note that we found that the opus models are particularly strong at preserving the cefr levels of the original source text in the mt output text. 7 we also use the reference set provided by tsar as a second test second, following the methodology of katinskaia et al. (2025), we train a regression model to predict difficulty. we were unable to gather a sufficient amount of training data and tune our regression model in time for the actual tsar competition; therefore, as a fallback, we used alllang2-cefr2 rather than the regression model as a critic in our submission for the shared task. we next check how well difficulty prediction works-on its own, apart from the simplification task. for test set 1, the difficulty prediction results are in figures 6 and 7 . the regression model shows a clear advantage over the alllang2-cefr2 model, exhibiting a clear step-wise pattern that aligns well with cefr levels. it consistently outperforms the baseline across all evaluation metrics. the evaluation metrics for difficulty prediction are shown in the top part of for test set 2, the evaluation metrics for difficulty prediction are in the bottom of several factors may compromise the performance of our regression model. first, the dataset figure 9: difficulty estimation using regression model in test set 2. is machine-translated, which may distort the true difficulty of the texts. ideally, training data is manually annotated for difficulty. however, manual annotation is very complex and time-consuming. second, the translated dataset is still small, restricting the model's ability to generalize across different linguistic phenomena. in future work, we plan to extend the setup relying solely on gpt-4o for text simplification, to consider other models, including smaller models fine-tuned for the simplification task. we will investigate more advanced models to improve the assessment of difficulty, which is central for the simplification pipeline. larger, more accurate, and more diverse training datasets should further improve performance and generalization. ## lay summary this study investigates text simplification, in the context of the shared task on text simplification, accessibility, and readability (tsar). we present a difficulty-aware simplification pipeline based on large language models (llms) and small models for simplification assessment. we use text data in english, of varying levels of difficulty, ranging from a1 to c1 on the cefr scale. we evaluate performance according to several criteria, including error rates of difficulty assessment models in their assessment of the difficulty of texts in a held-out test set, and the success rates of the simplification pipeline, relative to reference texts provided by the organizers of the shared task. the paper a. discusses the performance of a number of critic models for assessing difficulty of a text, and b. compares the performance of the simplification pipeline driven by the different critics. kristian woodsend and mirella lapata. 2011. learning to simplify sentences with quasi-synchronous grammar and integer programming. in proceedings of conference on empirical methods in natural language processing, edinburgh, scotland.