Exploring Paraphrasing Strategies for CEFR A1-Level Constraints in LLM s (2025)

Abstract: Large language models are increasingly used for teaching and self-learning foreign languages. However, their capability to meet specific linguistic constraints is still underexplored. This study compares the effectiveness of prompt engineering in guiding ChatGPT (4o and 4o-mini), and Llama 3 to rephrase general-domain texts to meet CEFR A1-level constraints in English and Italian, making them suitable for beginner learners. It compares 4 prompt engineering approaches, built upon iterative paraphrasing method that gradually refines original texts for CEFR compliance. The approaches compared include paraphrasing with or without Chain-of-Thought, as well as grammar and vocabulary simplification performed either simultaneously or as separate steps. The findings suggest that for English the best approach is combining COT with separate grammar and vocabulary simplification, while for Italian one-step strategies have better effect on grammar, and two-step strategies work better for covering the vocabulary. The paraphrasing approach can approve compliance, although at this point it is not cost-effective. We release a dataset of pairs original sentence-beginner level paraphrase (both in Italian and in English) on which further work could be based.

## introduction large language models can generate vast amounts of content for language teaching and learning, offering new possibilities for personalized and scalable materials. however, llms exhibit difficulties in consistently adhering to strict constraints specified via prompting. @xcite while occasional exposure of foreign language learners to unfamiliar elements is necessary for language acquisition @xcite , the texts that remain fully within given grammatical and lexical boundaries can be valuable in instructional settings-for instance, when designing controlled practice activities focused on a specific target structure or vocabulary set. language proficiency levels are commonly assessed using the cefr, a comprehensive framework developed by the council of europe. cefr (common european framework of reference for languages) categorizes language ability into six levels: a1 and a2 (basic user), b1 and b2 (independent user), and c1 and c2 (proficient user). its descriptors focus on communication tasks that language learners can complete at each level. @xcite language generation under cefr constraints is notoriously a difficult task for llms (see section 2). in this study we explore the combination of two prompting strategies in a task of text simplification. addressing text simplification rather than text generation allowed us to start from existing texts and have a better control of the task. furthermore, it might be a useful approach to use for adaptive tutoring systems for foreign language learning @xcite . this study focuses on the task of rephrasing existing texts to make them suitable for a1 cefr level reading activities. beginner-level adult learners require texts that are tailored both to their level and to their interests @xcite . while authentic texts are appropriate at higher levels, offering their paraphrased versions at a1 is a tradeoff between integrating authentic materials, which increases students' engagement and motivation, @xcite and making them accessible for beginners. a1 is the most restricted level in terms of both grammar and vocabulary (council of europe, 2020), which makes it a particularly demanding target, as the criteria go beyond general readability. careful control of grammar structures, such as limiting tense usage and avoiding irregular verbs @xcite is not normally re-quired in simplified text for native speakers. this makes a1-level paraphrasing a lower-bound evaluations which allows to see how capable are llms of paraphrasing texts using an extremely constrained language inventory. specifically, the research question addressed is the following: to what extent can large language models be prompted to rephrase input texts in accordance with cefr a1-level grammar and vocabulary constraints as specified in inventories for english and italian? we address two languages, english and italian, and three llms in order to provide a tentative test for generalization of our results. our findings suggest that guided paraphrasing can enhance compliance with a1 standards, although imperfectly. due to the complexity of a1 inventories, this improvement comes at a considerable cost in terms of processed tokens. for english, one specific combination of strategies appears to be significantly more effective. in contrast, the results for italian are less consistent, and no single best strategy clearly emerged. ## related works for cefr-constrained text generation and simplification, researchers mainly use prompt engineering and model fine-tuning, often together. @xcite combined these approaches by using few-shot prompting, supervised fine-tuning, and reinforcement learning to develop calm (cefr-aligned language model), a fine-tuned version of llama2-7b, which they report outperforms gpt-4 for cefr-aligned content generation. similarly, @xcite utilized both few-shot learning and fine-tuning to control grammatical complexity in educational text generation, finding that fine-tuned mistral yielded the best results among tested models. other studies have relied solely on prompt engineering. bektaş et al. (2024) demonstrated that chatgpt-4o could generate simplified turkish texts comparable to humansimplified versions using zero-shot prompting with detailed rules. imperial and madabushi (2023) found that open-source models like bloomz and flant5 were more effective than chatgpt for readability-controlled text generation, though the latter improved with refined prompts. the research is mostly focused on english, but bektaş et al. (2024) worked with turkish and spring and rios (2021) explored text-simplification for german. research on italian materials remains more limited, though franzoni et al. (2024) developed morpho-phraseological model for classifying the cefr level of italian l2 learner writing. a critical question is to what extent llms can apply cefr-based constraints during text generation. @xcite conducted a comprehensive assessment of various models' cefr actionable knowledge-a term they introduce to describe the models' ability to operationalize cefr categories-across text classification, essay scoring, and student simulation tasks. their findings reveal that while llms demonstrate partial cefr knowledge, they often struggle to apply it accurately in practical tasks. among the models evaluated (including gpt-3.5, gpt-4o-mini, mistral, gemma, and llama 3), gpt-4o-mini performed best overall, though all models showed inconsistencies. @xcite evaluated chatgptgenerated reading texts from a systemic functional perspective, finding that text complexity often failed to align with claimed cefr levels. their analysis suggested that chatgpt's output complexity correlated more strongly with text length than with appropriate lexical or grammatical calibration for specific cefr levels. similarly, uchida (2025) specifically examined chatgpt-4o's ability to generate texts at specified cefr levels, discovering low accuracy-only 5% of generated texts matched their target cefr level according to computerized analysis. this study highlighted a tendency to oversimplify a1 and a2 level content, suggesting challenges in maintaining appropriate complexity for lower proficiency levels. ## instruments the two instruments used in the work are the cefrbased inventories for reading/writing competences (that is, the grammar rules that should be mastered at a1 proficiency level), and the vocabulary lists (that is, the words that should be known at a1 level), for english and for italian. @xcite explored prompting strategies that referenced cefr levels either by only naming them or through brief descriptions of the communicative abilities expected at each level. their findings suggest that such high-level prompts may be insufficient to guide models toward consistent level-appropriate output. to address this issue, we turned to detailed, language-specific cefr inventories in order to provide models with more concrete grammatical and lexical constraints for english and italian. for english, we rely on the council of europe breakthrough inventory @xcite . for italian, two inventories were considered: the comprehensive profilo della lingua italiana @xcite and the guidelines for the cils exam @xcite . the latter was chosen for its compactness and practical utility. the original constraints were categorized by part-of-speech, specifying both the permitted and prohibited forms for each category. we made a slight adjustment to the italian constraints, since there was a discrepancy in irregular verbs usage between the @xcite guidelines and their sample exam version (università per stranieri di siena, 2017). to address this, we included irregular verbs both from guidelines and from exam sample in the final constraints list. the resulting lists of grammatical constraints for english and italian are available in appendix a. ## procedure the task consisted of requesting different llms to paraphrase sentences with four different prompting strategies. the paraphrases have then been controlled against the inventories and the vocabulary lists in an automatic way. for a small subset, a manual check on linguistic soundness and meaning preservation have also been performed. ## results starting from our vikidia sentences dataset, 100 samples within a length range of [240-290] characters were randomly selected for english (mean @xmath0.72 chars, @xmath1.796) and italian (mean @xmath2.79 chars, @xmath3.682). this sampling approach ensured comparable test datasets across both languages. llm-driven paraphrasing was performed using three different models: gpt-4o (gpt-4o-2024-11-20), gpt-4o-mini (gpt-4o-mini-2024-07-18), and llama-3.3-70b. all four paraphrasing strategies were used for each model. vocabulary coverage checks and automated grammar analysis were performed on the entire data sample, while manual annotation was performed only on a sub-set of 30 sentences (15 per language). for statistical analysis, we focused on three automated metrics available for all samples: linguistic difference, a1 vocabulary coverage, and constraint violation counts. for both llm-driven paraphrasing and automated grammar analysis, we opted for parameter settings of @xmath4.0 and @xmath5.0. we found these settings appropriate for consistent results in linguistic analysis tasks. for further statistical analysis and tests, we decided to partition the collected data based on input language. while the automated analysis process is identical between our two languages, the a1 cefr aligned linguistic constraints selected (see appendix a) differ significantly and do not overlap across parts-of-speech. to evaluate for significant differences between paraphrasing strategies and model choices, the kruskal-wallis test was performed on the metrics of linguistic difference (diff.), a1 vocabulary coverage (a1 coverage %), and the count of violated constraints (errors). this statistical test was found appropriate for our key metrics, as none of them present a normal distribution or homogeneous variance across groups. post-hoc pairwise comparisons were conducted using dunn's test, with effect sizes measured using rank biserial correlation and classified as small (*), medium (**), or large (***) based on standard conventions for interpreting non-parametric effect sizes. tables 2 and 3 present the comprehensive results of our analysis. error counts showed statistically significant differences between strategies for both languages (@xmath6.18, @xmath7.001 for english; @xmath8.70, @xmath9.001 for italian), with more pronounced effect sizes for italian, but only showed marginal significance between models for english (@xmath10.35, @xmath11.042) and no significant differences between models for italian (@xmath12.35, @xmath13.114). ## discussion our results provide some evidence that guided paraphrases can improve compliance with a1 inventories and vocabulary lists, although not perfectly and with high cost (in terms of processed tokens). the paraphrases seem to be more effective for italian than for english (but our original sentences tended to be less compliant in the italian dataset than the english one). for what concerns the prompting strategies, although the results have medium to small statistical effect size, it seems that for english the strategies that include cot work better in reducing the grammatical violations, while those with two steps cover better the vocabulary. therefore the best choice would be the strategy that combines cot with the two-steps. however, the best choice is not so clear for italian because the one-step strategies seem to have a better effect on grammatical constraints violations while the two-steps seem to cover better the vocabulary, with an unclear effect of the cot. this contrast might be due to the fact that grammatical constraints for italian are more restrictive than those for english. as a result, when vocabulary simplification is applied as a separate step, in case of italian there is more risk to inadvertently introduce constructions that violate earlier grammatical restrictions. therefore, for italian there is no best strategy. as for the models, llama seems to better cover the vocabulary for both english and italian. on the grammatical violations, there is no difference among the models for italian, while for english gpt-4o and gpt-4o-mini seem to perform better than llama. although our approach can still be improved, our results may contribute to a better understanding of the task of language adaptation to cefr constraints using llms. previous studies have highlighted the limitations of relying solely on general cefr-level prompts. @xcite note that communication-based descriptors from the cefr companion volume (council of europe, 2020) are not specific enough to effectively control language model output. similarly, in the study by @xcite , prompting with only the target cefr level led to oversimplified a1 and a2 texts. these limitations suggest the need for more fine-grained control. like @xcite , who successfully used zero-shot prompting with detailed rules to generate simplified turkish texts, we adopted a more detailed constraint-based approach. however, while the method of @xcite was grounded in expert-written simplification guidelines, our constraints were derived from language-specific cefr inventories by @xcite and @xcite . using detailed inventories along with a task to paraphrase the text maintaining as much information as possible might be the reason why in our case outputs tended to exceed the intended constraints boundaries. although the language specific grammar inventories are not easy to formalize for prompt usage and to check against, they are the base for learning and assessing foreign language competence, and it is important for intelligent tutoring system to control those constraints. an important aspect that deserves further investigation might be the counterintuitive results on the relationship between the complexity of the paraphrases and the constraints compliance (which is more pronounced for italian but is also partially present for english). it can perhaps be partially due to the reduced competence on italian of these models. yet, it has to be noted that the task of paraphrasing to beginner cefr levels is complicated for humans too. indeed, torregrosa benavent and sánchez-reyes peñamaría (2011) note that authentic materials are difficult to adapt for beginner learners, often requiring extensive rewriting and @xcite suggest writing reading assessment texts from scratch or adapting existing texts freely, rather than trying to preserve all the original details, in order to better adhere to specific linguistic requirements, like those that we have imposed on the models we tested. ## release all the data and source code used for our experiments is released on github foot_3 under creative commons attribution 4.0 international license (cc-by-4.0) for the data and apache 2.0 for the code. the package contains: (i) the data downloaded from vikidia in english and italian; (ii) the sentences selected for our experiments, with original and rephrased versions, along with manual annotation of meaning preservation and linguistic soundness, and automatic tagging of compliance; (iii) the python source code used to run the experiments. ## conclusion our study shed some light on the problem of constraining llms output to a specific level of language competence, by exploring in particular cefr a1 constraints. specifically, we explored the possibility of paraphrasing the text with testing the combination of four strategies with three different llms. furthermore, we investigated both english and italian. our results provided some evidence that the compliance can be improved, although our approach did not reach a satisfactory level since the number of grammatical errors and lack in vocabulary coverage are still high for all the strategies and for both english and italian. still we believe that, given the relevance of the task for improving intelligent tutoring systems for language learning, the paraphrasing approach, although complex, is worth investigating. our results are a promising first step. finally, another relevant outcome of our work is a dataset of cleaned pairs of sentences and paraphrases (in two languages) on which further work could be based.