Chain-of- M eta W riting: Linguistic and Textual Analysis of How Small Language Models Write Young Students Texts (2025)

Abstract: Large Language Models (LLMs) have been used to generate texts in response to different writing tasks: reports, essays, story telling. However, language models do not have a metarepresentation of the text writing process, nor inherent communication learning needs, comparable to those of young human students. This paper introduces a fine-grained linguistic and textual analysis of multilingual Small Language Models’ (SLMs) writing. With our method, Chain-of-MetaWriting, SLMs can imitate some steps of the human writing process, such as planning and evaluation. We mainly focused on short story and essay writing tasks in French for schoolchildren and undergraduate students respectively. Our results show that SLMs encounter difficulties in assisting young students on sensitive topics such as violence in the schoolyard, and they sometimes use words too complex for the target audience. In particular, the output is quite different from the human produced texts in term of text cohesion and coherence regarding temporal connectors, topic progression, reference.

## introduction recent llms have proven some performance in generating different types of texts such as summaries @xcite , essays @xcite , or short stories @xcite . however, llms still struggle with keeping the same meaning overall during summarization, as shown by an australian governmental study 1 . in an educational context, some studies showed that llms can be used to help students to deepen learning or help with scoring and feedback @xcite @xcite . moreover, openai proposes a guide dedicated to help students use chatgpt for their writing assignments, suggesting that the llm can give 1 https://archive.is/itqbm iterative feedback to improve their productions foot_0 . while human students think, plan, produce, and revise their written production, as illustrated by psycholinguistic and communicative models @xcite , llms do not have an inherent writing process meta-representation (as illustrated in figure 1 ). as suggested by @xcite , llms may be used to generate wellformed written texts, provided that the content (data, specific information, etc.) is produced by the user her/himself 3 . however, this option might be operational for the performance-goal productions only, and not for learning-goal production, unless used as a source of (re)formulation variants. some other differences concern the lack of revision/rewriting processes for llms, and stereotyped, monovalent communication situation (adult to adult, and involving a generation task). llms can, to some extent, imitate human tasks with chain-of-thought (cot) prompting techniques @xcite . cot step-by-step prompting has proven useful for reasoning tasks, such as close or open domain reasoning @xcite @xcite , code generation @xcite or mathematical reasoning @xcite . a different approach to cot, called metacognitive prompting @xcite , showed improvement in llms "understanding" of their rationale in different qa tasks. in this paper, we present a fine-grained linguistic and textual analysis of language models (lms) exploitation to accompany primary to middle school level students, as well as undergraduate students, in the writing process 4 . in fact, essential components of the writing process, such as planning and revision, are not part of the lm's text generation process (figure 1 ). to tackle this issue, we introduce chain-of-metawriting (comw), a prompting framework that illustrates a step-by-step writing production rationale, involving planning, revision and feedback, thus imitating the human cognitive and metacognitive process of writing. following recent research directions, we tested three open source multilingual small language models (slms) of 3b parameters, llama-3.2 @xcite , qwen-2.5 @xcite , phi-3.5 @xcite , and one proprietary model, chatgpt-4o mini @xcite . we chose slms because they are adapted for in-device application (mobile phones), while having faster inference and low computational cost. we evaluated open-source slms to ensure the reproducibility of our study @xcite . for our experiments, we tested to what extent slms can imitate higher levels of writing, such as thinking, planning, linguistic expression, editing, and revision. we investigated whether a slm can help a 10 to 12-year-old or an undergraduate student learn how to write a text in french as a mother tongue, while adapting to the student's linguistic expertise. due to lack of space, we have not detailed all the analyses. we decided to take 4 the primary to middle school students whose written assignments were used in this study are around 10 to 12 years old. a closer look at 10-12-year-old's writing, which is not addressed in the research on text generation. moreover, language models are known to lack the personal experience that schoolchildren employ in their narratives which represents an additional challenge, also given the limited learning corpus produced by this age group. the contributions of this paper are threefold: ## related work llms generate language based on algorithms trained on very big corpora of textual data. the ingredients of llm generated texts are therefore derived from texts produced by humans, whose layout rules have been identified and applied. researchers investigated to what extent llm-generated and human-produced content are different or comparable. several types of texts were compared: hotel reviews @xcite , scientific texts (casal and kessler, 2023), narrative texts @xcite , argumentative essays @xcite . to evaluate the differences between the llm and human texts, several types of features were exploited in the literature: intrinsic features, like the proportion of different pos, punctuation, linguistic diversity, style, structural features (readability), affective/evaluative markers, content's nature, and impact on the receivers, like helpfulness (casal and kessler 2023; @xcite ). ## method our global framework is illustrated in figure 2 . we tested and evaluated different types of prompts: the first two prompts aimed to generate texts comparable to those produced by the schoolchildren. the third prompt aimed to test the potential of the lm to be incorporated into teaching, helping students learn how to use lms effectively to edit texts, how to evaluate subtle differences in style and content, and how to determine whether an assertion is supported by evidence @xcite . the fourth, chain-of-metawriting (comw), is the prompting framework we propose to test whether lms can simulate high level writing. figure 2 shows each type of prompt with the generated answer: auto-censorship, surpassing censorship, list of steps to write a text, and, with comw, the expected output, a text written as a schoolchild. we discuss results in section 4. we tested three multilingual and open-source slms of the same size, 3 billion parameters (3b), that were pre-trained on french data, among other languages: llama-3.2 @xcite , qwen-2.5 @xcite , phi-3.5 @xcite , and a proprietary model, chatgpt-4o mini @xcite . to assess the models' multilingual capacities, we used two languages for our prompts: english and french. thus, we evaluated the slms performance with prompts in a different language than english, to test its multilingual and cross-lingual performance @xcite . ## results and discussion our results show that slm might produce texts which might be considered as human-generated texts at the first glance. a detailed analysis of the generated output show some degree of incoherence, inconsistent output, and use of words too complex for the target audience. the explanation of the comw prompt are not always convincing. we detail each analysis below. ## writing process analysis in order to track the difficulties encountered by schoolchildren during the writing process, we analyzed the contexts of long pauses (above 7 seconds, twice the average threshold). with a salient threshold at 10.5 seconds, we observe that r-bursts tend to be produced after longer pauses, (whereas as pause length increases, the number of rb-bursts and p-bursts decreases). as previously noted on comparable data from schoolchildren narrations, @xcite , most of the r-bursts are lexical reformulations, (typo) error correction, deletions reorienting the incipit, with very little syntactic and merely no textual-level revision. we sought to identify the relevant events characterizing the other two types of bursts: p-bursts and rb-bursts. for the first two intervals, pauses before pbursts more often concerned the production of intrasentence connectors (and their continuation) than sentence starters. while strong punctuation attracts long pauses, the presence of weak punctuation shows, along with the previous observation, that it's informational segmentation rather than syntactic segmentation that is problematic in the writing of narratives by schoolchildren. more than half of rb-bursts involved complete deletions, sometimes of quite long segments, up to several dozens characters. nearly a quarter of rb-bursts applied to immediate revisions to follow on from new beginnings (mainly lexical and referential choices, see tables 8 and 9 , appendix c). this seems to indicate a genuine focus on content. to sum up, the stumbling blocks in children's narrative production were the textual segmentation of information on the one hand, and the shaping of content on the other. in both cases, these are stages that are totally absent from the comw prompt, in addition to the revision stage mentioned above. ## undergraduate students' writing generating texts following the writing instructions for the students resulted in products around 25-40% longer than the texts produced by the students. in terms of textual format, we identified a number of peculiarities in the generated texts, such as the presence of "waffle-language" sequences, i.e. segments that do not allow to identify a precise referent relevant to the context: this is the case, for example, of "intensity of educational content" and "promoting academic security". similarly, texts may contain factual errors due to lack of understanding and the probable unavailability of reliable textual data on the subject of wi-fi jammers: for example, qwen2.5 promotes jammers as tools for improving connections, while their installation in universities is legally forbidden in france. the meta-chain on the subject of reducing greenhouse gas emissions at airports includes a section on the definition of the "airport" object and documentation on the aeronautical activities that produce these gases. the rest of the chain consists of the introduction, the body of the text and the conclusion. the body of the text is structured in terms of findings-measures-positive effects of measures, and it is advisable to use examples and facts. two interesting features are worth mentioning: i) the argumentative focus on positive effects only; ii) the second part of the conclusions calling for action and/or positioning. ## conclusion and further work we proposed a fine-grained analysis of the role of slms in the content generation and writing process, and a new writing framework, chain-of-metawriting (comw). we evaluated the potential benefits, dangers, and limitations of slms as writing aids for schoolchildren. results showed that slms produce texts that in some respects are too far from expectations, which is why they cannot be recommended as models for learning to write. one of the outcomes of our study is the necessity to provide accessible and exploitable rules and schema for text composition. we conclude that llama3.2 is the most performant, surpassing chatgpt-4o mini. further work could include analyzing the impact of synthetic data in lms, as phi-3.5 was trained on natural and synthetic data. we could explore lm test-time computation to mimic human students writing conditions @xcite .