Towards Fine-Grained Pedagogical Control over E nglish Grammar Complexity in Educational Text Generation (2024)

Abstract: Teaching foreign languages and fostering language awareness in subject matter teaching requires a profound knowledge of grammar structures. Yet, while Large Language Models can act as tutors, it is unclear how effectively they can control grammar in generated text and adapt to learner needs. In this study, we investigate the ability of these models to exemplify pedagogically relevant grammar patterns, detect instances of grammar in a given text, and constrain text generation to grammar characteristic of a proficiency level. Concretely, we (1) evaluate the ability of GPT3.5 and GPT4 to generate example sentences for the standard English Grammar Profile CEFR taxonomy using few-shot in-context learning, (2) train BERT-based detectors with these generated examples of grammatical patterns, and (3) control the grammatical complexity of text generated by the open Mistral model by ranking sentence candidates with these detectors. We show that the grammar pattern instantiation quality is accurate but too homogeneous, and our classifiers successfully detect these patterns. A GPT-generated dataset of almost 1 million positive and negative examples for the English Grammar Profile is released with this work. With our method, Mistral’s output significantly increases the number of characteristic grammar constructions on the desired level, outperforming GPT4. This showcases how language domain knowledge can enhance Large Language Models for specific education needs, facilitating their effective use for intelligent tutor development and AI-generated materials. Code, models, and data are available at https://github.com/dominikglandorf/LLM-grammar.

## introduction the arrival and accessibility of well-performing large language models (llms) created a flood of applications in personalized education for tutoring and material creation @xcite . despite their ability to follow instructions, it is underexplored to what extent prompting can systematically affect the linguistic properties of the generated output to satisfy educational needs. if llm-generated text was finely adjustable regarding the grammatical constructs used, personalized and engaging learning materials could systematically support learners' language development by exposing them to the optimal linguistic complexity @xcite . this control would enable a stronger connection to input-oriented theories of language acquisition. due to their data-driven nature, llms' grammatical knowledge has to be empirically examined. on the one hand, they have been successfully used for text simplification and grammar construction detection @xcite . on the other hand, transformer models still benefit from explicit syntactic information during training @xcite . because of missing labeled training data and systematic evaluations, it is uncertain to what extent neural text generation can be controlled for the presence of a comprehensive set of pedagogically relevant and teachable grammatical constructions. this work pursues the questions of how well llms can create valid examples for grammar constructs (rq1), how well bert sentence embeddings represent these grammar constructs (rq2), and how well text generation can be controlled for these constructs (rq3). we build on an empirically established and validated taxonomy of english grammar, the english grammar profile (egp) (o' @xcite , precisely characterizing the development of english across the proficiency spectrum with 1,222 grammar patterns. we first evaluate how well gpt3.5 and gpt4 can generate positive and negative instances on a subset of the egp (rq1). we then alleviate the lack of examples by automatically creating 946k labeled example sentences for all entries of the egp, which we make available to the public. this unique dataset serves to fine-tune and evaluate bert-based classification models on detecting examples of the egp's grammar patterns in sentences (rq2). using these models, a grammar-controlled text generation approach to strategically decoding an open pre-trained llm, mistral-7b, provides a proof of concept with 600 generated texts (rq3). to generate them, we sample multiple candidate sentences at inference time and rank them by the grammar patterns detected by the classifiers. we show that the accuracy of generated instances of grammar patterns is 87.1% with gpt3.5 (92.9% with gpt4), and the classifiers distinguish the positive from negative examples in our generated dataset with an average accuracy of 95.1%. the grammar-controlled text generation approach at least doubles the grammatical constructions on each level of the standard common european framework of reference for languages (cefr)({council of europe}, 2020). going beyond the specific task, our work highlights how explicit domain knowledge relevant to language learning and broader language-sensitive educational contexts can be fused with the versatility of llms. it is a step towards better control over a powerful tool compared to pure prompting. the approach can readily be extended to other pedagogically desirable attributes of llm-based tutors and educational material. ## method our approach comprises validating the egp instantiation capabilities of state-of-the-art llms, training neural rule detectors on a generated large-scale grammar dataset, and using these rule detectors to rank candidates when sampling from an open text generation model. the analysis was conducted with standard python libraries for natural language processing and deep learning on up to 16 nvidia geforce rtx 2080 ti gpus provided by the computing cluster of the university of tübingen. the code and data are available on github foot_0 . seeds are provided for reproducibility. ## . { p r e v i o u s prompt and r e s p o n s e } r e w r i t e e a c h c r e a t e d e x a m p l e a s a m i n i m a l p a i r t h a t d o e s n o t show t h e u s a g e o f t h e g i v e n r u l e . using regular expressions, the model responses are parsed based on the enumeration, cleaned from prefixes and explanations in parenthesis, and cleared from repetitions of the positive examples in the case of negative instances. the presence_penalty parameter that penalizes repetitions of tokens during sampling from the model was increased to 0.5 for the initial prompt to diversify the vocabulary within one response. the model temperature that makes the output more random for higher values was decreased to 0.5 for the negative prompt to favor correctness over diversity. this assumes that there is only a small number of possible modifications to make a positive example negative and therefore the sampling should favor the most likely tokens. the egp may or may not be part of the training set of openai's models. even if this is the case, it remains unclear how well they can transfer the patterns to a wider range of topics and sentence meanings than the few included examples. for a small-scale quality assessment (before generating the large dataset in the next step), 36 egp patterns are randomly drawn, stratified by cefr level and type, and the two models generate each 20 (in two batches of 10) positive and 20 negative examples each, resulting in 2,880 examples. the set of sentences is hand-coded on whether they include the intended grammar pattern or not in a blinded manner, i.e., without knowing the model or intended label. these labels serve to calculate the models' accuracy. an automatic evaluation based on the rouge and bleu scores assesses how close the negative examples are to the most similar positive example. the rouge-1 score (ranging from 0 to 1) reflects the number of common unigrams between a text and the set of reference texts, measuring lexical similarity. the bleu score is in the same range but focusses more on precision instead of recall and also takes longer subsequences into account. furthermore, the average cosine similarity of embeddings with the recent ember-v1 model 4 between all positive example sentences and 4 https://huggingface.co/llmrails/ember-v1 between all negative sentences is calculated per egp pattern and compared to the baseline of the renowned brown corpus @xcite . to improve the diversity of negative examples, positive examples from other egp entries are mixed in, assuming that these do not contain the pattern. ## discussion and conclusion this work showcases how large language models can be controlled based on the qualitative egp augmented to a large-scale dataset to align with pedagogical use cases, specifically -but not limited to -language teaching. we first verified the sufficient quality of llm-generated instances of an established grammar repository, the english grammar profile. the validation emphasizes the strength of the most recent closed-source model, gpt4. nevertheless, the quality of instances generated by gpt3.5 could almost keep up with the flagship model. because of this positive finding, we generated 946k labeled grammar construction examples, which we publicly share for further research. the binary grammar construction classification on this data shows satisfying results within the distribution @xcite . controlling an open llm such as mistral on the used grammar constructions with these classifiers significantly affects the frequency of desired grammar patterns. it can even beat the baseline of prompting gpt4. while the prompt-based strategies already improve over the static baseline for most cefr levels, our proposed approach has improved text on every proficiency level and at least doubled the default frequency of constructs on all levels. it also solves the shortcoming of @xcite that had difficulties generating text on the simpler levels a1 and a2. with the advent of performant open llms, such as llama3 and mixtral of experts, educational applications can be further tailored to align with pedagogical expectations than with prompting alone. currently, instructors can only use commercial model interfaces such as chatgpt or third-party wrappers around the model endpoints. our method advances the possibilities from prompt engineering approaches to fine adjustment of the model output. we freely release our augmented dataset and the trained grammar classifiers to provide learning engineers with a tool to introduce this level of control to their applications. a possible application is adjusting the grammatical complexity of an ai tutor in science to the language proficiency level of each student. non-native speakers in the same classroom can interact with the seemingly "same" agent that adapts its language to them under the hood. language instructors can use models to generate texts of students' interests while ensuring the use of particular grammar that aligns with their curriculum.