From Tarzan to T olkien: Controlling the Language Proficiency Level of LLM s for Content Generation (2024)

Abstract: We study the problem of controlling the difficulty level of text generated by Large Language Models (LLMs) for contexts where end-users are not fully proficient, such as language learners. Using a novel framework, we evaluate the effectiveness of several key approaches for this task, including few-shot prompting, supervised finetuning, and reinforcement learning (RL), utilising both GPT-4 and open source alternatives like LLama2-7B and Mistral-7B.Our findings reveal a large performance gap between GPT-4 and the open source models when using prompt-based strategies. However, we show how to bridge this gap with a careful combination of finetuning and RL alignment. Our best model, CALM (CEFR-Aligned Language Model), surpasses the performance of GPT-4 and other strategies, at only a fraction of the cost. We further validate the quality of our results through a small-scale human study.

## introduction large language models (llms) are powerful tools for content generation. however, these models often output text at a native level of speech (figure 1 @xcite . this makes llms challenging to use for contexts where the end users are not fully proficient, such as for language learners, young children, or non-native speakers. when generating content for these use cases, we need the ability to control the proficiency level of the generated text. in this work, we formally define the proficiency control task (pct): a new framework that assesses a model's ability to modulate language proficiency level, while also generating high-quality content consistent with given instructions. we evaluate models with respect to the three essential criteria: (1) controlerror which measures how close the generated text is to the target proficiency, (2) qualityscore, which measures relevance of the text to the instructions, and (3) cost which measures the resource-intensiveness of the approach. using this evaluation framework and the tinystories dataset @xcite , we investigate several key approaches to the pct on the task of short story generation from a plot summary. prompt-based approaches first, we thoroughly explore the space of few-shot, prompt based strategies with openai's gpt-4 and open source alternatives (section 6). our findings demonstrate the strong capability of gpt-4 at the pct, resulting in low controlerror and high qualityscore. we also identify an improvement in controlerror as prompts are made more complex, resulting in better proficiency control at the cost of more tokens. although gpt-4 is successful at the pct, it is a proprietary model and its generations are several times more costly than open source alternatives. however, we find instruction-tuned models like llama-2-7b and mistral-7b perform poorly at the pct through prompting. finetuning open source models to bridge the gap between open source models and gpt-4, we turn to supervised finetuning approaches from the controllable text generation literature @xcite . specifically, we use the outputs of an effective gpt4 prompting strategy to generate data for the pct that can be used to directly train open source models. using this data, we are able to finetune llama2-7b and mistral-7b to come significantly closer in performance to gpt-4 at the pct (section 7). moreover, we show how additional training with proximal policy optimisation (ppo) can further align the outputs of these models with the desired proficiency levels. our best such model, we call calm (cefr-aligned language model), has a controlerror equal to that of gpt-4, at only a fraction of the cost. boosting pct models through sampling finally, in section 8 we present a simple but powerful sampling strategy that allows us to boost any pct model to one with arbitrarily better controlerror , albeit at a higher cost. with this technique, we are able to show that calm is a strictly dominant strategy in the pareto sense compared to gpt-4 with any kind of prompting. we run a small-scale human evaluation (section 9) to further validate the quality of generations from calm and gpt-4 with prompting. the generations of both models are highly rated in terms of quality (≈ 4.7 out of 5). we also show that our measure of controlerror aligns closely with human perceptions of "proficiency level". ## the proficiency control task we now formally define the proficiency control task (pct), which measures a model's ability to generate content relevant for a given prompt while also controlling the proficiency level of its output. formally, let σ * denote the set of strings. let @xmath0, 2, 3, 4, 5, 6} be a target proficiency (corresponding to each cefr level). we denote a proficiency control model as a function m : (σ * × {1, 2, 3, 4, 5, 6}) → σ * that takes a prompt and target proficiency as input and outputs a generated text for the given prompt. we assess the pct on three key criteria: control this evaluates how close the generated text was to the target proficiency level. let s cefr : σ * → r be an automatic proficiency scoring function. we define the controlerror between a target proficiency t and a generated text @xmath1 quality this measures the relevance and quality of the generated content to the given prompt. for example, if the prompt asks for an english story with a certain plot, then the text should be in correct english and closely align with the given plot. cost this measures how expensive the control strategy is with respect to various resources e.g. flops, time, or dollars. our primary resource of interest for llms will be flops, which are a function of the size of the model and the number of tokens used by the strategy. ## strategies for proficiency control in this section we discuss several approaches to proficiency control for llms. these approaches are broadly categorised into prompt-based techniques, supervised finetuning on a pct dataset, and a general sampling strategy to improve any pct model. ## experimental setup to experiment with the different proficiency control strategies, we run an experiment using the tinystories dataset @xcite , which is a collection of english short-stories that also includes a plot summary for each story (cdla-sharing-1 license). using this data, we construct the following task: a model is given the plot summary of a story as well as a uniformly random target cefr level from 1 to 6. the model is then asked to generate a short story (around 3-5 paragraphs) which adheres to the given plot and also sits at the target level. we select a subset of 50 random story plots from the tinystories dataset to evaluate on. see appendix b for all training details. foot_0 ## results: prompt-based approaches in ## distilling gpt-4 for open source the high qualityscore but low controlerror of the open source models suggests that they are quite capable at story generation, but lack the ability to be steered through prompting. a promising path forward is to directly finetune these models for controllable cefr generation. following a similar idea to tinystories, we investigate whether gpt4's effectiveness at the pct can be leveraged to improve the open source models. ## human evaluations in order to further validate our results from the automatic cefr scorer and the gpt-4 based evaluation of quality, we ran a small human evaluation. we recruited 13 volunteer participants from peers and colleagues to do a blind evaluation. we gave the volunteers two tasks. ## conclusion we present a new challenge for controlling the proficiency level of llm generated content: a highly practical and important task for practitioners in the domain of education and/or language learning. we demonstrate effective strategies for generating at a desired proficiency level, using both proprietary models such as gpt4 and open source techniques. through a careful cost analysis, we show that our calm model is dominant in terms of cost and performance, and generates content rated by humans to be of high quality. we release this model as well as a synthetic toy dataset called tinytolkien for future use in proficiency control research.