Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models (2023)

Abstract: Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of educational materials before administering them for classroom use. In this study, we select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives—tasks that teachers perform—using standard-guided prompts controlling text readability. Our extensive findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks compared to other open-sourced models such as BLOOMZ and FlanT5—which have shown promising results.

## introduction the introduction of public-facing text generative models with easy-to-use interfaces, such as chat-gpt by openai, perplexity ask by perplexity ai, and bard by google, has catalyzed the research progress of large language models (llms) that can follow and execute complex instructions in human language. this particular advantage over regular language models has seen a rapid growth of appreciation and utilization across a number of disciplines and sectors, such as medicine and healthcare @xcite , teaching and assessment in education @xcite @xcite , business and e-commerce @xcite , and software development @xcite @xcite to name a few. one of the primary drivers of this advancement in llms is instruction tuning. this process involves finetuning an llm on a diverse collection of multi-task corpora transformed in an instructionanswer pair format, which in turn allows the model to learn and improve upon tasks it was not trained on @xcite . in the same vein, other advancements explored the involvement of human raters where a reward-driven language model learns from the aggregated preferences and is incentivized through reinforcement learning if its generated content from a series of executed instructions is acceptable @xcite . these training methodologies, in essence, allow llms to have some form of knowledge in relation to what aligns with humans and bridge the gap between the llm-oriented goal of next token prediction and a user-oriented objective. likewise, specifications from various instruction-answer corpora act as signals of constraint to control a model's output @xcite . however, one of the main research gaps that these powerful instruction-following models may need to be rigorously tested with is the ability to capture human standards. standards or domainspecific frameworks are expert-defined sets of rules that humans follow in various interdisciplinary fields. for example, a teacher must be properly knowledgeable of assessment standards such as the common european framework of reference for languages (cefr) for evaluating the quality of text-based educational content before they can use it in a classroom setting @xcite . therefore, if llms such as chatgpt are to be utilized to generate educational content for the teacher, then it would be ideal for these models to be evaluated or trained based on how they accept inputs, such as prompting or finetuning, to acquire some form of knowledge of how cefr works and how it is used to assess the quality of texts. in this work, we tackle the main research question: to what extent can instruction-tuned large language models capture readability level specifications from prompts and reflect it to the generated content? towards this end, our major contributions are as follows: ## prompt-based story completion our first choice of generation task to measure the generation quality of instruction-following language models is the open-ended story completion. we selected this task as it aligns with the natural task of teachers prompting language model-driven interfaces such as chatgpt for educational content generation such as stories or short narratives @xcite . ## related work the majority of literature on evaluating instructiontuned models has spotlighted chatgpt due to its global recognition amongst interdisciplinary fields. specifically, these evaluation works have focused on aspects such as multilinguality @xcite @xcite , reasoning @xcite , truthfulness @xcite , toxicity @xcite to name a few. in terms of incorporating forms of control to guide generations, related works have explored style @xcite , tone @xcite , topic coherence @xcite @xcite , sentiment and emotion @xcite , and text complexity @xcite @xcite . the main gap in literature that our study fills is the evaluation of llms and their alignment with realworld text assessment standards used by teachers, such as the cefr framework. ## conclusion in this work, we tackled a unique perspective of evaluating the capabilities of instruction-tuned language models by integrating readability-specific information anchored on realistic assessment standards such as the cefr framework used by teachers and educators. our findings expose the advantages and weaknesses of open and closed-source generative models such as llama, flant5, and chatgpt for the story completion and narrative simplification tasks, in which we trace back each model's performance to the quality of instruction datasets used for finetuning them. we hope this study sheds light on both the technical and nontechnical audiences, especially the members of the education community, regarding the true capabilities of these generative models in producing educational content.