Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts (2024)

Abstract: Using large language models (LLMs) for educational applications like dialogue-based teaching is a hot topic. Effective teaching, however, requires teachers to adapt the difficulty of content and explanations to the education level of their students. Even the best LLMs today struggle to do this well. If we want to improve LLMs on this adaptation task, we need to be able to measure adaptation success reliably. However, current Static metrics for text difficulty, like the Flesch-Kincaid Reading Ease score, are known to be crude and brittle. We, therefore, introduce and evaluate a new set of Prompt-based metrics for text difficulty. Based on a user study, we create Prompt-based metrics as inputs for LLMs. They leverage LLM’s general language understanding capabilities to capture more abstract and complex features than Static metrics. Regression experiments show that adding our Prompt-based metrics significantly improves text difficulty classification over Static metrics alone. Our results demonstrate the promise of using LLMs to evaluate text adaptation to different education levels.

## introduction large language models (llms) today can answer wide-ranging questions and explain complex concepts with high accuracy @xcite . this development has motivated explorations into their uses for education, ranging from automated student assessment and personalised content to dialogue-based teaching @xcite @xcite . effective teaching requires that the difficulty of content and explanations is tailored to the education level of the students. human teachers are trained to do this, and adjust their material and style without much prompting. however, this adaptation is not just the adjustment of one variable. it is a complex undertaking, touching upon lexicon, syntax, figure 1 : schematic overview of our approach to text difficulty classification. we calculate relevant static and prompt-based metrics for a given input text. either or both metrics are then fed into a regression classifier that makes a final classification. measuring whether a given output is appropriate for a given level of education, however, is a very difficult task. existing static metrics, like the flesch-kincaid reading ease score @xcite , are based on simple formulas, heuristics, and word counts. they share the brittleness of all heuristic approaches and are known to be noisy measures of text difficulty at best. also, these metrics were developed for longer-form explanations, like those found in textbooks, rather than dialogue-style teaching. due to their reliance on counts, their estimates are unreliable in shorter formats. we need better metrics to make improvements on the adaptability of llms to education levels measurable. only when we can measure improvements can we make tangible progress in leveraging llms for educational applications. 1 as an alternative to static metrics, we can use classifiers to predict the educational level of a given text. they generalize better and can be applied to texts of varying lengths. however, these classifiers are expensive to train and require more training data than we usually have for a niche domain like educational purposes. similarly, human assessment of difficulty may provide a gold standard, but it is expensive to collect and, like all annotation tasks, suffers from disagreement. in this paper, we introduce and evaluate a new set of prompt-based metrics for text difficulty as complements to existing static metrics. prompt-based metrics are llm prompts that exploit the general language understanding capabilities of llms to capture more abstract features of educational texts than static metrics. for example, llms can flexibly classify the topic of a text, which is one adaptation technique used by teachers to adjust the content which called curriculum compacting in pedagogy @xcite . this would be difficult to do with static approaches. we develop our selection of prompt-based metrics based on a user study, where we ask a group of university students to 1) assess the difficulty of educational texts and explain their reasoning, and 2) come up with prompts for an llm to change the difficulty of a given text. we then translate the qualitative findings from both parts of the study into concrete llm prompts that serve as prompt-based metrics. we incorporate prompts from other studies to manage text readability with llms @xcite . we evaluate the ability of our new prompt-based metrics to measure text appropriateness for different education levels with a series of regression experiments. while prompt-based metrics perform on par or better than zero-shot and few-shot llm classifiers, they are less useful for text difficulty classification by themselves than static metrics. how-ever, combining prompt-based and static metrics significantly improves performance. this suggests that prompt-based metrics capture relevant signals beyond those captured by the large number of static metrics. a combination of static and prompt-based metrics also provides a deeper understanding of the key metrics or features that influence complexity than classifiers could. additionally, the factors that contribute to complexity in a scientific text differ from those in a medical or a legal document. by considering a range of metrics, we can develop more accurate domain-specific measures. our multifaceted approach allows us to break down complexity into its basic components, such as its appropriateness for different education levels, lexical or syntactic complexity, thematic topics, and text readability. overall, prompt-based metrics empower educators to develop more effective content development strategies with llms to engage learners of all levels and backgrounds. we could have directly trained classifiers; however, this approach would not have enabled us to identify the most relevant metrics. ## user study our prompt-based metrics for text difficulty are prompts based on the results of a one-day user study we ran with a group of university students in november 2023. ## experiments we conduct a series of classification experiments to evaluate the usefulness of our novel prompt-based metrics for measuring text difficulty. we use a subset of the scienceqa dataset, which contains question-answer pairs across several topics and education levels. specifically, we run multinomial logistic regressions based on static metrics, prompt-based metrics, and the combination of the two to evaluate the marginal benefits of our new prompt-based metrics. we also compare these regression approaches to using an llm for zero-shot and few-shot classification. ## conclusion good teachers succeed in making the material understandable for their respective audiences. this adaptation is a complex process that goes well beyond replacing individual words and phrases. how-ever, existing static metrics for text difficulty, like the flesch-kincaid reading ease score, still focus on precisely those elements. as a result, these metrics are crude and brittle, failing to adapt to new domains and working mainly on long-form documents. our experiments reveal the promising potential of llms in predicting educational difficulty through using the prompt-based metrics rather than prompting the model directly. these metrics were derived from a small-scale user study involving students. empirically, we demonstrate that when combined with traditional static metrics, these prompt-based metrics enhance text difficulty classification. our study paves the way for novel applications of llms in educational contexts. by involving more educational stakeholders, such as teachers, we can gather more representative prompt-based metrics, facilitating future advancements in difficulty classification.