Large Language Model-based Pipeline for Item Difficulty and Response Time Estimation for Educational Assessments (2024)

Abstract: This work presents a novel framework for the automated prediction of item difficulty and response time within educational assessments. Utilizing data from the BEA 2024 Shared Task, we integrate Named Entity Recognition, Semantic Role Labeling, and linguistic features to prompt a Large Language Model (LLM). Our best approach achieves an RMSE of 0.308 for item difficulty and 27.474 for response time prediction, improving on the provided baseline. The framework’s adaptability is demonstrated on audio recordings of 3rd-8th graders from the Atlanta, Georgia area responding to the Test of Narrative Language. These results highlight the framework’s potential to enhance test development efficiency.

## introduction standardized tests are essential tools for evaluating knowledge and ability for academic and professional purposes, and thus must be rigorously designed and meet stringent criteria. key aspects include diverse item difficulty for comprehensive skill evaluation and appropriate response time allocation -insufficient time compromises fairness, while excessive time leads to inefficiencies @xcite . traditionally, item difficulty and response time optimization have relied on pretesting, where new items are embedded in live exams. however, this labor-intensive process limits the number of new items and introduces security risks through potential overuse @xcite . in high-stakes examinations like the united states medical licensing examination (usmle) foot_0 , these challenges necessitate the exploration of alternative approaches for more secure and efficient test design. in response to these challenges, recent research explores automated prediction using the text of items themselves. this approach promises to streamline test development, enhance exam fairness, and mitigate security risks associated with item overexposure. the automated prediction of item difficulty and item response time shared task at the 19th bea workshop aims to address this gap @xcite . advancements in large language models (llms), trained on massive text corpora, hold significant potential for discerning language patterns indicative of item difficulty and response time. this paper outlines our methodology for automated prediction of these characteristics, leveraging named entity recognition, semantic role labeling, and llms. we further evaluate the framework's validity across modalities by applying it to a dataset of children's oral responses to the test of narrative language. our approach integrates these technologies to analyze the complexities of test item texts, aiming to accurately predict both difficulty level and response time. ## related works in recent years, the prediction of item difficulty and response time has garnered significant attention in the field of educational assessment research. prior work in this field employed techniques rooted in classical test theory and item response theory. more recently, the advent of sophisticated machine learning approaches has enabled novel methods for modeling these parameters @xcite . in @xcite an lstm-based method for chinese reading comprehension tests was proposed. it achieved high accuracy utilizing word embeddings and text correlation networks. similarly, @xcite employed word embeddings within a semantic space to analyze relationships between multiple-choice test components, finding correlations between semantic similarity and item difficulty. research on item difficulty prediction in medical exams has also advanced significantly with qiu et al. (2019) introducing the document enhanced attention based neural network (dan) framework using semantic relevance and similarity for difficulty assessment. ha et al. (2019) further demonstrated that embeddings and linguistic features extracted from test documents outperform simple text complexity measures in predicting construct-relevant difficulty in mcqs. baldwin et al. (2021) incorporated item response time prediction, emphasizing the importance of understanding how test-takers interact with items. in a similar vein, xue et al. (2020) found transfer learning beneficial for usmle item difficulty prediction, suggesting stems alone are optimal for difficulty, while the entire question benefits response time prediction. despite these advancements, the joint prediction of item difficulty and response time remains under-explored, motivating our proposed technique designed to address this gap. ## data we evaluate our framework primarily on the 2024 bea shared task dataset constructed from the usmle. as an auxiliary task, we also test its validity on the test of narrative language.