Item Difficulty and Response Time Prediction with Large Language Models: An Empirical Analysis of USMLE Items (2024)

Abstract: This paper summarizes our methodology and results for the BEA 2024 Shared Task. This competition focused on predicting item difficulty and response time for retired multiple-choice items from the United States Medical Licensing Examination® (USMLE®). We extracted linguistic features from the item stem and response options using multiple methods, including the BiomedBERT model, FastText embeddings, and Coh-Metrix. The extracted features were combined with additional features available in item metadata (e.g., item type) to predict item difficulty and average response time. The results showed that the BiomedBERT model was the most effective in predicting item difficulty, while the fine-tuned model based on FastText word embeddings was the best model for predicting response time.

## introduction in standardized exams, the examination of item characteristics is highly crucial for ensuring the fairness and validity of test results. for example, the difficulty of items pertains to the likelihood of an examinee answering the items correctly. incorporating a broad range of item difficulty levels in a standardized exam can help reduce measurement error and thereby improve the accuracy of the measurement process @xcite . in addition, while response time is often linked to item difficulty (i.e., more difficult items require more time to answer) @xcite , this variable itself can also offer new insights into examinees' test completion processes, such as their testing engagement and cognitive processes, thereby supporting the validity of test results. furthermore, understanding item characteristics can also be advantageous for modern test administration methods, including applications in automated item assembly, computerized adaptive testing, and personalized assessments @xcite . the difficulty of items and the average response time required to answer them are typically estimated based on empirical data collected during test pretesting. however, pretesting and obtaining robust results often require a large sample of examinees, which can incur substantial test administration costs. as a result, researchers have explored various methods to predict item characteristics without an actual test administration. for instance, researchers have sought estimates of item difficulty from domain experts and test development professionals. however, this approach has not consistently produced satisfactory or reliable estimations @xcite @xcite . another line of research seeks to predict item characteristics based on only item texts, such as the passages in source-based items, item stem, and response options @xcite . this approach employs text-mining techniques to extract surface features (e.g., the number of words in the texts) and complex features (e.g., semantic similarities of sentences) from item texts, to make predictions using advanced statistical models. building on the second line of research in predicting item characteristics based on item texts, the national board of medical examiners (nbme) initiated the bea 2024 shared task ( https:// sig-edu.org/sharedtask/2024 ) for automated prediction of item difficulty and item response time. the released dataset contained 667 previously used and now retired items from the united states medical licensing examination® (usmle®). the usmle is a series of high-stakes examinations (also known as steps; https://www.usmle.org/ step-exams ) to support medical licensure decisions in the united states. the items from usmle steps 1, 2 clinical knowledge (ck), and 3 focus on a wide range of topics relevant to the practice of medicine. in the bea 2024 shared task, research teams were invited to utilize natural language processing (nlp) methods for extracting linguistic features of the items and using them to predict the difficulty and response time of the items. our team employed state-of-the-art large language models (llms) to extract the features and build predictive models for item difficulty and response time. this paper documents the methods and results of our bestperforming models for predicting item difficulty and response time separately. the interest and effort in predicting item difficulty based on item texts dates back decades in the measurement literature. early work in item difficulty prediction primarily focused on identifying how item difficulty is influenced by a set of readily available, easily extracted, or manually coded item-level features. for example, @xcite predicted the difficulty of 210 reading comprehension items using various surface structure variables and word frequency measures for the text, such as the number of words, content words, or content-function words. @xcite predicted the difficulty of 213 reading comprehension items using 12 categories of sentential and discourse variables, such as vocabulary, length of texts, and syntactic structures (e.g., the number of negations). @xcite employed artificial neural networks to predict the item difficulty of 29 items in a reading comprehension test. they coded the items to extract three types of features: text structure (e.g., the number of words, lines, paragraphs, sentences, and content words), propositional analysis of passages and stems (e.g., the number of arguments, modifiers, and predicates), and cognitive process (e.g., identify, recognize, verify, infer, generalize, or problem-solving). research focused on the prediction of item characteristics such as difficulty and response time has been significantly influenced by the availability and application of emerging techniques in nlp and machine learning @xcite . for example, @xcite employed nlp methods to extract syntactic features to predict item difficulty, which were identified as crucial predictors. another application of nlp methods involves assessing the linguistic complexity or readability of item texts to predict item difficulty. @xcite , for instance, calculated readability indices for item texts and combined them with other fea-tures to predict item difficulty. however, readability indices did not perform well as predictors of item difficulty-a finding consistent with @xcite who noted that readability indices were among the least important predictors of item difficulty. nlp methods can also be used to extract term frequency-inverse document frequency (tf-idf) features. tf-idf measures the frequency of words or word sequences in a document and adjusts this count based on their frequency across a collection of documents. this approach emphasizes the importance of specific words to a particular document, with higher values indicating greater potential importance @xcite . in a relatively recent study predicting item difficulty for newly generated multiple-choice questions, @xcite extracted tf-idf features and achieved a root mean square error of 0.753. an important application of nlp techniques is the extraction of semantic features from item texts. word embedding is a technique that converts texts into numerical values in vector space, capturing the meanings of words across different dimensions @xcite . pre-trained nlp models such as word2vec and glove allow researchers to extract word embedding features from item texts (e.g., @xcite . for example, hsu et al. (2018) transformed item texts into semantic vectors and then used cosine similarity to measure the semantic similarity between different pairs of items. additionally, @xcite extracted word embedding features from multiple-choice items in high-stakes medical exams. along with other linguistic and psycholinguistic features in predicting item difficulty, they found that word embedding features contributed most to the predictive power. more recently, a significant breakthrough in the nlp field has been the development of llms such as bert @xcite and its variants, which were trained using different mechanisms or training datasets. for example, zhou and tao (2020) utilized a bert-variant model to predict the difficulty of programming problems. their results showed that compared with bert, distil-bert, a small version of the bert base model, was the best-performing model when the only available data for fine-turning was the text of the items. benedetto et al. (2021) also compared the performance of bert and distilbert in predicting the difficulty of multiple-choice questions and found that the bert-based models significantly outper-formed the two baseline models. unlike the prediction of item difficulty, the prediction of response time has not been widely investigated in the literature. this is mainly due to the limited availability of response time data. however, with the increasing use of digital assessments, such as computer-based and computerizedadaptive tests, in operational testing, the collection of response data has become easier, which motivated researchers to employ predictive models to predict the average response time required to solve the items (e.g., @xcite @xcite . ## discussion and conclusion the competition results for the bea 2024 shared task indicated that it is difficult to predict item characteristics such as difficulty using linguistic features @xcite . only 15 teams out of 43 managed to perform better than a baseline dummy regressor when it comes to predicting item difficulty using textual features extracted from the items. these results suggest that linguistic features may not be sufficient to capture the complex interplay between item features and item difficulty. unlike predicting item difficulty, predicting the average response time using linguistic features appears to be a more promising task. out of 34 submissions, 24 teams performed better than a baseline dummy regressor in predicting the average 1 our codes for predicting item difficulty and response time are available at https://osf.io/dwe4n/ . overall, the results for the bea 2024 shared task indicate that predicting item characteristics such as difficulty remains challenging and requires factors beyond linguistic or textual features.