UPN - ICC at BEA 2024 Shared Task: Leveraging LLM s for Multiple-Choice Questions Difficulty Prediction (2024)

Abstract: We describe the second-best run for the shared task on predicting the difficulty of Multi-Choice Questions (MCQs) in the medical domain. Our approach leverages prompting Large Language Models (LLMs). Rather than straightforwardly querying difficulty, we simulate medical candidate’s responses to questions across various scenarios. For this, more than 10,000 prompts were required for the 467 training questions and the 200 test questions. From the answers to these prompts, we extracted a set of features which we combined with a Ridge Regression to which we only adjusted the regularization parameter using the training set. Our motivation stems from the belief that MCQ difficulty is influenced more by the respondent population than by item-specific content features. We conclude that the approach is promising and has the potential to improve other item-based systems on this task, which turned out to be extremely challenging and has ample room for future improvement.

## introduction the item difficulty is a core problem in the construction of exams. the exam items should encompass a broad spectrum of difficulty levels to efficiently ascertain the competencies of the test takers being assessed. traditionally, item difficulty has been a manual task done by human experts @xcite despite its inherent disadvantages compared to other approaches based on data @xcite . nevertheless, recent progress in natural language processing (nlp) has facilitated the automated prediction of item difficulty from textual content @xcite , serving as an alternative to traditional pretesting and manual task @xcite . these recent studies underscore the growing importance and interest in the topic of item difficulty prediction. in response, bea has launched the shared task "automated prediction of item difficulty and item response time" @xcite . this initiative represents an effort to push the boundaries of current research in item parameter prediction. the data provided for this task includes multiple-choice questions from steps 1, 2 ck, and 3 of the usmle, which is a sequence of examinations used to facilitate medical licensing in the united states. recent studies have leveraged nlp and machine learning techniques to address these challenges, providing insight into the factors that contribute to difficulty of multiple-choice questions (mcqs). four seminal studies are reviewed below that, together, show the approaches and advances that have been made in the automated prediction of usmle item difficulty. @xcite laid foundational work by developing a method to estimate the difficulty of usmle mcqs based on a diverse array of linguistic features and embedding types (elmo and word2vec), including measures quantifying the difficulty for an automated question-answering system. their approach surpassed various baselines significantly (zeror, word count, average sentence length, average word length in syllables, and the flesch reading ease formula). the study emphasized that information from all levels of linguistic processing contributes to item difficulty, with semantic ambiguity and psycholinguistic properties of words being particularly influential. in an study by @xcite , they provide an approach towards predicting item survival using linguistic features, two types of embeddings (word2vec and elmo), and information retrieval (ir) features in a high-stakes medical exam context. they implemented these features within a random forests algorithm framework and validated their approach using a dataset of 5,918 pretested mcqs from usmle. their findings indicated that the combination of all feature types outperformed the baselines, with elmo being the strongest individual predictor, followed by word2vec, linguistic features, and ir features. xue et al. (2020) explored the application of transfer learning to predict the item difficulty and response time for approximately 18,000 mcqs from usmle. they used three types of item text configurations as input: i) item stem, ii) item options, and iii) a combination of the stem and options. they were used to train three different elmo models. this research demonstrated that while transfer learning significantly enhances predictions for response time, when item difficulty is used as an auxiliary task, the converse is not true. difficulty prediction was most effective using signals from the item stem, while response time was best predicted using information from the entire item. building on ha et al. (2019) approach, yaneva et al. (2021) classified 18,961 mcqs from step 2 of the usmle into two categories in an unsupervised way: low-complexity items and high-complexity items, with the purpose of identifying interpretable relationships between item text and item complexity. they maintain that examining the linguistic features of the items can assist test developers in gaining a more detailed understanding of how cognitively more complex items differ from those with more straightforward solutions. similar to previous studies, they provide empirical evidence that linguistic features, both syntactic and semantic, play a crucial role in determining the complexity associated with the item response process. unlike previous studies, we investigated the hypothesis that item difficulty depends more on the features of the test-taking population than on the items themselves. to explore this, we simulated medical students' answers to various mcqs across different examinations by prompting a large language model (llm). this approach allowed us to understand how certain features influence item difficulty, providing insights that challenge previous methods of educational assessment. in this paper we describe our participating system in the bea 2024 shared task: automated prediction of item difficulty, which used a llm as core approach. ## discussion the results presented in regarding the strategy of decomposing the mcq item into yes/no questions, as presented in the results from similarly to the mutilation strategy, variations applied to the temperature parameter t resulted in efficient predictors of item difficulty ( since item difficulty is determined from item answers by a heterogeneous human population, the implementation of strategies to simulate this population is important in the effort to predict item difficulty. given that these two strategies produced the most effective predictors in our system, exploring combinations of these and other similar strategies emerges as a promising research perspective for addressing this challenging task. finally, figure 3 shows that the single regression system parameter, α, exhibits robust behavior over a wide range of its values, which likely contributed to the good performance of our system in the task. ## conclusion we conclude that the strategy of simulating test takers using llms offers a novel and promising perspective for the prediction of mcq difficulty. the strategy of random and incremental mutilation of the question stem appears to effectively simulate humans using different reading strategies of the questions, such as skimming or scanning. similarly, the manipulation of the "temperature" parameter in chatgpt llm appears to simulate human conditions that could be influenced by emotions or other factors experienced during the taking of an exam. these strategies allow for the simulation, using a single llm, of a heterogeneous population responding to an exam and obtaining differential results. this population of simulated humans produced the necessary input to obtain competitive item difficulty predictions without using features extracted from the item content. these results support the idea that item difficulty lies probably more in the population answering these questions than in the content or linguistic or cognitive factors extracted from the content of the items.