U nibuc LLM : Harnessing LLM s for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions (2024)

Abstract: This work explores a novel data augmentation method based on Large Language Models (LLMs) for predicting item difficulty and response time of retired USMLE Multiple-Choice Questions (MCQs) in the BEA 2024 Shared Task. Our approach is based on augmenting the dataset with answers from zero-shot LLMs (Falcon, Meditron, Mistral) and employing transformer-based models based on six alternative feature combinations. The results suggest that predicting the difficulty of questions is more challenging. Notably, our top performing methods consistently include the question text, and benefit from the variability of LLM answers, highlighting the potential of LLMs for improving automated assessment in medical licensing exams. We make our code available at: https://github.com/ana-rogoz/BEA-2024.

## introduction high-stakes medical licensing exams, like the united states medical licensing examination (usmle), require well-crafted questions to accurately assess a candidate's knowledge and skills. traditionally, determining item difficulty and response time (average time to answer) relied on pretesting, which can be carried out by embedding new items alongside scored items in live exams. however, this method has been recognized as impractical due to resource limitations @xcite . this year's workshop on innovative use of nlp for building educational applications (bea 2024) directly addresses this problem and its resource limitations by proposing a shared task on automated prediction of item difficulty and item response time for usmle exam items. this initiative fosters collaboration and innovation in developing reliable prediction methods, while also contributing to creating more efficient, secure, and informative medical licensing exams. this paper details our participation in the shared task @xcite , where we investigated the use of large language models (llms) to predict difficulty and response time for retired usmle multiple-choice questions (mcqs). our main contribution is to augment the dataset by incorporating answer choices generated by several zeroshot llms @xcite . to solve the two prediction tasks (question response time prediction, question difficulty prediction), we employ transformer-based models that alternatively employ six different feature combinations. our findings indicate that predicting question difficulty proves to be a more complex task. interestingly, the most successful models consistently incorporate the question text, and benefit from the augmentation based on llm-generated answers. our results highlight the potential of llms to enhance automated assessment methods in medical licensing exams. we also present post-competition methods that obtain better results than the originally submitted models. these newer models are aimed at addressing overfitting and our wrong choice of features. ## related work the need for alternatives to the traditional processes motivates the exploration of new methods for estimating item difficulty and response time. recent research @xcite @xcite @xcite suggests promising results using machine learning models trained on item text data to predict these characteristics. one of the seminal studies in this direction @xcite investigated the feasibility of using machine learning models to predict both item difficulty and response time for multiple-choice questions in a high-stakes medical exam. the authors focused on extracting various features from the question text data, including linguistic features and embedding types. their models were then trained to predict these difficulty and response time characteristics. the encouraging results from this study suggest that machine learning offers a promising alternative to traditional, resource-intensive pretesting methods for estimating these important exam design elements. while the prior studies focused on predicting difficulty and response time separately, xue et al. (2020) explored a method that could predict both simultaneously, using transfer learning. their research suggests that this approach offers potential benefits in terms of efficiency. in addition to predicting difficulty and response time, researchers explored another valuable application of machine learning: item survival prediction @xcite . this task focuses on estimating the likelihood of an item to be included in the final exam based on its difficulty and other question characteristics, and highlights the versatility of machine learning for various stages of exam design. another approach was presented by @xcite , who study the use of linguistic features to predict the response process complexity, which refers to the mental steps required to answer a medical mcq. instead of predicting difficulty, yaneva et al. (2021) leveraged the use of linguistic features to predict the response process complexity associated with answering medical mcqs. their work sheds light on the underlying factors that contribute to the difficulty of these questions. in summary, automated approaches offer several advantages, such as efficiency (predicting from text eliminates the need for pretesting, saving time and resources), security (reduced reliance on pretest-ing minimizes the risk of question exposure), and scalability (automated methods allow for creating larger pools of high-quality questions). therefore, continuously validating the use of machine learning to replace traditional methods is currently an active research topic. ## methods we start by annotating the original dataset with answers obtained by prompting llms in a zero-shot setup. the extended dataset is further processed by scaling the target labels, creating additional feature combinations, text cleaning, and setting the data split for cross-validation. the cleaned dataset is employed to fine-tune two transformer-based models. the end-to-end overview of the employed framework is presented in figure 1 . below, we describe each step of our pipeline in detail. ## conclusion in this paper, we presented our approaches to the bea 2024 shared task on automated prediction of item difficulty and item response time of retired usmle mcqs. our main contribution is a taskspecific data augmentation method based on adding answers to mcqs using llms prompted in a zeroshot setup. we carried out exhaustive experiments for both tasks, using two strong transformer-based models, in both fine-tuning and linear probing settings. we employed seven different types of feature combinations, while leveraging llm-based answers. the empirical results showed four key findings. first, the difficulty prediction task is significantly harder than the response time prediction task. second, we noticed that the top-performing approaches always made use of the question text. third, llm answers had a positive impact on performance, especially on the more difficult prediction task. fourth, linear probing (training an svr on frozen pre-trained features) shows a better generalization capacity than end-to-end fine-tuning, most likely due to the small training set available for the competition.