ITEC at BEA 2024 Shared Task: Predicting Difficulty and Response Time of Medical Exam Questions with Statistical, Machine Learning, and Language Models (2024)

Abstract: This paper presents the results of our participation in the BEA 2024 shared task on the automated prediction of item difficulty and item response time (APIDIRT), hosted by the NBME (National Board of Medical Examiners). During this task, practice multiple-choice questions from the United States Medical Licensing Examination® (USMLE®) were shared, and research teams were tasked with devising systems capable of predicting the difficulty and average response time for new exam questions.Our team, part of the interdisciplinary itec research group, participated in the task. We extracted linguistic features and clinical embeddings from question items and tested various modeling techniques, including statistical regression, machine learning, language models, and ensemble methods. Surprisingly, simplermodels such as Lasso and random forest regression, utilizing principal component features from linguistic and clinical embeddings, outperformed more complex models. In the competition, our random forest model ranked 4th out of 43 submissions for difficulty prediction, while the Lasso model secured the 2nd position out of 34 submissions for response time prediction. Further analysis suggests that had we submitted the Lasso model for difficulty prediction, we would have achieved an even higher ranking. We also observed that predicting response time is easier than predicting difficulty, with features such as item length, type, exam step, and analytical thinking influencing response time prediction more significantly.

## introduction in the medical domain, standardized tests act as crucial gatekeepers, allowing only the best healthcare professionals into the field. an example is * supported by a fellowship from the research foundation flanders (fwo) prior studies by nbme researchers have shown that predicting the difficulty and response time of medical exam questions is a challenging task @xcite @xcite . as a result, the nbme launched an international challenge where they provided researchers with a set of retired exam questions from the usmle®. research teams were tasked with developing a system or model that takes as input a multiple-choice question and produces as output two estimates: (a) how challenging it is for test-takers and (b) how long it would take them to respond (see figure 1 for an illustration). the comprehensive details and results of this shared task are outlined in the overview paper authored by @xcite . we participated in the competition with the itec 1 team, an interdisciplinary research group affiliated with ku leuven and imec. our collaborative efforts span various fields, including artificial intelligence, educational sciences, language technology, machine learning, psychometrics, and statistical modeling. our strategy involved a fusion of statistical models, machine learning models, and language models. we integrated traditional a 65-year-old woman comes to the physician for a follow-up examination after blood pressure measurements were 175/105 mm hg and 185/110 mm hg 1 and 3 weeks ago, respectively. she has well-controlled type 2 diabetes mellitus. her blood pressure now is 175/110 mm hg. physical examination shows no other abnormalities. antihypertensive therapy is started, but her blood pressure remains elevated at her next visit 3 weeks later. laboratory studies show increased plasma renin activity; the erythrocyte sedimentation rate and serum electrolytes are within the reference ranges. angiography shows a high-grade stenosis of the proximal right renal artery; the left renal artery appears normal. ## method the shared task unfolded into two phases. during the training phase, spanning from january 15 to february 9, 2024, we received 466 multiplechoice questions along with additional metadata, such as the item type and exam step. our goal in this phase was to develop models that could predict two key targets: item difficulty and response time. transitioning to the evaluation phase, which took place from february 10 to february 16, 2024, we received an additional set of 201 multiple-choice question items, accompanied by the same supplementary metadata, excluding the two targets. utilizing our top-performing models from the previous phase, our focus was on predicting the unknown targets of difficulty and response time. each target allowed up to three final predictions to be submitted, and the submissions were then ranked based on the root mean squared error (rmse). in this section, we provide more detailed information about our methodology. ## discussion as previous research by the shared task organizers has shown @xcite @xcite , predicting response time and difficulty of multiple-choice questions for medical licensing exams is a challenging task. in this study, our team tried to solve this challenge by adopting a multidisciplinary perspective, combining insights from statistical modeling, machine learning, and natural language processing. while previous studies have primarily concentrated on examining the influence of exam and item metadata, along with certain linguistic complexity features (e.g., @xcite , we explored the integration of several novel, unexplored features. while our results validate the importance of specific raw metadata features (such as the number of answer options), they also highlight the significance of features derived from liwc and taales, as well as embeddings from biomedical language models. notably, the liwc feature indicating the degree of "analytical thinking" for answers emerged as particularly noteworthy for predicting response time. regarding the models, it is noteworthy that the more sophisticated ones did not surpass the less intricate models. simple models proved more accurate in predicting the response time of multiplechoice questions. this resonates with occam's razor principle, which favors simpler models as long as their performance matches or exceeds that of more complex alternatives (e.g., @xcite . in our study, models utilizing lasso or rf with principal component features outperformed the fine-tuned language model with embeddings. this suggests that, for this specific task, traditional machine learning methods incorporating dimensionality reduction were more effective and robust compared to complex statistical models. ## conclusion our team's contribution to the shared task of predicting the difficulty and response time of medical exam questions demonstrates that simpler models like lasso (l 1 -regularized) or rf regression, which utilize principal component features derived from linguistic features and clinical embeddings, outperform more complex, fine-tuned nlp models. in the winner announcement, the rf model secured the 4th position out of 43 submissions for difficulty, while the lasso model attained the 2nd position out of 34 for response time. post-hoc analyses revealed that if we had submitted the predictions of the lasso model of difficulty to the leaderboard, we would have surpassed the second position in predicting difficulty as well. moreover, predicting the response time for medical multiple-choice questions has proven to be a more straightforward task compared to predicting the difficulty of such questions. response time primarily hinges on item length (i.e., word count and number of answers), item type, exam step, and the level of analytical thinking required for the answers, as illustrated by permutation feature importance analyses. conversely, predicting item difficulty poses greater challenges, with all models approaching an average baseline performance. nevertheless, post-hoc analyses suggest that more extensive experimentation with fine-tuned language models could potentially aid in discerning the difficulty of multiple-choice questions. while response time can be more accurately predicted from linguistic features like word count, predicting difficulty may require more intricate modeling of deep clinical text representations.