Leveraging Physical and Semantic Features of text item for Difficulty and Response Time Prediction of USMLE Questions (2024)

Abstract: This paper presents our system developed for the Shared Task on Automated Prediction of Item Difficulty and Item Response Time for USMLE questions, organized by the Association for Computational Linguistics (ACL) Special Interest Group for building Educational Applications (BEA SIGEDU). The Shared Task, held as a workshop at the North American Chapter of the Association for Computational Linguistics (NAACL) 2024 conference, aimed to advance the state-of-the-art in predicting item characteristics directly from item text, with implications for the fairness and validity of standardized exams. We compared various methods ranging from BERT for regression to Random forest, Gradient Boosting(GB), Linear Regression, Support Vector Regressor (SVR), k-nearest neighbours (KNN) Regressor, MultiLayer Perceptron(MLP) to custom-ANN using BioBERT and Word2Vec embeddings and provided inferences on which performed better. This paper also explains the importance of data augmentation to balance the data in order to get better results. We also proposed five hypotheses regarding factors impacting difficulty and response time for a question and also verified it thereby helping researchers to derive meaningful numerical attributes for accurate prediction. We achieved a RSME score of 0.315 for Difficulty prediction and 26.945 for Response Time.

## introduction the automated prediction of item difficulty and item response time is a critical task in the field of educational assessment, with implications for the fairness and validity of standardized exams. traditionally, item characteristics such as difficulty and response time have been obtained through laborintensive pretesting processes, posing challenges related to time, cost, and security. to address these challenges, there is a growing interest in leveraging natural language processing (nlp) techniques to predict item characteristics directly from the item text. @xcite in this paper, we present our system developed for the shared task on automated prediction of item difficulty and item response time, organized by the association for computational linguistics (acl) special interest group for building educational applications (bea sigedu). the shared task was held as a workshop at the north american chapter of the association for computational linguistics (naacl) 2024 conference. our participation in this shared task aimed to advance the state-of-the-art in predicting item characteristics and contribute to the ongoing efforts to improve the efficiency and fairness of standardized testing. in this paper, we provide an overview of our system architecture, including methodologies employed for predicting item difficulty and item response time. we describe the features utilized, the model architectures, and the training procedures. furthermore, we present the experimental setup, including the dataset used for training and evaluation, data augmentation, as well as the evaluation metrics employed to assess the performance of our system as prescribed by shared task. through our participation in the shared task, we aim to demonstrate the effectiveness of our approach in predicting item characteristics and contribute to the collective efforts in developing more accurate and efficient models for automated assessment in educational contexts. additionally, we discuss the implications of our findings and potential future directions for research in this area. we believe that our system holds promise for enhancing the fairness and effectiveness of standardized testing, ultimately benefiting both test developers and test takers alike. ## related work the paper "automated prediction of item difficulty in reading comprehension using long short-term memory" by li-huai lin, tao-hsing chang, fu-yuan hsu focuses on utilizing long short-term memory (lstm) to predict the difficulty of test items in reading comprehension. traditional methods of estimating item difficulty relied on expert validation or pretests, which were labor-intensive and costly. by automating the prediction process using lstm, the study aims to overcome these challenges. experimental results indicate that the proposed method shows a good prediction agreement rate. the use of lstm in predicting item difficulty offers a more efficient and accurate approach compared to manual methods, showcasing the potential of machine learning in educational assessment @xcite the paper question difficulty prediction for reading problems in standard tests by huang, z., liu, q., chen, e., zhao, h., gao, m., wei, s., su, y. and hu, g. initially involves converting the input into embeddings, followed by passing it through a bidirectional long short-term memory (bi-lstm) network to capture semantic relationships. subsequently, an attention layer is employed to identify words within the document or option that hold significant relevance to a given question. this process aids in selecting pertinent information. finally, the prediction layer displays the predicted difficulty scores. @xcite jump-starting item parameters for adaptive language tests by the authors mccarthy, a.d., yancey, k.p., laflair, g.t., egbert, j., liao, m. and settles, b address the challenge of calibrating test item difficulties in high-stakes language assessments, either with limited pilot test data or without any prior information. they propose a multi-task generalized linear model utilizing bert features to jump-start the estimation of item difficulties. with only 500 test-takers and a small sample of item exposures from a large item bank, their approach rapidly improves the quality of difficulty estimates. this joint model facilitates the comparison of testtaker proficiency, item difficulty, and language proficiency frameworks such as the common european framework of reference (cefr). moreover, it allows for the generation of new item difficulty estimates without the need for piloting, reducing item exposure and enhancing test security. the authors validate their method using operational data from the duolingo english test, demonstrating strong correlations between the derived difficulty estimates and lexico-grammatical features associated with reading complexity. @xcite 3 task description in recent times, efforts have been made to enhance the prediction of item parameters for high-stakes medical exams such as the usmle, have been hampered by challenges in sharing exam data. to address this gap, a shared task is proposed focusing on predicting item parameters using practice item content and characteristics from the usmle examination. refer @xcite the objective of this shared task is to advance the state-of-the-art in item parameter prediction, specifically focusing on two tracks: ## dataset description task comprises 466 previously utilized and retired multiple choice questions (mcqs) from the united states medical licensing examination (usmle) steps 1, 2 ck, and 3. the usmle is a series of examinations handled and developed by the national board of medical examiners (nbme) and the federation of state medical boards (fsmb) to support medical licensure decisions in the united states. the dataset is structured with the following attributes: itemnum: consecutive number assigned to the item in the dataset. itemstem_text: textual description of the clinical case or scenario presented in the mcq stem. answer_a to answer_j: text for response options a to j. unused columns remain blank for items with fewer than j response options. answer_key: letter denoting the correct answer for the item. answer_text: text corresponding to the correct response for the item. itemtype: denotes whether the item contained an image (pix) or not (text). images are not part of the dataset. ## experimental results and discussion our baseline model, bert regressor, achieved rmse scores of 0.307 for predicting difficulty and 88.502 for predicting response time. these scores demonstrate the model's performance in predicting both the difficulty level of exam items and the time intensity required for examinees to respond to them. fig. 1a and fig. 1b shows that most of predicted values are in a specified range and hence we assumed that the imbalance in data as shown in fig. 2a . hence we balanced the data. we also extracted few numerical features as discussed in feature engineering section and experimented with them. instead of simply finetuning bert, we trained bio-bert embeddings with ann and results are as shown in moving on to our 2nd approach consisting of training ml models with word2vec embeddings, the results are as in hence we considered the issue is in ann. since ann being deep learning model, with such limited data it is unable to capture patterns essentially and hence we tried training ml models with bio-bert embeddings. they outperformed ann, hence we came to conclusion of using ml models for prediction. in order to understand importance of extracted numerical features, we used the same ml models to perform regression on only question embeddings and only numerical data and results for each are shown in hence we concatenated both and trained the ml models to get results as shown in ## error analysis the final test data for shared task had 201 data points and team scalarlab had made three submissions/prediction files obtained by best three models of which we trained viz. random forest regressor, gradient boosting and multilayer perceptron trained on concatenated bio-bert embeddings and extracted numerical features. the rmse scores are as reported in table6. fig. 8 , fig. 9 and fig. 10 , we clearly can see they outperformed ann and bert for regressor. ## conclusion and future work we have achieved 0.315 rmse for difficulty prediction and 26.945 rmse for response time prediction. we successfully compared and explained why deep learning model ann failed in making better predictions, we discussed the importance of data augmentation and how results improved, and also proposed five hypotheses that seem to impact difficulty, response time of mcqs. as future work, we would like to explore how deep learning models can learn better with limited data and which embeddings are better fir such tasks where limited data is available. we would also explore what are ther factors that impact difficulty and response time of questions (mcqs) and incorporate that info in models to be trained to achieve better rmse scores.