Findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions (2024)

Abstract: This paper reports findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions. The task was organized as part of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA’24), held in conjunction with NAACL 2024, and called upon the research community to contribute solutions to the problem of modeling difficulty and response time for clinical multiple-choice questions (MCQs). A set of 667 previously used and now retired MCQs from the United States Medical Licensing Examination (USMLE®) and their corresponding difficulties and mean response times were made available for experimentation. A total of 17 teams submitted solutions and 12 teams submitted system report papers describing their approaches. This paper summarizes the findings from the shared task and analyzes the main approaches proposed by the participants.

## introduction for standardized exams to be fair and defensible, test items must meet certain criteria. one important criterion for many exams is that the questions cover a wide range of difficulty levels to allow information about a wide range of examinee proficiencies to be collected effectively. additionally, it is often essential to allocate an appropriate amount of time for each question: too little time can make the exam speeded, while too much can make it inefficient. often, item difficulty and response time data are collected via a process called pretesting, wherein new items appear on live exams alongside scored items. while robust, the need for a statistically sufficient sample of examinees to complete these items restricts the number of items that can be pretested, potentially leading to overexposure and jeopardizing item security @xcite . the problem of estimating item characteristics with little to no response data is a decades-old research topic. early studies used what is sometimes referred to as auxiliary or collateral informationincluding various properties of an item's text-to improve parameter estimation within a bayesian framework @xcite @xcite . recent advances in nlp have led to a renewed interest in predicting item characteristics based on item text. as with the earlier research, it is hoped that such predictions may be used to "jump-start" parameter estimation @xcite allowing items to be exposed to fewer test-takers, or improve fairness by making the time intensiveness of test forms that include pretest items less variable @xcite . while there is evidence that nlp techniques may offer a potential solution (see section 2), the absence of publicly available datasets has resulted in fragmented efforts to advance the state-of-the-art in item parameter prediction, impeding meaningful comparisons between different approaches, exacerbating issues of reproducibility, and stifling collaboration. furthermore, as outlined in section 2, the existing literature has concentrated on difficulty prediction, neglecting other crucial item parameters such as response time, which also have important implications for exam fairness and validity. to address these shortcomings and advance this area of research, we organized the first shared task on automated prediction of difficulty and response time for multiple choice questions 1 . the shared task was organized as part of the 19 th workshop on innovative use of nlp for building educational applications (bea'24), collocated with naacl 2024, and took place between january 15 and march 10, 2024. an ideal shared dataset for this task would encompass test items along with their corresponding difficulties and response times based on responses collected from a sufficiently large and diverse examinee sample under standardized test conditions. to this end, 667 retired clinical multiple-choice questions (mcqs) from a high-stakes medical exam foot_1 were released for the exploration of two topics: predicting item difficulty (track 1) and predicting item response time (track 2). overall, 48 teams enrolled as participants, of which 17 submitted solutions and 12 submitted system review papers describing their approaches. this paper summarizes the organization and main findings from the competition. the data are available upon request at https://www. nbme.org/services/data-sharing . ## related work this section summarizes the main approaches used in item difficulty and response time prediction research, with special emphasis on clinical mcqs, the domain of the shared task. for a systematic review of the literature, we refer the reader to alkhuzaey et al. (2023). ## shared task description the data for the shared task comprises 667 previously used and now retired mcqs from steps 1, 2 ck, and 3 of the united states medical licensing examination (usmle ® ). usmle is a sequence of examinations (called steps), developed by the national board of medical examiners (nbme ® ) and federation of state medical boards (fsmb), that is used to support medical licensure decisions in the united states. each step includes 7 to 12 blocks of mcq items (a block ranges between 45 and 60 minutes), and each item is answered by approximately 300+ examinees. item characteristics used in this shared task were based on examinees who were medical students from accredited foot_3 us and canadian medical schools taking the exam for the first time. an example practice item from the dataset is given in each item is tagged with metadata indicating whether or not it contains an image, the step exam it was presented on, as well as difficulty and response time data, as shown in the structure below: the task was divided in two tracks as follows: out of the full sample, 466 items were made available as a labeled training set and the other 201 items were retained as an evaluation set. training data outside of the specified training set were allowed, provided these data were publicly available and their license allows use for research purposes. use of one target variable in the prediction of another was not permitted, since in most cases, predicting these variables will be most beneficial prior to the collection of response data-at which time neither the difficulty nor the time intensity parameters can be estimated. submissions were requested as separate .csv files for each track. each file had to contain the item number (item_num) and corresponding predicted value for each item. teams were allowed to submit up to three attempts per track, differentiated by adding run1, run2, or run3 to the name of their uploaded .csv file; however, such teams were required to explain how each attempt differed within their system report paper-i.e., changes in methodology, parameters, models used, prediction strategy, etc. in both tracks, the evaluation was based on rmse, and teams that achieved the lowest rmse value were considered winners. there were two separate leaderboards for track 1 and track 2. in both, submissions were ranked according to the rmse metric from python's scikit-learn library @xcite . ## results a total of 17 teams submitted up to 3 solutions for item difficulty prediction and 15 teams submitted up to 3 solutions for response time prediction. in track 1, predicting item difficulty, there are minor differences between the rmse of the top 15 solutions; however, even the best solution outperformed the baseline by only a small margin (#1, @xmath0.299, #16, @xmath1.311). these results are consistent with the prior literature ## main approaches the solutions submitted by the participants encompassed several approaches that had not been previously applied to the problem of modeling item characteristics. some of these were comparatively simpler models that performed unexpectedly well, such as the case of the submission that ranked #1 in predicting response time @xcite . in the case of modeling item difficulty, several approaches used classical methods such as linguistic features combined with embeddings but expanded the set of features to include novel predictors. these traditional solutions were not as successful for item difficulty prediction, which favored more novel approaches. these novel approaches can be broadly categorized as transformer model modifications, question answering using llms, and data augmentation techniques. these categories are not necessarily mutually exclusive (e.g., some approaches use both data augmentation and linguistic features); however, we found this broad classification scheme useful in describing the submitted solutions, as shown below. the main techniques used in the studies are further summarized in section 5.6. ## discussion the presented shared task is the first effort to benchmark the success of different methodologies on a common dataset of mcqs with known difficulties and response times. several innovative approaches, previously unexplored in this context, were formulated. the findings are consistent with prior work, which showed that, for clinical mcqs, the prediction of item difficulty is more challenging than the prediction of response time. ## conclusion the first shared task on automated prediction of difficulty and response time featured a set of 667 mcqs from a high-stakes clinical exam. seventeen teams submitted solutions and twelve teams submitted system report papers. for track 1, item difficulty prediction, the best-performing solution achieved an rmse of 0.299 compared to the dum-myregressor baseline of 0.311. for track 2, response time prediction, the best solution achieved an rmse of 23.927 compared to 31.68 for the baseline. the paper summarized the methodologies proposed by the participants and discussed the contributions and limitations of the competition. despite the progress made, the challenge of predicting item characteristics remains formidable. meeting this challenge necessitates not only the continued development of innovative methodologies but also the establishment of shared resources, such as public datasets containing reliable parameter estimates across various domains. such efforts will facilitate cross-domain evaluation, fostering a more comprehensive understanding of the underlying mechanisms driving item difficulty and response time prediction.