UNED team at BEA 2024 Shared Task: Testing different Input Formats for predicting Item Difficulty and Response Time in Medical Exams (2024)

Abstract: This paper presents the description and primary outcomes of our team’s participation in the BEA 2024 shared task. Our primary exploration involved employing transformer-based systems, particularly BERT models, due to their suitability for Natural Language Processing tasks and efficiency with computational resources. We experimented with various input formats, including concatenating all text elements and incorporating only the clinical case. Surprisingly, our results revealed different impacts on predicting difficulty versus response time, with the former favoring clinical text only and the latter benefiting from including the correct answer. Despite moderate performance in difficulty prediction, our models excelled in response time prediction, ranking highest among all participants. This study lays the groundwork for future investigations into more complex approaches and configurations, aiming to advance the automatic prediction of exam difficulty and response time.

## introduction in this paper, we describe the proposals sent by our team to the bea 2024 shared task @xcite . this task aims to predict standardized exams' difficulty (track 1) and response time (track 2). the data used in this task is from a high-stakes medical exam called the united states medical licensing examination 1 . the exams are provided in a multiple-choice format, with answer candidates ranging from 4 to 10. adjusting the difficulty of exams to align with the intended level of evaluation is crucial for ensuring the validity and fairness of assessments. educators can accurately gauge students' understanding and proficiency within the targeted subject matter by calibrating the difficulty appropriately. this practice also promotes an equitable assessment environment where students can handle their 1 https://www.usmle.org/ challenges, allowing for a more reliable measure of their knowledge and skills. moreover, it encourages a more constructive learning experience, as students are motivated to engage with material that appropriately matches their abilities, fostering growth and development. ultimately, the careful adjustment of exam difficulty supports the effectiveness and integrity of the assessment process. several human examiners showed us these difficulties and asked for our help, opening the possibilities for an exciting application of language technologies to this problem. this is why our group is quite interested in this problem and participated in this task. actually, we are working on automatically predicting the difficulty of examinations for new language learners. the exams of our work are also in a multiple-choice format but, the number of options is lower (3 or 4, depending on the exam). our primary objective in this task was centered on the initiation of experiments utilizing transformer-based systems @xcite to explore their applicability to the given problem domain. instead of using the most modern generative models such as chatgpt foot_0 , llama-2 @xcite or mixtral @xcite , we explored the use of several bert-based models @xcite , which require less computational resources. we experiment with different input sequences and use the same data and approaches for both tracks. while our results in track 1 (item difficulty prediction) were relatively low (13th position for our best run), we obtained good results in track 2 (response time prediction), where we ranked in 1st, 3rd, and 4th position with our proposed systems. the paper is structured as follows: we describe the main features of our approach in section 2, while we detail the runs submitted to the task in section 3. then, we analyzed our results in section 4. finally, we give some conclusions and future work in section 5. ## systems description in this section, we describe the main features of our systems. in the development period, we tested different configurations using 10% of the training collection as test data. all our experiments are based on a bert-base model foot_1 fine-tuned for regression @xcite . we experimented with similar models like deberta @xcite and distilbert @xcite , obtaining the best results with bert. we focused on the base versions of these models instead of the large ones because we wanted to study the use of simple approaches that do not require big gpu units. we applied the same pre-processing to all our models and focused on testing the effect of using different inputs for the model. we provide more details in the next subsections. ## submitted runs we submitted the same three configurations to each track, where the only difference between tracks is the target label used for training the models. all the runs were trained using the whole training set provided by the organizers, with the hyperparameters selected in the development period. the only difference among the three runs was the input format. all the runs were trained using a v100 gpu in google colab, selecting the best model after ten training epochs. the details of the three runs are: or the correct answer. we use a batch size of 4, a learning rate of 1e-6. each run's batch size and learning rate were selected based on our experiments in the development period. we use the adam optimizer and the mean-squared error as the loss function for all our experiments, while the other hyperparameters were the default provided by the transformers 5 library. ## analysis of results the official measure for both tracks was the root mean squared error metric (rmse), which compares the prediction with the correct value. systems are ranked according to rmse, with the best systems obtaining the lowest error scores. we show and discuss the results of each track in the next subsections.