Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction (2024)

Abstract: In this paper, we present the details of ourcontribution to the BEA Shared Task on Automated Prediction of Item Difficulty and Response Time. Participants in this collaborativeeffort are tasked with developing models to predict the difficulty and response time of multiplechoice items within the medical domain. Theseitems are sourced from the United States Medical Licensing Examination® (USMLE®), asignificant medical assessment. In order toachieve this, we experimented with two featurization techniques, one using lingusitic features and the other using embeddings generated by BERT fine-tuned over MS-MARCOdataset. Further, we tried several different machine learning models such as Linear Regression, Decision Trees, KNN and Boosting models such as XGBoost and GBDT. We found thatout of all the models we experimented withRandom Forest Regressor trained on Linguisticfeatures gave the least root mean squared error.

## introduction to conduct fair standardized tests for evaluating the learning outcomes of students, it is necessary to design tests that cover variety of questions of all difficulty levels such as 'easy', 'moderate' and 'difficult' ones. allowed exam time is another component that impacts the difficulty of exam. allowing ample amount of time to solve the questions can considerably reduce the difficulty whereas providing very little time to solve the exam questions can on other hand, make the exam unreasonably difficult. thus, the difficulty level of questions and the time taken to solve the questions(response time) are two critical factors to determine the overall difficulty of exam. determining the difficulty of items as well as the response time for this task, is a challenge in itself. conventionally, item difficulty and the response time are gathered through pretesting, where new items are incorporated into live exams alongside scored items. however, this process is laborintensive and costly, often limiting the number of items that can be created. furthermore, the reliance on pretesting poses security risks, as items may be copied or leaked due to their repeated usage. to tackle these challenges, there's a growing interest in predicting item characteristics such as difficulty and response time directly from the item text. this approach, known as the "cold-start parameter estimation problem" @xcite aims to streamline the process and enhance fairness by reducing the reliance on pretesting. by utilizing predictive models, estimates of item difficulty and response time can be generated, enabling a more efficient parameter estimation process with a smaller sample of test-takers. in this paper, we examine several approaches which build on predictive machine learning models (for example, linear regression, decision trees) and deep learning models(such as bert). our best model for the task of item difficulty achieved rmse of 0.31 and the best model for the task of predicting response time achieved rmse of 31.68. we hope that the exploration of models in this paper is able to help future researchers in the evaluation of exams. ## related work one of the earliest applications of predicting item difficulty emerged in the realm of language testing. here, a framework was introduced to assess learners' language proficiency in english, german, or french @xcite . controlling the difficulty of tests has also been important for automated generation of mcq format tests @xcite . another application can be found in context of automated grading where question difficulty estimates guide test creation @xcite . thus, predicting item difficulty has been a subject of growing research and with passage of time has extended to high-stakes applications such as medical or clinical exam @xcite . in order to automate difficulty prediction, machine learning and nlp based approaches using word lengths, sentence lengths and tf-idf featurization were proposed @xcite . a further improvement to it can be seen in the form of introduction of linguistic features @xcite which drastically improved the performance of approaches based on using machine learning models. on similar lines, machine learning and deep learning approaches have been researched upon for predicting the response time @xcite . other techniques employed in this regard include using transfer learning @xcite and language models such as bert @xcite . ## experiments for assessing performance, root mean squared error (rmse) serves as the metric for predicting response times and item difficulty in regression tasks. its suitability stems from the intuitive nature of rmse, making it well-aligned with the nature of these regression tasks. ## conclusion in conclusion, our research demonstrates the efficacy of machine learning models and feature engineering in addressing key challenges of standardized testing. linear regression coupled with linguistic features gave the lowest rmse scores of 0.302 and 26.181 for the difficulty prediction and response time prediction respectively. these findings highlight the potential of predictive models to streamline assessment processes and improve fairness. by reducing reliance on labor-intensive pretesting, our approach offers a scalable alternative while ensuring the integrity of assessment materials. future research should explore additional techniques and validate findings across diverse educational contexts. overall, our work advances educational assessment by offering innovative solutions to test design challenges.