Predicting Item Difficulty and Item Response Time with Scalar-mixed Transformer Encoder Models and Rational Network Regression Heads (2024)

Abstract: This paper describes a contribution to the BEA 2024 Shared Task on Automated Prediction of Item Difficulty and Response Time. The participants in this shared task are to develop models for predicting the difficulty and response time of multiple-choice items in the medical field. These items were taken from the United States Medical Licensing Examination® (USMLE®), a high-stakes medical exam. For this purpose, we evaluated multiple BERT-like pre-trained transformer encoder models, which we combined with Scalar Mixing and two custom 2-layer classification heads using learnable Rational Activations as an activation function, each for predicting one of the two variables of interest in a multi-task setup. Our best models placed first out of 43 for predicting item difficulty and fifth out of 34 for predicting Item Response Time.

## introduction according to @xcite , assessments are arguably among the core components of education. they help diagnose and monitor learners' skill levels and, thus, function as a basis for downstream educational decisions. depending on their concrete function, they can be further categorized. placement assessments are needed to recommend courses for learners at an appropriate level. formative assessments are required to monitor learning progress. summative assessments are needed to measure learners' outcomes. each assessment comprises multiple items, i.e., individual tasks test-takers must complete. for standardized assessments, items must be evaluated to guarantee fair and comparable outcomes. in this context, multiple factors must be assessed as listed in the standards for educational and psychological testing @xcite . among these factors are item difficulty, i.e., a numerical variable describing the overall difficulty of solving a given item, and item response time, which encodes the overall time needed to solve an item measured in seconds. traditionally, item difficulty has been assessed using methods such as rasch analysis @xcite or item response theory @xcite . both of them rely on collection data from pre-evaluations with cohorts of test takers. as administering respective pre-evaluation steps is still a labour-intensive and costly process @xcite , there has been ongoing research on automating these procedures using machine learning methods with a higher potential for generalization. one of these instances is the first shared task on automated prediction of difficulty and response time for multiple choice questions @xcite . in this paper, we describe a submission to this shared task, which placed first for predicting item difficulty and fifth for predicting item response time. ## related work both the prediction of item difficulty and item response time for multiple choice questions utilizing natural language processing are comparably novel tasks. earlier research on predicting item difficulty tackled mostly other item formats such as c-tests, a form of fill-in-the-blank test aimed at testing language proficiency, @xcite or constructed response items @xcite . in the context of language learning, settles et al. (2020) developed a method to assess the difficulty of various types of items for language learning in terms of the cefr framework. early research on predicting item difficulty for multiple choice questions was conducted by @xcite , who fit various feature-based models using heterogeneous sets of features incorporating embeddings, as well as lexical, syntactic, semantic, cohesion-based, and psycholinguistic features to predict item difficulty for a large-scale dataset comprised of united states medical licensing ex-amination® (usmle®) items. the authors also use features derived from information retrieval systems. they reason that retrieving an answer for a given question through information retrieval might predict the difficulty of cognitively retrieving an answer. subsequent work by @xcite and @xcite used similar approaches to predict item survival and item response complexity. for predicting item response time, baldwin et al. (2021) used feature-based models using primarily the same features and algorithms which ha et al. (2019) applied for predicting item difficulty. they found that embeddings and linguistic features were robust in predicting item response time, with ir-based features being less predictive while still holding some degree of predictive power. yaneva et al. (2023) combined linguistic features with static embeddings produced by word2vec and contextual word embeddings produced by non-finetuned bert models to predict a range of item characteristics, including item response time. what becomes apparent when reviewing the past literature on the topic is that transformer-encoder language models such as bert @xcite have not been fine-tuned for the prediction of item difficulty and item response time as of now. this can be regarded as a clear research gap, given that transformer encoders could push the state of the art for a wide range of tasks in natural language processing and outperformed more traditional feature-based approaches for these @xcite . ## method to close this gap, we aim to evaluate the overall predictiveness of pre-trained transformer encoder language models for item difficulty and item response time in our submission for the first shared task on automated prediction of difficulty and response time for multiple choice questions @xcite . ## discussion the research at hand has multiple implications. first, we proved that using established pre-trained transformer-encoder language models for predicting the item difficulty and the item response time can be a viable choice overall. moreover, we could also show that our adjustments to the typical bert architecture proved fruitful. these adjustments let our models achieve very competitive performance in the shared task, with our best model even winning one of the two tracks (item difficulty). in theory, our approach can be easily integrated with the past feature-based models published by @xcite @xcite and @xcite . for this purpose, one needs to fine-tune a respective model. one can then use the output of all intermediate layers as embeddings. using an algorithm such as random forests or gradient boosting, selecting appropriate features from these internal representations should be possible. works as those by @xcite , @xcite , ro-taru (2021), @xcite or @xcite show that the integration of task-specific transformer-based contextualized embeddings with more traditional feature-based algorithms can yield fruitful outcomes. considering systems such as the ones published by @xcite , one could easily replace the generalized embeddings they use with task-specific ones. future work could thus involve testing whether such embeddings can add to a more traditional feature set to improve the overall predictive power of a given model. it is also visible that the prediction of the item difficulty remains a challenging task since even the best participating models barely outperformed a dummy baseline model. on average, the models underestimate the difficulty of input items. in the case of this shared task, this effect might result from the data set being comparably small and from a highly specialized domain, namely the biomedical one with its comparably complicated and specialized language. however, since all past work on predicting the difficulty and required response time of multiple choice questions using machine learning models was aimed at assessments from this domain, it is hard to make generalized judgements on the overall difficulty of this problem. what is required here is the publication of additional datasets from different domains and the evaluation of models using these. in this context, cross-domain evaluations especially would be of high use. predicting the item response time was a more fruitful endeavour, with models outperforming the dummy baseline by a larger margin. however, with an rmse rate of 23.92 for the best-performing model, one still needs to consider that the predicted item response time is far from accurate. the same issue for predicting the item difficulty holds true for the item response time: the dataset at hand is from a highly specialized domain, and data from other domains is not generally available. ## conclusion this paper explains our submissions for the bea 2024 shared task on predicting the item difficulty and the item response time, of which the best placed first for predicting the item difficulty and fifth for predicting the item response time. our architecture combines pre-trained transformer encoder models with multi-task learning and custom regression heads, expanding upon an architecture published by @xcite by combining them with scalar mixing and rational activations. the results suggest predicting item response time and especially item difficulty are comparably difficult tasks. however, the dataset used for this paper stems from the biomedical domain. this domain uses a very specialized language. for this reason, the tasks need to be evaluated with data from more domains to make a general claim. this could be the objective of future work.