Using Machine Learning to Predict Item Difficulty and Response Time in Medical Tests (2024)

Abstract: Prior knowledge of item characteristics, such as difficulty and response time, without pretesting items can substantially save time and cost in high-standard test development. Using a variety of machine learning (ML) algorithms, the present study explored several (non-)linguistic features (such as Coh-Metrix indices) along with MPNet word embeddings to predict the difficulty and response time of a sample of medical test items. In both prediction tasks, the contribution of embeddings to models already containing other features was found to be extremely limited. Moreover, a comparison of feature importance scores across the two prediction tasks revealed that cohesion-based features were the strongest predictors of difficulty, while the prediction of response time was primarily dependent on length-related features.

## introduction item difficulty and response time are among the important requirements in high-standard test development. for instance, in large-scale assessment, there is often a need to develop equivalent versions of the same test to be administered to different groups of people @xcite . when deciding on the inclusion of items in each version, it is necessary to know the difficulty level of each item and an estimate of the time needed to answer that item. such information is traditionally gained only through pretesting @xcite . pretesting, however, is not a very efficient method, as it is expensive @xcite and raises security concerns @xcite . therefore, it would be highly advantageous to devise a method to ascertain item difficulty and response time without resorting to the pretesting of items. with this motivation, a shared task was organized as part of the building educational applications (bea) workshop at the 2024 annual conference of the north american chapter of the association for computational linguistics (naacl) 2024. the shared task invited people to develop ml models for the prediction of item difficulty and response time of a sample of 466 items from the united states medical licensing examination (usmle). the present study was conducted in relation to this shared task. for a review of the complete set of submissions to the shared task, please see @xcite . ## related work in the last decade, educational assessment has witnessed a surge of interest in predicting item difficulty. having reviewed 38 papers on item difficulty prediction, @xcite provided a summary of the most frequent prediction models used, most studied domains and item types, and features with highest prediction power. ml algorithms such as neural networks and support vector machines (svm) are commonly employed along with a variety of natural language processing (nlp) techniques used for feature extraction from text data. language assessment was found to be the most investigated domain, and multiple-choice items were most frequently studied. a greater contribution of @xcite lies in its review of the most influential features in item difficulty prediction. while most features can be categorized as either syntactic or semantic, a few studies have used psycholinguistic features (e.g., @xcite , taking into account the processing of linguistic elements in the brain. the age of acquisition (aoa), as one of such "cognitively-motivated" features, offers an index of lexical difficulty based on how early/late in life certain words are acquired @xcite . word concreteness is another psycholinguistic feature. concrete words are assumed to be processed faster in the brain and thus would expectedly be easier than abstract words @xcite . the use of psycholinguistic features is not a novel approach, however. aoa and word concreteness, among similar features such as word imaginability, have long been on the list of the indices calculated by coh-metrix @xcite . a more recent trend is the use of semantic similarity as a feature, which is discussed further in the following. most recently, @xcite compared the performance of several ml algorithms in predicting the item difficulty of reading comprehension tests using features extracted from item texts. their extracted features include word counts, word frequencies, readability indices, and lexical similarity. for lexical similarity, using euclidean distance and cosine similarity, they calculated the textual similarity between the question and the correct option as well as between the correct option and the distractors. it was assumed that a higher similarity in the former comparison can make the question easier, while a larger similarity in the latter is associated with higher difficulty @xcite . their results indicated that regularization-based models in general, and the elastic net (@xmath0.666) in particular, outperformed other models. although we have recently seen an increasing number of attempts to predict item characteristics such as difficulty, the wide range of test domains and other differing contextual factors make it rather difficult to make generalizations across contexts. therefore, more studies are still needed before more valid conclusions can be drawn regarding the predictability of item characteristics. the purpose of the present study was to contribute to the line of research on predicting item characteristics in medical tests (see, for example, @xcite by exploring how an assortment of linguistic and non-linguistics features can be utilized along with word embeddings to predict the item difficulty and response time of multiplechoice medical test items. ## results ## discussion a comparison of the rmse results across the three models in both prediction tasks indicates that the addition of embeddings (i.e., model 2) had a very small contribution to model performance. while this finding was against our initial expectation, it does bear credence when taking into account the large number of features already fed into model 1. the selected coh-metrix indices coupled with our extracted features (such as rare words sum and medical terms sum) captured most of the variance, leaving not much else to be explained by embeddings. a similar scenario has been present in some other studies. in (ha et al., 2019), for example, adding either word2vec or elmo embeddings to a list of various linguistic features improved rmse results by minimal margins. as for model 3 in both prediction tasks, the ensemble method was ineffective in reducing rmse values because there was no possible combination of algorithms that would result in a better model performance. in both tasks, the difference between the top-performing algorithm and the rest of the algorithms was wide; therefore, combining the top algorithm with any other one would only harm the performance. another reason could be that the algorithms are making similar predictions, meaning that there is a high correlation between their predictions. the ensemble method usually works best when models trained by different algorithms have different strengths and weaknesses, so combining models could lead to one model compensating for deficiencies in another. the feature importance scores exhibited dissimilar patterns in the two prediction tasks. features measuring the cohesion of the stem text played a major role in predicting difficulty: the vast majority of the top predictors of difficulty are cohesionbased coh-metrix features. on the other hand, non-coh-metrix features, especially length-related ones, constituted the main group of predictors of item response. length, measured as either the maximum number of words in a sentence (i.e., sentence length max) or the total number of words in the stem text (i.e., stem length), is the predominant predictor of item response. compared to difficulty, response time can be considered less complicated to explain, as it is highly dependent on simple length-related features. ## limitations two limitations need to be taken into account when interpreting the results of the study. firstly, the quality of extracted features was dependent on the quality of stem text preprocessing. while preprocessing text data (e.g., tokenization and lemmatization) is generally challenging, the text of medical items can pose additional challenges. the stem of many medical items typically contains a tabulation of data, e.g., laboratory results and blood pressure measures. when embedded within the text, such data can negatively impact the accuracy of feature calculations. for example, a list of items and numbers within a syntactically simple sentence can make it appear as a complex sentence in measures of sentence complexity. it can also interfere with coherence measures calculated through the coh-matrix. the second limitation concerns the results of feature importance. different algorithms may produce relatively different feature importance sets as they try to reach their highest prediction accuracy. therefore, the top three or five features in one algorithm can differ from those in another algorithm even with a very close rmse. to better understand the contribution of each feature to the prediction model, experimental studies are recommended, as the direct effect of individual variables can be more reliably examined through experimental control and manipulation @xcite . ## conclusion the present study explored a selection of diverse features to predict the difficulty and response time of a sample of multiple-choice medical test items using a variety of ml algorithms. in either of the prediction tasks, the addition of embeddings to the list of features did not make a considerable contribution to model performance, and the use of the ensemble method was not effective either. in feature importance scores, however, the two tasks showed dissimilar patterns. features measuring cohesion were especially effective in predicting difficulty, while length-related features were the main predictors of response time. while future studies can examine the role of many other features in predicting item characteristics of medical tests, we would like to draw attention to collecting data from item writers to be used as a potential feature. especially in the case of item difficulty, medical test writers can be asked to rate the difficulty of the items they develop. while students might perceive items differently from what test writers would assume, item writers' ratings could still correlate highly with actual difficulty values. this feature enjoys high practicality and low cost, as item writers can give difficulty ratings as they write their own items. a more advanced, but also more expensive approach is to have item writers rate each others' items as well.