Predicting the Difficulty and Response Time of Multiple Choice Questions Using Transfer Learning (2020)

Abstract: This paper investigates whether transfer learning can improve the prediction of the difficulty and response time parameters for 18,000 multiple-choice questions from a high-stakes medical exam. The type the signal that best predicts difficulty and response time is also explored, both in terms of representation abstraction and item component used as input (e.g., whole item, answer options only, etc.). The results indicate that, for our sample, transfer learning can improve the prediction of item difficulty when response time is used as an auxiliary task but not the other way around. In addition, difficulty was best predicted using signal from the item stem (the description of the clinical case), while all parts of the item were important for predicting the response time.

## introduction the questions on standardized exams need to meet certain criteria for the exam to be considered fair and valid. for example, it is often desirable to collect measurement information across a range of examinee proficiencies but this requires that question difficulties span a similar range. another consideration is the time required to answer each question: allocating too little time makes the exam speeded whereas allocating too much time makes it inefficient. typically, difficulty and response time measures are needed before new questions can be used for scoring. currently, these measures are obtained by presenting new questions alongside scored items on real exams; however, this process is time consuming and costly. to address this challenge, there is an emerging interest in predicting item parameters based on item text (section 2). the goal of this application is to filter out items that should not be embedded in live exams-even as unscored items-because of their low probability of having the desired characteristics. in practice, there may be situations where data are available for one item parameter but not for another. for example, when a pen-and-paper test is being migrated to a computer-based test, response time measures to individual questions will not be among the historical pen-and-paper data whereas item difficulty measures will be. in this scenario, the only available response-time data would be those collected from the small sample of examinees who first piloted the computer-based test. yet, since item characteristics like response time and difficulty are often related (e.g., more difficult items may require longer to solve), it is conceivable that information stored while learning to predict one parameter then could be used to improve the prediction of another. in this paper, we explore whether approaches from the field of transfer learning may be useful for improving item parameter modeling. we hypothesize that transfer learning (tl) can improve the prediction of difficulty and response time parameters for a set of ≈18,000 multiplechoice questions (mcqs) from the united states medical licensing examination (usmle r ). we present two sets of experiments, where learning to predict one parameter is used as an auxiliary task for the prediction of the other and vice versa. in addition to our interest in parameter modeling, we investigate the type of signal that best predicts difficulty and response time, which is done both in terms of exploring potential differences in the level of representation abstraction required to predict the two variables and in terms of the part of the item that contains information most relevant to each parameter. this is accomplished by extracting two levels of item representations, embeddings and encodings, from various parts of the mcq (answer options only, question only, whole item). predictions are compared to i) the predictions for each parameter without the use of an auxiliary task, and ii) a zeror baseline. the results from the transfer learning experiments show the usefulness and limitations of this approach for modeling item parameters with a view to practical scenarios where we have more data for one parameter. the results for the source of the signal suggest item writing strategies that may be adopted to manipulate specific item parameters. ## related work the majority of work related to predicting question difficulty has been done in the field of language learning @xcite @xcite . some exceptions include estimating difficulty for automatically generated questions by measuring the semantic similarity between the a given question and its associated answer options @xcite @xcite and measuring the difficulty and discrimination parameters of questions used in e-learning exams @xcite . with regards to medical mcqs, previous work has shown modest but statistically significant improvements in predicting difficulty using a combination of linguistic features and embeddings @xcite as well as predicting the probability that an item meets the difficulty and discriminatory power criteria for use in live exams @xcite . the literature on response time prediction is rather limited and comes mainly from the field of educational testing. the range of predictors that have been explored includes item presentation position @xcite , item content category @xcite , the presence of a figure @xcite , and item difficulty and discrimination @xcite . the only text-related feature used in these studies was word count. a more recent study by @xcite 2020 ) modeled the response time of medical mcqs using a broad range of linguistic features and embeddings (similar to @xcite ) and showed that the predicted response times can be used to improve fairness by reducing the time intensity variance of exam forms. to the best of our knowledge, the use of transfer learning for predicting mcq parameters has not yet been investigated. the next sections present an initial exploration of this approach for a sample of medical mcqs. ## data the data consists of ≈ 18,000 mcqs from a highstakes medical licensing exam. an example of an mcq is presented in here, the difficulty of an item is defined by the proportion of its responses that are correct. in the educational testing community this metric is commonly referred to as p-value. for example, a pvalue of .67 means that the item was answered correctly by 67% of the examinees who saw that item. (since greater p-values are associated with greater proportions of examinees responding correctly, pvalue might be better described as a measure of item easiness than item difficulty.) response time is measured in seconds and represents the average amount of time it took all examinees who saw the item to answer it. the distribution of p-values and log response times for the data set is presented in figure 1 . the correlation between the two parameters for the set of items is .37. ## method three types of item text configurations were used as input: i) item stem, ii) item options, and iii) a combination of the stem and options (this combination was used both as a single vector and as two separate vectors). after preprocessing the raw text (tokenization, lemmatization and stopword removal), it was used to train an elmo @xcite ) model 2 . the model was trained with two a 55-year-old woman with small cell carcinoma of the lung is admitted to the hospital to undergo chemotherapy. six days after treatment is started, she develops a temperature of 38c (100.4f). physical examination shows no other abnormalities. laboratory studies show a leukocyte count of 100/mm3 (5% segmented neutrophils and 95% lymphocytes). which of the following is the most appropriate pharmacotherapy to increase this patient's leukocyte count? (a) darbepoetin (b) dexamethasone (c) filgrastim (d) interferon alfa (e) interleukin-2 (il-2) (f) leucovorin as shown in two modeling approaches were applied. the first approach (method 1) used the pre-trained elmo parameters as the initialization and trained on the mcq data with the aim of predicting the prediction part was implemented using the scikit-learn library. the nvidia tesla m60 gpu was used to accelerate the model training. elmo types number of output parameters parameter dimension updating small 13.6 million 128 tuning middle 20.8 million 256 tuning original 93.6 million 1024 freezing in other words, if the target variable in the prediction part was p-value, then the target variable in the training part was response time and vice-versa. since we are also interested in understanding the effects of different levels of abstraction on parameter prediction (as captured by the embeddings and encodings), we used linear regression (lr) to predict the item characteristics using the extracted features as input. the training set, the validation set and the testing set consisted of 12,000 samples, 3,000 samples, and 3,000 samples, respectively. ## results and discussion the results for the experiments are presented in not all parts of the item were equally important for predicting the two parameters. signal from the stem alone provided the best results for the p-value variable in method 1 (23.32) and when pvalue was used as an auxiliary task for predicting response time (0.31) in method 2 (i.e., adding information from the answer options did not improve the result). by contrast, signal from the full item outperformed other configurations when the response time was predicted using method 1 (0.29) and when response time was used as an auxiliary task for predicting the p-value (23.04). therefore, the stem contained signal that was most relevant to the p-value variable, while the response time was best predicted using information from the entire item. this suggests that deliberating between the different answer options and reading the stem all have effects on the response time. however, the difficulty of the clinical case presented in the stem seems to have a stronger relation to the p-value than the difficulty attributed to choosing between the answer options. using the stem and options content as two predictors (stem + options) had no significant effects but, on average, provided slightly more accurate results than the single predictor (full item). finally, no clear pattern emerged with regards to the predictive utility of using embeddings vs. encodings or the embedding dimensions and weight tuning produced by training the three elmo models (small, middle and original). these results represent a first step towards the exploration of transfer learning for item parameter prediction and may have implications for both parameter modeling and item writing. ## conclusion this study investigated the use of transfer learning for predicting difficulty and response times for clinical mcqs. both parameters were predicted with a small but statistically significant improvement over zeror. this prediction was further improved for p-value by using transfer learning. it was also shown that the item stem contained signal that was most relevant to the p-value variable, while the response time was best predicted using information from the entire item.