Using Linguistic Features to Predict the Response Process Complexity Associated with Answering Clinical MCQ s (2021)

Abstract: This study examines the relationship between the linguistic characteristics of a test item and the complexity of the response process required to answer it correctly. Using data from a large-scale medical licensing exam, clustering methods identified items that were similar with respect to their relative difficulty and relative response-time intensiveness to create low response process complexity and high response process complexity item classes. Interpretable models were used to investigate the linguistic features that best differentiated between these classes from a descriptive and predictive framework. Results suggest that nuanced features such as the number of ambiguous medical terms help explain response process complexity beyond superficial item characteristics such as word count. Yet, although linguistic features carry signal relevant to response process complexity, the classification of individual items remains challenging.

## introduction the success of high-stakes exams, such as those used in licensing, certification, and college admission, depends on the use of items (test questions) that meet stringent quality criteria. to provide useful information about examinee ability, good items must be neither too difficult, nor too easy for the intended test-takers. furthermore, the timing demands of items should be such that different exam forms seen by different test-takers should entail similar times to complete. nevertheless, while an extreme difficulty or mean response time can indicate that an item is not functioning correctly, within these extremes variability in difficulty and item response time is expected. for good items, it is hoped that this variability simply reflects the breadth and depth of the relevant exam content. the interaction between item difficulty (as measured by the proportion of examinees who respond correctly) and time intensiveness (as measured by the average time examinees spend answering) can help quantify the complexity of the response process associated with an item. this is valuable, since the more we know about the way examinees think about the problem presented in an item, the better we can evaluate exam validity. although easier items usually require less time than difficult items, the interaction between these two item properties is not strictly linear -examinees may spend very little time responding to certain difficult items and, likewise, examinees may spend a great deal of time on items that are relatively easy. the idea of response process complexity is best illustrated with items that have similar difficulty but different mean response times. in such cases, one item may require the formation of a complex cognitive model of the problem and thus take a long time, while another item with a similar level of difficulty may require factual knowledge that few examinees recall (or that many recall incorrectly) and thus take a short time on average. the interaction between item difficulty and time intensity can therefore provide valuable information about the complexity of the response process demanded by an item, which, we argue, can be further explained by examining the linguistic properties of the item. in this paper, we use a data-driven approach to capture the interaction between item difficulty and response time within a pool of 18,961 multiplechoice items from a high-stakes medical exam, where each item was answered by 335 examinees on average. for our data, this resulted in the definition of two clusters, one of which consisted of items that are relatively easy and less time-intensive, and another one which consisted of items that are relatively difficult and/or time-intensive. for the purposes of this study, we name these two clusters low-complexity class and high-complexity class, respectively. the use of the term response process a 16-year-old boy is brought to the emergency department because of a 2-day history of fever, nausea, vomiting, headache, chills, and fatigue. he has not had any sick contacts. he underwent splenectomy for traumatic injury at the age of 13 years. he has no other history of serious illness and takes no medications. he appears ill. his temperature is 39.2°c (102.5°f), pulse is 130/min, respirations are 14/min, and blood pressure is 110/60 mm hg. on pulmonary examination, scattered crackles are heard bilaterally. abdominal shows a well-healed midline scar and mild, diffuse tenderness to palpation. which of the following is the most appropriate next step in management? (a) antibiotic therapy (b) antiemetic therapy (c) ct scan of the chest (d) x-ray of the abdomen (e) reassurance studying the linguistic characteristics of these two categories may help test developers gain a more nuanced understanding of how cognitively complex items differ from those with a straightforward solution. provided that strong relationships are found, such insight can also be used to guide item writers or inform innovative automated item generation algorithms when seeking to create high-or low-complexity items. for this reason, our goal is not to train a black-box model to predict item complexity; instead, our goal is to isolate interpretable relationships between item text and item complexity that can inform our understanding of the response process and provide better itemwriting strategies. in addition to its utility for improving highstakes exams, the problem of modeling response process complexity is interesting from an nlp perspective because it requires the modeling of cognitive processes beyond reading comprehension. this is especially relevant for the data used here because, as we explain in section 3 below, the items in our bank assess expert-level clinical knowledge and are written to a common reading level using standardized language. contributions: i) we use unsupervised clustering to define classes of high and low responseprocess complexity from a large sample of items and test-takers in a high-stakes medical exam; ii) the study provides empirical evidence that linguistic characteristics carry signal relevant to an item's response process complexity; iii) the most predictive features are identified through several feature selection methods and their potential relationship to response process complexity is discussed; iv) the errors made by the model and their implications for predicting response process complexity are analysed. ## related work this section discusses related work on the topics of modeling item difficulty and response time. most nlp studies modeling the difficulty of test questions for humans have been conducted in the domain of reading comprehension, where the readability of reading passages is associated with the difficulty of their corresponding comprehension questions @xcite @xcite . for other exams, taxonomies representing knowledge dimensions and cognitive processes involved in the completion of a test task have been used to predict the difficulty of short-answer questions @xcite and identify skills required to answer school science questions @xcite . difficulty prediction has also been explored in the context of evaluating automatically generated questions @xcite @xcite through measures such as question-answer similarity. response time prediction has mainly been explored in the field of educational testing using predictors such as item presentation position @xcite , item content category @xcite , the presence of a figure @xcite , and item difficulty and discrimination @xcite . the only text-related feature explored in these studies was word count, and it was shown to have a very limited predictive power in most domains. several studies have explored the prediction of item difficulty and response time in the context of clinical multiple choice questions (mcqs). @xcite propose a large number of linguis-tic features and embeddings for modeling item difficulty. the results show that the full model outperforms several baselines with a statistically significant improvement, however, its practical significance for successfully predicting item difficulty remains limited, confirming the challenging nature of the problem. continuations of this study include the use of transfer learning to predict difficulty and response time @xcite , as well as using predicted difficulty for filtering out items that are too easy or too difficult for the intended examinee population @xcite . baldwin et al. (2020) ## data the data foot_0 used in this study comprises 18,961 step 2 clinical knowledge items from the united states medical licensing examination (usmle ® ), a large-scale high-stakes medical assessment. all items were mcqs. an example practice item foot_1 is given in ## features we use a set of interpretable linguistic features, many of which were previously used for predicting item difficulty @xcite and response time @xcite in the domain of clinical mcqs. these features were extracted using code made available by @xcite and to these, we add several predictors specifically related to the medical content of the items, as well as standard item metadata. ## results the linguistic and clinical content features improve predictive accuracy above the baselines, yielding a higher f1 score than the strongest baseline (.67 compared to .59). the reduced feature set does not lead to a meaningful performance drop compared to the full feature set, suggesting that no signal was lost due to feature elimination. figure 2 reports the eight best-performing features: umls phrases count, unique word count, polysemic word count, average noun phrase length, automated readability index, prepositional phrases, umls distinct terms count, and concreteness ratio. ## error analysis the output of the selected-features prediction model was analyzed further in order to get insight into this model's performance. as could be expected, the majority class of low-complexity items was predicted more accurately than the highcomplexity class, as shown by the confusion matrix in ## discussion the results presented in the previous section lead to three main findings: i) the linguistic characteristics of the items carry signal relevant to response 4 classes were balanced using the balanced subample setting of the class weight parameter in scikit-learn's randomforrestclassifier process complexity; ii) no individual features stand out as strong predictors, and iii) the most important features were those related to syntax and semantics. the first of these findings relates to the fact that the linguistic characteristics of the items carry signal that is predictive of response process complexity, revealing that the problems posed by lowcomplexity and high-complexity items are described using slightly different language. while this signal outperformed several baselines, the overall low predictive utility of the models suggests that there are other factors, yet to be captured, that have a significant effect on response process complexity. the retention of 56 features indicates that individual linguistic predictors provide a weak classification signal but, taken together, they complement each other in a way that ultimately provides a higher accuracy. the fact that there are many predictive features with none standing out is also a positive evaluation outcome for item writing quality, as it shows that the response process complexity associated with an item is not distributed along a small number of linguistic parameters. the most important features that helped with classification were those related to syntax and semantics (figure 2 ). the poor performance of the word count baseline suggests that differences in response process complexity cannot be explained solely by item length and that more complex linguistic features capture some of the nuance in the response process. as can be seen in figure 2 , high-complexity items contain a slightly higher number of umls phrases and (distinct) medical terms, as well as a higher number of unique words. these features suggest high-complexity items repeat words less frequently and may contain a higher concentration of new information and specialized terminology than low-complexity items. the individual phrases in high-complexity items are also slightly longer, which naturally influences readability metrics that are based on word and sentence length, such as the automated readability index (higher values are indicative of a more complex text). prepositional phrases were also identified as more important than other phrase types in distinguishing between response process complexity. prepositional phrases often serve as modifiers of the primary noun phrase and the higher number of prepositional phrases in the high-complexity items suggests the use of more specific descriptions (e.g., "small cell carcinoma of the ovary" instead of just "small cell carcinoma"). the words contained in the high-complexity items also have slightly higher concreteness levels, providing another indication that they may contain more terms, as terms tend to be more concrete than common words. finally, the words contained in the high-complexity items also tend to have more possible meanings, as indicated by the polysemous word count variable, which results in higher complexity owing to disambiguation efforts. overall, these features indicate that the language used in the low-complexity items is less ambiguous and descriptive, and potentially contains fewer medical terms. one limitation of the study is the fact that it treats item difficulty and time intensiveness as independent variables. this may not always be the case, as examinees do employ strategies to optimize their time. given finite time limits, examinees may ig-nore time intensive items if they believe the time needed for such items can be better utilized attempting other, less time intensive items. therefore, the relationship between difficulty and response time and their association with item text would differ for exams that do not impose strict time limits. when using data-driven approaches to defining item classes, our data did not lend itself to a categorization that would allow investigating high difficulty/low response time items and vice-versa. while the approach taken in this paper has a higher ecological validity, studying such cases in the future may lead to a greater understanding of various aspects of response process complexity and their relationship to item text. other future work includes exploration of potential item position effects. ## conclusion the experiments presented in this paper are, to the best of our knowledge, the first investigation of the relationship between item text and response process complexity. the results showed that such a relationship exists. to the extent that items were written as clearly and as concisely as possible, the findings suggest that high-complexity medical items generally include longer phrases, more medical terms, and more specific descriptions. while the models outperformed several baselines, they required a large number of features to do so and the predictive utility remained low. ultimately, this shows the challenging nature of modeling response process complexity using interpretable models and the lack of a straightforward way to manipulate this item property.