Automated Prediction of Examinee Proficiency from Short-Answer Questions (2020)

Abstract: This paper brings together approaches from the fields of NLP and psychometric measurement to address the problem of predicting examinee proficiency from responses to short-answer questions (SAQs). While previous approaches train on manually labeled data to predict the human-ratings assigned to SAQ responses, the approach presented here models examinee proficiency directly and does not require manually labeled data to train on. We use data from a large medical exam where experimental SAQ items are embedded alongside 106 scored multiple-choice questions (MCQs). First, the latent trait of examinee proficiency is measured using the scored MCQs and then a model is trained on the experimental SAQ responses as input, aiming to predict proficiency as its target variable. The predicted value is then used as a “score” for the SAQ response and evaluated in terms of its contribution to the precision of proficiency estimation.

## introduction the automated scoring of short answer questions (saqs) has been a longstanding research area in both psychometric measurement and nlp applications. typically, a saq consists of a description of a problem followed by an open question which requires some form of a free short response by the test-taker. an example of a saq testing medical knowledge is presented in despite the fact that the automated scoring of saqs is a vibrant research area in both psychometrics and educational nlp, there are almost no interdisciplinary efforts that capitalize on the strengths of both fields (section 2). in psychometrics, the majority of the effort is focused on assessing the reliability, validity and fairness of the scoring procedures for measuring examinee proficiency, with low exposure to state-of-the-art nlp techniques. by the same token, nlp studies utilize sophisticated language technology but do not evaluate the scoring systems in a way that answers the main question of "does this scoring method help with the more precise measurement of examinee proficiency?". this paper presents an interdisciplinary study on automated saq scoring positioned in the intersection between nlp and psychometric measurement. unlike previous nlp studies, the scoring method is evaluated on its ability to produce better assessment of examinee proficiency. as a step further, the proposed approach does not rely on manually rated responses for training, instead utilizing information available at the stage of the evaluation of new saqs for their inclusion in standardized exams, as explained below. a previously healthy 26-year-old man is brought to the emergency department because of a tingling sensation in his fingers and toes for 3 days and progressive weakness of his legs. he had an upper respiratory tract infection 2 weeks ago. he has not traveled recently. he was unable to get up from bed this morning and called the ambulance. temperature is 37.3c (99.1f), pulse is 110/min, respirations are 22/min, and blood pressure is 128/82 mm hg. pulse oximetry on room air shows an oxygen saturation of 99%. physical examination shows weakness of all four extremities in flexion and extension; this weakness is increased in the distal compared with the proximal muscle groups. deep tendon reflexes are absent throughout. sensation is mildly decreased over both feet. what is the most likely diagnosis? sample of correct answers: guillain-barré syndrome, acute immune-mediated polyneuropathy in this study, we propose the use of proficiency as a dependent variable in a machine-learning model which takes pretest saq responses as input and aims to predict proficiency as output. notably, the proficiency is measured independently using the live items, while the model predicts it using the responses to the pretest items. during the training phase, the model learns to map similar responses to one another and identify patterns associated with the responses of high-proficiency students versus those of lower proficiency students. the underlying hypothesis is that people with similar proficiency would provide similar responses to the saqs. the predicted proficiency for a given new response is then used as its "score" and is evaluated in terms of its contribution to estimating the overall proficiency of the examinee who gave the response to the experimental question. we use data from a large medical exam, where in each form two pretest saqs are presented alongside 106 scored mcqs. contributions the main contribution of this study is an approach for automated saq scoring in the context of standardized testing that does not rely on manually labeled training data. the evaluation of the results is done using psychometric approaches investigating whether or not the developed scoring procedure improves the measurement of proficiency for the examinee who gave the response. core psychometric concepts that relate to nlp for automated scoring are explained in detail. ## related work this section describes the use of saqs in testing, followed by studies on automated saq scoring. ## data an example saq is presented in item writing: the saqs used in this study were directly derived from existing mcqs by removing their answer options and slightly rewriting the lead in question (e.g., "which of the following is the most likely diagnosis?" is converted to "what is the most likely diagnosis?"). the item text for the initial mcqs was developed by experienced item-writers following strict guidelines for content, information structure and formatting. all mcqs were formerly pretested and met quality criteria for live use. item selection: when choosing mcqs to be converted to saqs, concern was paid to a breadth of topics (as opposed to choosing several items that covered the same diagnosis), difficulty, and discriminatory power. the latter two are common measures used in psychometrics, computed as follows: filtering on these criteria resulted in an initial pool of 150 mcqs. the next step was to conduct an editorial review and have a subject matter expert answer the items in the new format, as explained below. after following the elimination steps presented above, the final number of saq items included in the study was 100. data for those were collected for one full year (august 2018 through august 2019) and amounted to 50,894 individual responses from 25,447 test-takers. ## method this section presents the approach for automated prediction of examinee proficiency. first, various types of embeddings are generated and compared (section 4.1) in order to select the best-performing ones for the main experiments. after that, several baselines are developed using the correct answer from the mcq version of the items as a rubric (section 4.2). next, the proposed approach is presented in section 4.3. finally, the metrics used to compare different approaches can be found in section 4.4. ## results as can be seen from the item-level results ( ## discussion probably the most important finding from this study is that, for our sample, saq "scores" assigned through the proposed approach can contribute more to the precision of proficiency measurement than the mcq form of the same items (indicated by the higher correlation between the former to proficiency). in addition, the method outperformed baselines relying on matching the response to a rubric (the mcq key), which is the approach of choice in most existing approaches to saq scoring. for the baselines, the moderate success of matching the responses to a single key phrase is likely a consequence of the shortness of our responses and is somewhat similar to the observation made by @xcite that semi-supervised clustering approaches work for very short but not for lengthy responses. it is therefore possible that this approach may not generalize well over formats requiring longer responses and the baseline results for those could be lower. notably, the presented results are a product of a conservative scenario where only 50% of the data is used for training. overall, these preliminary results show that a model which learns the associations of different responses to proficiency has the potential to improve the precision of proficiency measurement, while also eliminating the need for labeled training data. validity: an interesting question related to the validity of this approach is the argument that the proficiency metric is developed based on mcqs, which may measure different kinds of skills compared to saqs. in fact, it is this hypothesis that ignited the interest in using saqs in the first place. while acknowledging that such subtle differences may exist, it is not far-fetched to assume that both formats measure the generic construct of clinical skills knowledge. proficiency was measured using 106 mcqs with known psychometric characteristics and can therefore be regarded as a reliable metric of clinical skill knowledge. it is also possible that the output would correlate better if proficiency is measured using other saqs rather than mcqs, but this remains to be tested empirically. limitations and operational feasibility: while offering better estimations of proficiency, the approach has the drawback of having low interpretability which is an important prerequisite for high-stakes standardized exams. in addition, its relevance needs to be regularly assessed to ensure that there are no major differences between the responses of different-year cohorts. with these caveats in mind, the approach can be easily applied to other standardized exams. since large-scale standardized tests typically require that any new item is pretested, there will always be other items (in mcq, saq or other format), which can be used to measure the ground-truth proficiency. once trained on the pretested data, the model can be applied for scoring an unlimited number of new responses and the item can be used as a live one. the method could also be useful for predicting examinee proficiency in formative assessments such as those in massive open online courses (moocs), which do not offer manual scoring due to the high volume of responses (in fact, they rarely use the saq format for the same reason). in these cases, high-proficiency students could be defined through other criteria such as likelihood of course completion, quiz answers, peer ratings of forum responses, etc. while this scenario would require research on defining proficiency in the context of moocs, the proposed approach could be well suited to provide short-answer scoring when manual labeling is not feasible. it is important that the findings presented in this study are regarded as preliminary until a large-scale evaluation is conducted across different samples of questions and examinees, as well as compared to results from human scoring. the validity of the approach needs to be further tested with proficiency measured using other saqs. in spite of its limitations and the preliminary nature of the findings, this study presented a proof of concept showing that pretest saq scoring through modeling proficiency is a viable way to assign meaningful scores to new responses. these were shown to contribute to better measurement of overall proficiency for the pretest items without relying on a labeled training set. ## conclusion this paper presented experiments towards modeling examinee proficiency as an alternative to the human scoring of saqs. first, an irt proficiency metric was computed based on the test-takers' performance on 106 mcqs. after that, the responses to saqs were used as an input to a model aiming to predict examinee proficiency. the results were compared to those achieved through mcq scores and through several automated-scoring baselines in terms of the correlation between the "scores" given by each method and the overall examinee proficiency. the results indicated that the predicted proficiency for items in the test set used as a response score for these items has a higher correlation to the proficiency metric at both the item level and the examinee level compared to the mcq scores and the automated scoring baselines.