ACTA : Short-Answer Grading in High-Stakes Medical Exams (2023)

Abstract: This paper presents the ACTA system, which performs automated short-answer grading in the domain of high-stakes medical exams. The system builds upon previous work on neural similarity-based grading approaches by applying these to the medical domain and utilizing contrastive learning as a means to optimize the similarity metric. ACTA is evaluated against three strong baselines and is developed in alignment with operational needs, where low-confidence responses are flagged for human review. Learning curves are explored to understand the effects of training data on performance. The results demonstrate that ACTA leads to substantially lower number of responses being flagged for human review, while maintaining high classification accuracy.

## introduction automated short answer grading (asag) has been a longstanding educational application of nlp. the task of classifying the free-text responses to short-answer questions (saqs) as correct or incorrect is made challenging by the fact that the same concept may be expressed in a myriad of different ways. the problem has received considerable attention, with several competitions organized on the topic such as a semeval shared task by @xcite or the asap 2 kaggle competition 1 . most broadly, the asag literature defines two scoring approaches: an instance-based approach, where a system is trained on a portion of the data and outputs a predicted score for a given new response, and a similarity-based approach, where each new response assumes the label of an annotated response it is matched to using some similarity metric @xcite . in early work, pre-neural similarity-based approaches were shown to lag behind the less interpretable instance-based approaches @xcite . since then, 1 https://www.kaggle.com/c/asap-sas neural similarity-based approaches have shown increasing promise by learning response (or questionresponse) embeddings and matching the pairs using cosine similarity (e.g. @xcite ). @xcite proposed that the similaritybased approach can be further improved if the similarity metric is appropriately optimized. in their work, a pretrained sentence-bert model @xcite is fine-tuned on answer pairs and then a k-nearest neighbors classifier is used to match a new response based on its similarity to the labeled ones. these advances have led to a considerable improvement over the instance-based approach not only in terms of accuracy, but also in terms of interpretability and the need for less annotated data for training. in this study, we present the acta system (analysis of clinical text for assessment), where we build upon the work of @xcite by exploring the use of contrastive learning @xcite as a way to optimize the performance of similarity-based approaches and by applying the approach to the clinical domain. the contributions of this paper are as follows: ## data we perform experiments on two datasets containing short free-text responses to clinical test items. set 1 consists of sharp items (short answer rationale provision items) -an item format where examinees see a patient chart and are asked to provide a free-text response regarding the most likely diagnosis (e.g., "plantar fasciitis", "dermatomyositis" ), most appropriate next steps (e.g., "administer corticosteroids then do arterial biopsys"), causes (e.g., "homocysteine and mma levels in blood"), etc. foot_0 a total of 44 items were administered in a pilot involving 177 4th-year us medical students. each student saw each item, resulting in a total of 7,788 responses (of which 2,807 were unique). set 2 consists of short-answer questions, which present a vignette foot_1 describing a clinical case. similar to set 1, the set 2 responses included diagnoses, causes, and treatments, among other categories of responses. these items were administered to 8,162 us medical students as part of their internal medicine school subject exam. there were 71 set 2 items, where each item was seen by an average of 176 examinees (@xmath0.620), resulting in a total of 12,508 free-text responses (5,696 unique). responses from both sets were scored as correct or incorrect by content experts (physicians and nurse practitioners) using a scoring rubric for each item. for set 1, two subject matter experts scored the items together as part of developing scoring guidelines for future pilots (hence agreement statistics for independent scoring cannot be reported). another group of physicians reviewed the scores and confirmed agreement with the scoring procedure. for set 2, four judges scored the items. kappa coefficients (based on unique responses) for the six possible pairs of judges ranged from 0.89 to 0.92, indicating strong agreement. scoring resulted in 5,201 correct responses (66.78%) for set 1 and 8,086 (64.64%) for set 2. ## method we use contrastive representation learning @xcite to encode responses into embedding vectors such that responses with the same score have similar embeddings and responses with dif-ferent scores have very different ones. for any given two responses, the degree to which they are matched can then be measured by the cosine similiarty between their embedding vectors. similar to @xcite , we use sentence-bert (a.k.a. sbert) to derive the embeddings for each response, since the model introduces a modification of the pretrained bert network that "reduces the effort for finding the most similar pair from 65 hours with bert / roberta to about 5 seconds" @xcite . first, we pair up every response with every other response for the same item. each pair is assigned a label of 1 if both responses have the same score (both correct or both incorrect), 0 otherwise. for each pair, the two responses are passed to sbert independently, producing two sentence embedding vectors (one for each response). the contrastive loss encourages the model to minimize the embedding distance when responses have the same score, and maximize the distance otherwise. to do that, the cosine similarity and the cosine distance between the sentence embedding of the first response e 1 and the sentence embedding of the second response e 2 are defined as: during inference, the trained model is used to compute the cosine similarity between the sentence embedding of the new response and the sentence embedding of every annotation (i.e., responses of the same item in the training set). if the highest ## experimental setup baselines: we compare the approach proposed in acta to three baselines: a majority class baseline (always predicting a correct response); acta no finetuning -a similarity-based approach using sbert, where the model was not trained to optimize the similarity metric. we use all-minilm-l6-v2 foot_2 , which has been pretrained on 1b sentence pairs, as our backbone model for both sbert-notraining and sbert. finally, the incite system @xcite , which is specifically developed to score clinical text by capturing a variety of ways clinical concepts can be expressed. incite is a rule-based modular pipeline utilizing custombuilt lexicons, which contain observed misspellings for medical concepts and non-standard expressions, as well as common concepts and abbreviations from online resources. the tool performs direct and fuzzy matching between a new response and an annotated response (or a lexicon variant of it) using a fixed or dynamic levenshtein ratio threshold (in our case -.95). full details about the incite system are available in @xcite . ## results the majority class baseline was .79 for set 1 and .794 for set 2. the remaining results for a threshold of .95 are presented in when looking at the unmatched responses, we see dramatic differences between the three systems. when training on more than 40 examinees, incite and acta no finetuning have significantly more responses that require human review and increasing the amount of training data leads to small improvements. acta finetuned leaves fewer unmatched responses and continuously improves with the addition of more training data. these results show the when finetuned using contrastive loss, acta can ultimately save more human effort than incite and that the gains increase with data size. next, we experiment with different matching thresholds by replacing the .95 value with a range of values: .98, .90, .85, .80, .75, .70, and .65. f1 remains high even with lower thresholds: for set 1, the lowest f1 is .937 (@xmath3.65 when training on data from 20 examinees). for set 2 it is .95 for the same configuration (for detailed f1 results for each threshold, see figures 1 and 2 ). the number of unmatched responses, however, decreases significantly (see figures 3 and 4 ) -there are either 0 or 1 unmatched responses in both sets across all training configurations for threshold .65. this shows that with more liberal thresholds, the need for human scoring almost disappears (except the need for continuous quality verification). selecting the right trade-off between f1 and number of responses that need to undergo human review remains an operational decision. ## conclusion this study showed that a similarity-based clinical asag system finetuned using contrastive loss outperforms the incite and acta no finetuning baselines. lowering the similarity threshold value significantly decreases the number of unmatched responses, while -contrary to expectation -the f1 score remains high @xmath4.93 across conditions. the condition of weakest supervision -training on 20 examinees from set 1 with a similarity threshold of .65 -shows that 880 annotated responses are sufficient to score all 1.5k test set responses with @xmath5.93. similarly, when training on 20% of the data from set 2 with threshold of .65, all 2.5k test set responses are scored with @xmath6.95. the evaluation setup allows operational experts to balance the confidence threshold with a minimum necessary f1 score, where items with more errors can have more stringent similarity thresholds and vice-versa. the threshold may also vary depending on intended use: formative exams may tolerate a lower f1 to gain wider coverage, while summative assessments may have stricter criteria. in addition to its accuracy and wider coverage of responses, the interpretability of acta as a similarity-based system is an important advancement in clinical assessment compared to instancebased asag systems (e.g., @xcite ). interpretability holds special significance in the realm of automated scoring, as the value of the scores depends on the trust placed by various stakeholders (such as faculty, students, and residency selection programs, among others) in their fairness, reliability, and validity. like many other products, automated scoring tools are complex systems that have a significant impact not only because of their technical capabilities but also due to how they are used and the way their results are interpreted. misusing these tools or interpreting their outputs incorrectly can lead to serious ethical issues. in a summative context, the models described in this article are intended to be used as hybrid systems, where human raters always review borderline cases. in a formative context, it is crucial to carefully examine the relationship between the use of the system and its impact on learning outcomes, as this is essential evidence for validity. next steps include exploration of the effects of different "gaming" strategies (e.g., intentionally providing generic instead of specific answers) and potential differential functioning across demographic groups. notably, acta is intended as a hybrid system, where cases of examinees who perform near or below the passing standard are reviewed by human experts.