Scoring with Confidence? – Exploring High-confidence Scoring for Saving Manual Grading Effort (2024)

Abstract: A possible way to save manual grading effort in short answer scoring is to automatically score answers for which the classifier is highly confident. We explore the feasibility of this approach in a high-stakes exam setting, evaluating three different similarity-based scoring methods, where the similarity score is a direct proxy for model confidence. The decision on an appropriate level of confidence should ideally be made before scoring a new prompt. We thus probe to what extent confidence thresholds are consistent across different datasets and prompts. We find that high-confidence thresholds vary on a prompt-to-prompt basis, and that the overall potential of increased performance at a reasonable cost of additional manual effort is limited.

## introduction whenever a (semi-)automatic process is used to assist humans in scoring free-text answers, there is a trade-off between the human workload required and the resulting scoring accuracy. without any human input, the accuracy of the automated rating is usually quite low @xcite , however, already little human input might go a long way in improving the automation quality. @xcite score answers in a setting that uses reference answers and operationalize the confidence of the model as the similarity to the closest reference answer. this concept is visualized in figure 1 . they find that setting a threshold on model confidence, deferring to manual evaluation what falls short of it, leads to reasonable manual effort and high scoring accuracy. we test the applicability of this method in a highstakes classroom setting, where items are usually not re-used. this sharply limits the amount of manual scoring effort that can be spent before automation becomes uneconomical. we thus use a small volume of reference answers and examine to what extent a sensible pre-set confidence threshold can be established. as we cannot make the high-stakes student answers publicly available, we additionally replicate our results on four widely used datasets. our study makes the important step of linking state-of-the-art natural language processing for rating free-text items with the practical questions of start-up costs for building the models. ## related work the idea to automatically score only parts of all answers or to defer answers with a particularly low confidence of the algorithm to human scoring has been explored before @xcite . the approach that is closest to ours is @xcite , where answers to medical exam questions are scored using a similarity-based scoring method @xcite and the confidence of the classifier is operationalized through the similarity to the closest reference answer. this method could be taken further to iteratively improve a classifier through those human-labeled low-confidence answers, i.e. using active learning @xcite , as in the scoring domain done by horbach and palmer (2016) and @xcite . such a procedure does however have the disadvantage that human annotators have to annotate small batches of answers over a longer period of time. other studies rely on the idea that similar answers should receive the same score. such a grouping of answers could be reached through surfacelevel normalization (cf. zehner et al. (2016) ), which reduces orthographic variance, or unsupervised clustering methods operating on the surface level @xcite @xcite , on the semantic level using, e.g. lsa approaches @xcite , or a combination of the two @xcite . ## data we conduct experiments on five datasets (see we thus also run experiments on four existing, publicly available english datasets, that we use to put results on the exam data into context: the asap 1 dataset consists of answers to ten prompts from the domains of biology, science, and english language arts. powergrading @xcite has answers to ten united states citizenship exam questions that were collected from amazon mechanical turk. the student response analysis (sra) dataset @xcite is split into 1 https://www.kaggle.com/c/asap-sas/overview two subsets: beetle and scientsbank (seb). beetle has answers to 56 questions about electricity and electronics, while scientsbank contains answers to 150 foot_0 prompts that are from a mix of 15 different science domains. we use the two-way labeled version of the sra dataset, where answers are classified as correct or incorrect. ## experimental setup data split we split the answers to each prompt into reference and test answers. our reference answers aim to simulate a teacher manually providing exemplary answers for the different outcome labels. in practice, this would mean a rather small volume of unique examples per label. for each prompt, we thus randomly sample 5 answers per label as references, ensuring that there are no duplicates in this sample. whenever a similarity metric is fine-tuned on the reference answers, we split them into four answers per label to train and one answer per label to validate. classifiers we compare three methods of similarity-based classification that differ with respect to the employed similarity metric. all use a set of reference answers to label the test answers: based on the respective similarity metric, we predict the label of the most similar reference answer. we compare the following metrics: (i) edit distance foot_1 and two variants of cosine similarity based on (ii) pretrained or (iii) fine-tuned sbert embeddings @xcite ). 4 for the english datasets, we use the all-minilm-l6-v2 base model, and for the german data the paraphrasemultilingual-minilm-l12-v2 one, both taken from huggingface. dataset edit sbert target pretrained finetuned universityexams .86 .86 .91 .95 asap .46 .43 .50 .60 beetle .65 .65 .68 .80 seb .68 .65 .71 .80 powergrading .87 .92 .93 1.00 to fine-tune sbert, we follow the approach by @xcite : we train with pairs of answers that are labeled with a similarity label of 1 if both answers have the same score and 0 otherwise. to form these training examples, we pair each training answer with each other training answer. to validate, we pair each validation answer with each training answer. at inference, each test answer is compared to each training and each validation answer, i.e. all reference answers. we train for 30 epochs with a batch size of 8, using an onlinecontrastiveloss and an embeddingsimilarityevaluator. evaluation we evaluate using weighted f1, reporting averages across all prompts of a dataset. ## experiments first, we report results of a fully automatic baseline. in this approach, all test answers are scored automatically, i.e. assigned the label of the most similar reference answer. we then explore confidencebased scoring, only scoring instances where similarity exceeds a given threshold automatically. the remaining answers are referred to a human for manual scoring. the fully automatic baseline can be seen as an extreme case of this threshold-based scoring, where the confidence threshold is set so that all classifier decisions are accepted. we speak of a baseline, as introducing a confidence threshold should discard misclassifications and thus increase scoring performance. ## conclusion while previous work showed that confidence-based scoring can be successful @xcite , we do not find this to hold in our experiments. this may in part be due to the lower volume of reference answers and the higher overall scoring difficulty of some of the datasets we use. on some prompts, there may be thresholds that lead to a desirable tradeoff between manual effort and performance increase, but we did not find a general range of threshold values that would be promising to apply to unseen prompts.