Similarity-Based Content Scoring - A more Classroom-Suitable Alternative to Instance-Based Scoring? (2023)

Abstract: Automatically scoring student answers is an important task that is usually solved using instance-based supervised learning. Recently, similarity-based scoring has been proposed as an alternative approach yielding similar perfor- mance. It has hypothetical advantages such as a lower need for annotated training data and better zero-shot performance, both of which are properties that would be highly beneficial when applying content scoring in a realistic classroom setting. In this paper we take a closer look at these alleged advantages by comparing different instance-based and similarity-based methods on multiple data sets in a number of learning curve experiments. We find that both the demand on data and cross-prompt performance is similar, thus not confirming the former two suggested advantages. The by default more straightforward possibility to give feedback based on a similarity-based approach may thus tip the scales in favor of it, although future work is needed to explore this advantage in practice.

## introduction approaches in automatic content scoring can be classified into two paradigms: instance-based scoring and similarity-based scoring @xcite . figure 1 gives a schematic overview of the two, with most work in the area of content scoring falling into the instance-based paradigm, where an algorithm is trained on learner answers as the only information source and learns about properties of correct and incorrect answers directly from these answers. in similarity-based scoring, in contrast, learner answers are compared to one or more target answers and correctness judgments are based on either the similarity to a correct answer (such as a sample solution) or on the label of the closest answer(s) to a given learner answer. in comparison to the instance-based paradigm, similarity-based scoring is substantially less well researched (see e.g. @xcite ). recent work by @xcite shows that similarity-based content scoring methods can yield comparable results to instance-based scoring if a similarity metric is substantially fine-tuned. however, it also showed that more research is needed to understand when it can be successful and how it compares to instance-based scoring. to do this, we first identify three possible advantages of similaritybased scoring: reduced data hunger, better crossprompt performance and explainability. these aspects would be highly beneficial when it comes to the application of automatic scoring in a realistic classroom setting: a typical classroom (ideally) does not consist of hundreds of students, meaning that collecting large amounts of answers to a question from students is unrealistic. since stateof-the-art content scoring builds on prompt-specific models, it would be highly desirable for a model to either be able to work well on this smaller amount directly or at least by making use of larger already existing cross-prompt data in training a promptspecific model. finally, feedback has been identified as one of the major influence factors for learning success @xcite , but oneon-one student-teacher time is limited, so a model that can justify why it awarded a certain number of points would be preferred over a performance-wise comparable one that simply returns a score. we perform a comparison of the two paradigms on different data sets typically used in one but not the other, focusing on a setup with limited data and also assessing to what extent using cross-prompt data can help overcome these limitations. we find that while overall highly-dependent on the choice of cross-prompt data, instance-based scoring benefits more. for a more encompassing comparison of the two paradigms, we also compute learning curves extending over a wider range of training data sizes and while we find that there is no one best method for smaller amounts of data, there is a instance-based scoring i would paint the dog house white, so it doesn't heat up as much. learner answer [2.87, 3.51, ..., 2.53] featurized representation 2 points prediction similarity-based scoring i would paint the dog house white, so it doesn't heat up as much. learner answer i would choose white to make sure it stays cool inside. reference answer(s) [0.23, 1.75, ..., 1.07] featurized representation [0.44, 1.38, ..., 0.96] point where similarity-based deep learning starts to consistently outperform all other methods, closely followed by instance-based deep learning. in comparing how much predictions vary based on the choice of training data, we find an overall smaller standard deviation for similarity-based predictions. we make all our code publicly available. foot_0 ## instance-based vs. similarity-based scoring instance-based scoring has become the de facto state of the art in automated scoring. recent experiments however showed that, with the emergence of deep learning, similarity-based models can keep up with instance-based ones: for essay scoring, @xcite use a bert model in a pairwise contrastive regression setup to score an essay in comparison to a reference, thereby outperforming the instance-based state of the art. for content scoring, @xcite reach comparable performance to an instance-based bert model by using fine-tuned sbert embeddings in a knn-like search for the most similar answer(s). @xcite introduce sentence transformer finetuning (setfit), which successfully uses sbert in a few-shot setting by using the finetuned embeddings to train a classification head. in line with this low-resource setup, similaritybased scoring is often applied to data sets containing only few answers per prompt. this includes work on computer science questions @xcite , english and german reading comprehension data @xcite and several approaches on the student response analysis data set @xcite , such as @xcite or more recently @xcite . even though in contrast, research on data with hundreds of answers per prompt or more is often associated with instance-based methods, such as most work on the asap data set (e.g., @xcite ; @xcite ; @xcite ), this does not necessarily mean that the data hunger is smaller for similarity-based models than for instance-based models as the former are often used to train a classifier across prompts. still, also considering the recent success of set-fit in a few-shot setting , we address the perceived dichotomy in data sets by contrasting the performance of both paradigms on both kinds of data sets. this gives insight into the difference regarding their data hunger. to investigate the supposed advantage of similarity-based scoring on limited data, we focus on learning curve experiments on smaller amounts of training data. previous work comparing instance-based to similarity-based scoring however showed similarity-based performance to be close to the respective best-performing instance-based model on both small (logistic regression) or larger amounts of training data (bert). @xcite , whereas logistic regression and bert have their strengths towards the lower and higher end of the training size spectrum, respectively. to further investigate this, we extend our learning curves beyond the low-resource spectrum and include a wider range of training sizes. another aspect @xcite already touched on the influence of the reference answer choice on scoring performance, thus asking how training inference sbert answer 1 2 points answer 2 2 points answer 1 1 point answer 2 1 point . . . . . . similarity label: 1 similarity label: 0 answer 1 2 points answer 2 2 points answer 1 1 point answer 2 0 points [0, 1] test answer 0 points max similarity: 0.45 max similarity: 0.87 max similarity: 0.23 answer answer . . . answer answer answer . . . answer answer answer . . . answer 0 points 1 point 2 points (un)lucky one can be when selecting these and whether it is worth investing time to carefully pick them. to investigate this, we compare the standard deviation of different training data samples for instance-based and similarity-based scoring. as mentioned above, the dichotomy of similarity-based and instance-based data sets is accompanied by instance-based scoring typically training one model per prompt, while similaritybased approaches often make use of data across different prompts, suggesting a possible superiority of similarity-based methods regarding crossprompt transfer. further supporting this notion is the fact that a similiarity-based model won the cross-prompt track of the 2021 naep automatic scoring challenge 2 , although the overall performance level of submissions lagged behind stateof-the-art instance-based models in within-prompt settings. it is however unclear how well a stateof-the-art instance-based model would fare on the same cross-prompt data, as such comparisons are lacking. @xcite use different ways of encoding answers to train a cross-prompt model in an instance-based fashion. they find sbert embeddings to be superior over word2vec embeddings or a bag of words approach, leaving open the question of whether using the sbert embeddings in a similarity-based fashion would have yielded even better performance. since the similarity-based zero-shot cross-prompt experiments by @xcite showed mixed results, we undertake a comparison of the non-zero-shot cross-prompt perfor-2 https://github.com/naep-as-challenge/ reading-prediction mance of instance-based and similarity-based methods. a third possible advantage of similarity-based scoring that requires user studies to investigate and is thus beyond the scope of this paper is that one can show which reference answer(s) led to a certain classification decision, by default lending it a certain degree of explainability that could serve as pedagogical feedback to students. this feedback is mainly aimed at students or teachers as opposed to ai experts, since we do not directly disclose the inner workings of the algorithm, but rather provide some rationale about why a score has been assigned. a similar direction is addressed by clustering approaches for automatic scoring (such as @xcite ; @xcite ; @xcite ) with clustering essential also being a similarity-based method bearing the advantage of structured output that can be used to provide human feedback to learners efficiently. to summarize, we identified three potential benefits of similarity-based models: a reduced training data hunger, the ability to abstract across prompts and the possibility of giving feedback based on reference answers, the latter of which we leave for future work. these sentence pairs are labeled with a similarity score of 1 (0), if both answers in the pair have the same (a different) label. in this manner, we create as many pairs as possible. figure 2 gives an overview of this fine-tuning setup, and also shows how the fine-tuned model is then used to obtain predictions on the test data: with the training data serving as a set of reference answers, each answer from the test set is compared against every answer from the training set, and the label of the most similar training answer is then used as prediction. we train for 5 epochs with batch size 8 and without warmup, using an onlinecontrastiveloss and an embeddingsimilarityevaluator, otherwise keeping all values at their defaults. validation is done after each epoch and we use the model with minimal validation loss for evaluation on the test data. similarity-based baselines since similaritybased scoring also works without any finetuning, we include similarity-based baselines that essentially perform only the inference step described in the above sbert setup. an answer from the test set is thus compared to all answers from the reference (i.e. training) set, predicting the scoring label of the most similar reference answer. while we also ran experiments using overlap and cosine similarity of word count vectors foot_2 , we for the sake of brevity only report results for edit distance, as an example of surface similarity, and the pretrained sbert model without any adaptation to the respective prompt, as an example of working on vectorized representations. instance-based approaches experimenting with a number of shallow algorithms foot_3 showed logistic regression (lr) to perform best, which is why we only report results for this method. we used the scikit-learn implementation in standard configuration (apart from setting max_iter to 1000) with token uni-to trigram features. as a representation of instance-based deep learning, we also fine-tune a bert model (bert_base_uncased) from huggingface foot_4 . we train this model for 20 epochs with a batch size of 8, running evaluation after each epoch and keeping the model with the lowest validation loss for evaluation on testing data. other than that, parameters are kept at their default values. we trained on nvidia quadro rtx 6000 and a100 gpus for a total of close to 4000 gpu hours. ## data hunger in comparing instance-based and similarity-based scoring methods, we focus on the amount of training data needed (i.e. how data hungry the approaches are). we focus on the low-resource setting, as (i) it is more realistic in a classroom setting, and (ii) the fact that similarity-based and instancebased perform on par has already been established when training data is abundant @xcite . results in figure 3 show that sbert has the upper hand on seb and asap, while it is outperformed by lr on beetle. other than on asap, baseline similarity-based methods are often surprisingly strong on both beetle and seb. we speculate that this might be due to shorter and simpler answers, which is also indicated by a higher overall performance. as expected, performance is overall higher on the 2-way-labeled data, but apart from this, relative results of the different methods are similar on the five-way-labeled data. note that results are averaged across all prompts of the respective data set and that individual performances per prompt again vary tremendously. one application that would benefit from models that are doing well on small amounts of data is the use of automated scoring in a realistic classroom setting, since the average number of students in a class does not allow collecting larger amounts of answers to any given question. if a teacher were however to make up exemplary answers for the different possible outcomes, they might produce a more balanced sample of reference answers than what we use in our random sampling of training data. in figure 4(a) , we therefore also show learning curves using balanced sampling of asap data, which means that samples will contain the same amount of answers for each label. 8 averaged for lr, bert and sbert over all training sizes, this yields a .09 increase in qwk compared to random sampling. the order of performance for individual methods does however vary substantially between the two settings and across different training sizes, with a tendency in most cases of sbert outperforming other methods and the baseline methods (pre-trained and edit) being inferior. a curious exception to this observation is the curve for bert on randomly drawn data. previous work on asap had found that both bert and sbert outperform lr on larger amounts of training data, while lr was superior on smaller data sizes @xcite . although our results do not find a general superiority of lr, we take a closer look at how the different methods compare for larger training sizes. we therefore extend the asap learning curve (with random sampling) to include up to 1000 training instances (figure 4(b) ). 9 we observe that soon after 100 training instances, there is a clear advantage of neural over shallow methods, with sbert outperforming lr much earlier. overall, sbert consistently outperforms or is at least on par with all other methods. ## cross-prompt scoring another claim often implicitly attached to similarity-based methods is that they might have greater capabilities of learning a cross-prompt model. this intuitively makes sense as instancebased approaches rely on the presence or absence of certain lexical material while similarity-based approaches can exploit the closeness to a model answer. @xcite did however find that in some cases fine-tuning an sbert model to one prompt before adapting it to another was actually detrimental to performance, with an off-the-shelf pre-trained sbert model sometimes even outperforming the fine-tuned one. since they did a zeroshot application to the new prompt, no data from the target prompt was used adapt the model to it. we therefore first fine-tune a model on 1000 answers from a base prompt, and then use a smaller amount (again building the learning curves from figure 3 ) to adapt this model to the target prompt. 11 figure 6 shows the change in performance for each combination of prompts in asap 12 compared to a prompt-specific setup without pre-training (i.e. the results from experimental study 1). to gain a better overview, results are not only averaged over all prompts but also all training sizes. like @xcite , we group prompts according to the underlying topics science, biology ela, as a transfer within the same topic group might be more successful than one across topic 11 we again only report results for the sota models bert and sbert for the sake of brevity. 12 as only this data set provides a large enough amount of answers, we only perform this experiment on asap. ## summary & future work we compared instance-based and similarity-based methods for content scoring, examining whether properties that are often implicitly attributed to the latter are in fact empirically observable. in a set of learning curve experiments directed at the claim of similarity-based methods being less data hungry, we find that a fine-tuned sbert model does often yield the best results, but not for beetle, where this method was outperformed by the instance-based logistic regression. the suggested superiority of similarity-based scoring when it comes to smaller training sizes could thus not be confirmed. when running experiments with larger training sizes on asap, sbert remains the bestperforming method up until using 750 training instances, from when on it is joined by bert. in a comparison of how much performance varies depending on the choice of training data, sbert had the upper hand, especially when a relatively large amount of balanced training data that is sampled. another proposed property of similarity-based scoring is the ability to transfer across prompts. this could however not be confirmed by our experiments, where the largest performance increases were observed for the instance-based bert model. examining performance of a hypothetical oracle condition showed that it might be worthwhile to learn a stacked classifier, thus combining the strengths of the different (both similarity-and instance-based) methods. other possible avenues of future work are topics that have been researched in the context of instance-based scoring but not, or at least not to the same extent, for similarity-based scoring. these include the importance of spelling errors or the vulnerability to adversarials.