Similarity-Based Content Scoring - How to Make S - BERT Keep Up With BERT (2022)

Abstract: The dominating paradigm for content scoring is to learn an instance-based model, i.e. to use lexical features derived from the learner answers themselves. An alternative approach that receives much less attention is however to learn a similarity-based model. We introduce an architecture that efficiently learns a similarity model and find that results on the standard ASAP dataset are on par with a BERT-based classification approach.

## introduction most work on automatic content scoring follows an instance-based approach, where the input is a single student answer and the output is its score @xcite . in contrast, similaritybased approaches compare a student answer with aor a set of -reference answers. the two approaches have rarely been compared directly, see @xcite as the rare exception, who found that instance-based methods outperform similaritybased ones. however, what many situations for which similarity-based methods are proposed have in common is that very little or no training data is available for an individual prompt. in the following discussion of previous work, we restrict ourselves to those similarity-based approaches. an early example of using reference answers and a similarity function is c-rater @xcite . other examples of pre-neural similarity-based approaches use wordnet-based and dependency graph alignment measures @xcite . similar approaches have been used for reading comprehension questions @xcite or scoring history exams @xcite . the semeval2013 student response analysis task @xcite links content scoring with recognizing textual entailment. due to the task setup (large number of individual questions with relatively few individual training data per prompt), some participants of the task used similarity-based methods for scoring @xcite , including methods for recognizing (partial) textual entailment @xcite . in recent years, neural similarity-based scoring models have been developed. @xcite use pretrained skip-thought vectors and learn a logistic classifier over the component-wise product and absolute difference vectors. @xcite report promising results on a notpublicly-available dataset by learning embeddings for question-answer-pairs and utilize cosine similarity as distance metric. while the work by @xcite seems to indicate that similarity-based approaches cannot compete with instance-based ones, such a comparison has so far to our knowledge not been made using powerful neural architectures. we thus propose a method, where a pretrained sentence-bert (s-bert) model is fine-tuned on answer pairs and then used in a knn-fashion to assign a score to a new learner answer based on the similarity to the already labeled ones. we present this approach in the next section. our code is publicly available here: https://github.com/ mariebexte/s-bert-similarity-based-content-scoring . ## similarity-based approach in our similarity-based approach, we learn and apply a similarity function between reference answers and learner answers (see figure 1 ). in the simplest case, we use the all-minilm-l6-v2 pre-trained sentence-bert model @xcite as is to encode answers (s-bert-orig). alternatively, we finetune the model using answer pairs from our dataset as input (s-bert-finetune). for doing this, we use a cosinesimilarityloss and a binaryclassificationevaluator. we consider answer pairs with the same human score as positive instances (i.e. highly similar) while we consider pairs with different scores as dissimilar. to reflect this (dis-)similarity, we assign positive instances a similarity score of 1 and negative instances a score of 0. we thus refrain from encoding the distance between the number of points with different levels of similarity (asap prompts are scored on a range from 0 to 2 or 0 to 3 points), i.e. both pairing an answer that received 0 points with one that received 1 or one that received 2 points gives the same similarity label of 0. this is beneficial when models trained on one prompt are used to score answers to another prompt that has a different number of outcomes. in the prediction step, we apply the s-bert model to encode each answer as a single dense vector. we compute the cosine similarity between an answer and each available reference answer. adopting a knn-inspired approach, we then take either the label of the closest reference answer (max) or the label of the group of answers with the same score that has the highest average similarity (avg). in both cases, the number of necessary comparisons is determined by the number of reference answers. therefore, scoring more answers will always just require comparing them to this fixed amount of reference answers, whose embeddings can be precomputed. note that in our experiments the same data was used to both fine-tune the similarity metric and for comparison in the prediction step. however, if runtime at test time is an issue, one could of course use fewer instances for the comparison than those used for fine-tuning. in our experiments, we observed that using a subset of just 60 of the over 1000 reference answers during inference lead to only a minor drop of qwk .01 in performance. do also note that, while we learn similarities between reference answers during training and use the same reference answers when later scoring answers, this does not reflect an inappropriate data leak between training and testing, as we are still scoring previously unseen answers. ## experimental setup we use the following setup to compare our approach against instance-based state-of-the-art systems. all results are averaged over five runs. instance-based baselines to establish a baseline for instance-based classification, we train one supervised classifier per prompt. we use a logistic regression (lr) classifier in standard configuration (@xmath0, @xmath1. as an instance for a neural classifier, we use a bert model based on the huggingface implementation. 1 we train for 6 epochs with batch size 16, crossentropyloss, and adam optimizer. dataset we use the asap-sas dataset from the kaggle short answer competition 2 containing 10 prompts with around 2,000 answers per prompt. the average answer length of the prompts ranges from 26.5 to 66.2 tokens per answer. broad prompt topics fall into three categories: sciences (prompts 1, 2 and 10), english language arts (ela; prompts 3, 4, 7, 8 and 9) and biology (prompts 5 and 6). we use this topic information later to check whether training on a different prompt from the same topic is beneficial. the dataset contains scoring rubrics but no specific set of reference answers for the individual scores. whenever we talk about reference answers, we mean answers drawn from the pool of training data. ## results figure 2 shows the comparison between the instance-based baseline and the similarity-based approach on both limited and full data. note that the different amounts of training data also mean that there are different amounts of reference answers. comparing s-bert-orig and s-bert-finetune reveals that finetuning is highly beneficial. max performs much better than avg with the pretrained model, perhaps due to just one similar enough answer sufficing for the max strategy to arrive at the correct classification outcome. for the finetuned models the performance difference between the two is much smaller, with avg even giving slightly better results than max. in the limited data setting, bert is not able to learn a sufficiently good model from the few training instances. lr performs best in this setting, beating our s-bert-finetune by qwk .06. in the full train setting, s-bert-finetune is on par with bert when using avg to determine predictions, with both models outperforming lr. variance between prompts cross-prompt evaluation one of the assumed benefits of similarity-based scoring approaches is that they generalize better between prompts and are thus often used for prompt-independent scoring @xcite @xcite . we hypothesize that using a model from the same topic (science, biology, ela) will work better than using a model from a different topic. we average across all prompts from the same topic, i.e. the cell train/science -test/science contains averaged results, where a model has been trained on one science prompt and tested on another science prompt. results show that only for biology prompts training on the same prompt is clearly beneficial as compared to training on other prompts. however, it still is much worse than finetuning directly on a single prompt. for example, the average qwk on the two biology prompts is over 0.70 for the fine-tuned results, while it is only half of that in the cross-prompt setting. for the other topic areas (science, ela) the cross-prompt results are even worse. another cross-prompt setting would be to use the pretrained s-bert-orig model as a zero-shot classifier (cf. the last line in ## conclusion in contrast to earlier work where instance-based methods outperformed similarity-based ones, the study in this paper finds that both paradigms are on par when a neural similarity model has been sufficiently fine-tuned. this seems to indicate that as soon as a similarity metric is complex enough, it incorporates the same capabilities as normally a classifier would. for the practitioner it might make little difference whether to use labeled instances to train an instance-based classifier or to fine-tune a similarity metric if both are applied in a promptspecific way. therefore, the next step in our line of research has to go into the direction of fully comparing the two paradigms, especially with respect to varying the amount of training data as well as exploring other datasets to allow for a better estimation which paradigm is preferable under which conditions. one step that we already took in this direction was to use the architecture described here for our participation in the naep-as challenge 3 , where our generic scoring model won a grand prize. in contrast to the successful application there, our cross-prompt experiments reported here showed results varying tremendously between prompts, hinting that sensible training data selection plays a crucial role. we will explore this further in future work. to foster more work in this area, we make our experimental code publicly available.