Evaluating Sampling Strategies for Similarity-Based Short Answer Scoring: a Case Study in T hailand (2025)

Abstract: Automatic short answer scoring is a task whose aim is to help grade written works by learners of some subject matter. In niche subject domains with small examples, existing methods primarily utilized similarity-based scoring, relying on predefined reference answers to grade each student’s answer based on the similarity to the reference. However, these reference answers are often generated from a randomly selected set of graded student answer, which may fail to represent the full range of scoring variations. We propose a semi-automatic scoring framework that enhances the selective sampling strategy for defining the reference answers through a K-center-based and a K-means-based sampling method. Our results demonstrate that our framework outperforms previous similarity-based scoring methods on a dataset with Thai and English. Moreover, it achieves competitive performance compared to human reference performance and LLMs.

## introduction automatic short answer scoring is a task that focuses on the development of a system or model capable of grading students' responses to question prompts in educational settings, such as short answers or other text responses @xcite . this can help reduce the workload for teachers and teaching assistants, particularly when grading homework in large courses. machine learning models can be trained to predict the score of a given answer. researchers have used svm @xcite , lstm @xcite , and bert @xcite to create such models. however, these require preexisting training data for each questions, which limits the applicability of such methods. large language models (llms) have also been explored to score students answers (lee and song, 2024). since llms have been trained on a wide range of domains, they can be potentially useful for evaluating student answers in zero-shot and fewshot settings @xcite . however, some university-level homework requires specialized technical knowledge, which may fall into domains for which no dedicated llm has been trained. fine-tuning an llm for specific courses presents further challenges, as universities offer many different subjects, making it a significant workload to prepare the necessary datasets for each course. additionally, llms are limited by high resource demands and the cost of api usage @xcite . another approach is similarity-based scoring @xcite , where students' answers are compared with a set of reference answers and given the score of the reference answer most similar to their own. bexte et al. (2023) explored this idea, sampling answers to be manually graded and use as reference with two methods: random sampling and balanced sampling. while the latter showed better performance, it is not applicable in a real grading scenario, since we cannot predetermine the score of each answer to create a balanced reference set for each class. while this could be simulated by having educators create their own reference answer for each score, it becomes quite challenging in higher educations, where more complex and diverse answers are expected. in this work, we present a semi-automatic, similarity-based scoring framework that eliminates the need for educators to create a separate reference answer set. instead, educators grade a subset of student answers selected through k-means-based sampling and k-center-based sampling without prior labeling, and the system uses these graded answers as the reference set. then, we evaluate our similaritybased scoring framework on real data collected from a university in thailand, which includes thai, english, and code-switched answers. our results ## method to score a set of student answers, our method consists of two main steps. first, a subset of answers is selected and graded manually to serve as reference answers. then, we assign scores to the rest of the answers by finding the most similar graded answer. an overview of our method is shown in figure 1 . in order to find the best representative subset of the answers, we can perform some kind of sampling in the text embedding space of the answers. we consider two sampling strategies that aim to maintain the diversity of the sampled subset: a k-means clustering-based strategy and a k-centerbased strategy. ## conclusion we propose a semi-automatic, similarity-based scoring framework that employs k-means clustering and k-center sampling to create a reference answer set and conduct a comparative study of our framework against llm inference and a human baseline. the results demonstrate that our framework outperforms similarity-based scoring methods that use random sampling to create a reference answer set and is comparable to both llm and human performance.