In-Context Meta-Learning vs. Semantic Score-Based Similarity: A Comparative Study in A rabic Short Answer Grading (2023)

Abstract: Delegating short answer grading to automated systems enhances efficiency, giving teachers more time for vital human-centered aspects of education. Studies in automatic short answer grading (ASAG) approach the problem from instance-based or reference-based perspectives. Recent studies have favored instance-based methods, but they demand substantial data for training, which is often scarce in classroom settings. This study compares both approaches using an Arabic ASAG dataset. We employ in-context meta-learning for instance-based and semantic score-based similarity for reference-based grading. Results show both methods outperform a baseline and occasionally even surpass human raters when grading unseen answers. Notably, the semantic score-based similarity approach excels in zero-shot settings, outperforming in-context meta-learning. Our work contributes insights to Arabic ASAG and introduces a prompt category classification model, leveraging GPT3.5 to augment Arabic data for improved performance.

## introduction automatic short answer grading (asag) has been a prominent subject of discussion in the field of ai in education, studied for more than half a century @xcite . this is not surprising given the potential asag systems hold for enhancing various aspects of educational systems. by automating routine grading tasks, teachers can focus more on their unique human role of being motivators of learning and nurturing students' curiosity, ultimately enriching the educational experience @xcite . the shift toward automation in grading not only enhances efficiency and eliminates human bias but also empowers educators to dedicate their time and expertise to the critical aspects of teaching that require human insight and empathy. over the last decade, progress in the field of asag has significantly accelerated, driven by ad-vancements in deep learning techniques and the availability of large datasets. asag systems can be broadly categorized into two main approaches: instance-based and reference-based @xcite . the majority of research in asag has primarily focused on the instance-based approach, which involves scoring individual student answers independently. on the other hand, reference-based approaches rely on measuring the similarity between the student's response and the reference answer and assigning a score based on this similarity. reference-based approaches not only have the potential to be more robust to variability but also have the advantage of being more interpretable and less data-hungry @xcite . however, only a few studies have been conducted comparing the 2 approaches and showing that the performance of reference-based approaches compared to instance-based approaches often yields worse or comparable results @xcite . while instance-based approaches have dominated the asag landscape, it's important to note that most of this research has been conducted in the context of the english language. english is one of the most widely studied languages, and therefore, a substantial amount of educational content and resources are available for it. however, the need for asag systems in other languages, such as arabic, is equally significant. even though modern standard arabic (msa) could also be considered a thriving language @xcite , datasets for asag in arabic are still scarce. in this study, we hope to contribute to the field of asag by presenting a comparison of two distinct approaches to asag in arabic. in our first instance-based approach, we leverage a pre-trained language model (i.e. bert) and train it on different questions with a shared type. for each instance, we create an input structure that provides contextual information for the model. in our reference-based approach, we train a score-based semantic similar-ity model using sentencetransformers @xcite . our results demonstrate that while both techniques perform similarly in conventional training circumstances, score-based semantic similarity has considerable potential for delivering superior results in zero-shot settings. we additionally propose a "prompt category" classification model to facilitate the selection of the most suitable scoring model for a given question. we show the effectiveness of this model in low-resource settings by augmenting the training data with synthetic examples generated by gpt-3.5. to the best of our knowledge, this is the first study to apply and compare the two distinct approaches to the problem of asag in arabic. finally, we make the code and models publicly foot_0 available to facilitate future research in this area. ## related work research in asag can be categorized into two main paradigms, instance-based or similarity-based methods @xcite . most recent asag research follows the instance-based paradigm, where algorithms are trained primarily using a large set of student answers to learn about the features of correct and incorrect responses. with the rise of large language models and transfer learning, most studies typically involve finetuning bert such as in the work by lun et al.. another example is the work of nael et al. where they fine-tune bert and electra models on a machine-translated asap dataset. condor et al. used sbert embeddings to train a model with an instance-based approach rather than using it in a similarity-based approach. fernandez et al. introduced a single shared scoring model for multiple questions using a specified input structure that provides contextual information for each item. similarly, to score mathematical questions, zhang et al. use an in-context learning approach that provides scoring examples as part of the input to a math-bert model to promote generalization. on the other hand, in the reference-based approach, student answers are evaluated by comparing them to one or more target answers. judgments of correctness are thus determined based on their similarity to a reference solution. in early work, reference-based approaches mainly employed feature-engineering methods such as utiliz-ing string-based or corpus-based similarity methods (gomaa and fahmy, 2014) and n-grams @xcite . more recently, meccawy et al. conducted a comparative study evaluating the efficiency of different word embedding approaches for conducting feature vectors. in their study, wang et al. introduced innovative metrics for score-based similarity to construct a text representation space that is optimized for both inter and intra-level distinctions, leading to improved scoring efficiency. in our reference-based approach, we define scorebased similarity in a manner similar to what they have presented in their research. ## dataset description in this study, we utilize the ar-asag dataset, which is the first publicly available dataset for automatic short-answer scoring in arabic @xcite . the dataset consists of 2133 short answers written by graduate students in response to 48 questions. the questions are taken from 3 different exams on cybercrime where each exam consists of 16 questions. the question prompts in the exams could be classified into 5 categories based on the type of answer they expect, namely: define, explain, consequences, justify, and compare. the answers in this dataset were independently annotated by two human raters on a scale of 0 to 5 where 0 is completely incorrect and 5 is considered a perfect answer. the raters were instructed to assign a score based on the similarity of the student's answers to a reference answer given for each question. determining the similarity between two answers not only is a subjective task but also requires a deep understanding of the topic. in cases like this, where no detailed scoring rubric is provided, the raters can find it especially difficult to determine the precise degree of similarity. this is reflected in the low inter-rate agreement of 35%. however, this is expected since the raters were also given the freedom to assign intermediate scores such as 4.5 or 3.25, etc. in our study, we treat the scoring problem as a classification problem instead of a regression one. we discretize the scores into 6 categories, 0, 1, 2, 3, 4, and 5 by taking the rounded-down median after ceiling the scores to the nearest 0.5. this is done to increase the inter-rate agreement to 56% instead of 35%. the distribution of the scores in the dataset can be seen in figure 1 where we can observe the majority of the scores being concentrated in the range of 3 to 4. ## methodology problem formulation we formulate the problem of automatic short answer scoring as follows: given a question q, a short answer a, and a reference answer a * , the goal is to predict a score @xmath0, 5] that represents the quality of the answer. usually, each question is treated as a separate task where a separate model is trained for each task or question. however, this approach is not feasible in low-resource settings where there is a lack of annotated data. hence, we propose a general in-context prompt-based scoring framework for automated scoring of short-answer questions where we divide the scoring problem into two sub-problems, prompt-category-based scoring and prompt category classifying. the prompt-category-based scoring problem can be formally defined as follows: we have a set of tasks @xmath1, where each task t i is defined by a question q i , its reference answer a * i , and a set of instances in order to facilitate the selection of the most suitable prompt-category-based scoring model for a given question, an auxiliary prompt category classifier is trained to identify the prompt category of a given question. the output of this model should then serve as a guide for selecting the most suitable model for a given question. this is illustrated in figure 2 . ## conclusion in this paper, we propose a general in-context prompt-based scoring framework for automated scoring of short-answer questions. we divide the scoring problem into two sub-problems, namely, prompt category classification and promptcategory-based scoring. for prompt-category classification, we utilize a few-shot, prompt-free framework to train the model. we also show that with data augmentation using gpt3.5, the performance could be significantly increased. we then propose two main approaches for the prompt-categorybased scoring problem, namely, instance-based incontext meta-learning and reference-based semantic similarity. utilizing the only publicly available arabic asag dataset, we evaluate both approaches in their ability to generalize to unseen answers and unseen questions. experimental results show that both proposed approaches outperform the majority class classifier and are comparable to human raters when grading unseen answers. however, the performance is highly prompt-dependent and no particular approach is consistently better than the other. in zero-shot settings, when generalizing to unseen questions, we observe a tendency for the reference-based semantic similarity approach to outperform the instance-based in-context metalearning approach. we thus believe that in classroom settings, the reference-based semantic similarity approach could be a more suitable solution due to its superiority in zero-shot settings.