LLM s in Short Answer Scoring: Limitations and Promise of Zero-Shot and Few-Shot Approaches (2024)

Abstract: In this work, we investigate the potential of Large Language Models (LLMs) for automated short answer scoring. We test zero-shot and few-shot settings, and compare with fine-tuned models and a supervised upper-bound, across three diverse datasets. Our results, in zero-shot and few-shot settings, show that LLMs perform poorly in these settings: LLMs have difficulty with tasks that require complex reasoning or domain-specific knowledge. While the models show promise on general knowledge tasks. The fine-tuned model come close to the supervised results but are still not feasible for application, highlighting potential overfitting issues. Overall, our study highlights the challenges and limitations of LLMs in short answer scoring and indicates that there currently seems to be no basis for applying LLMs for short answer scoring.

## introduction the constantly increasing demand placed on educators in today's educational landscape requires innovative solutions to replace traditional assessment methods. manual assessment, especially for large-scale exams, presents challenges for scalability, consistency and timely feedback to students @xcite . automated scoring has emerged as a potential solution, promising faster, more objective and feedback-rich assessments @xcite . extensive research has explored automated scoring, but many systems require large amounts of training data to achieve reliable performance @xcite . our focus is on finding a system that demonstrates strong performance across different datasets while minimizing the need for huge number of training examples. large language models (llms) seem promising in this regard @xcite . thus, in this paper, we explore llms performance in scoring openended student answers across three datasets. we compare two prominent llms, generative pretrained transformer (gpt) and large language model meta ai (llama), under different training settings, including zero-and few-shot learning, and fine-tuning specifically applied to the gpt model. additionally, we benchmark their performance against established baselines, specifically google's pre-trained language model bert @xcite and classical svm , known for its robustness in classification tasks @xcite . this evaluation aims to deepen our understanding of how llms handle various assessment tasks and shed light on their potential to enhance automated scoring in education, particularly with limited training data. ## related work very few studies have explored the performance of llms in zero-and few-shot settings within the context of automated scoring. @xcite introduced the matching exemplars as next sentence prediction (mensp) method, by employing a zero-shot prompt learning method using pre-trained language models. their results indicate that fewshot learning offered limited improvement in performance. latif and zhai (2024) compare the performance of a fine-tuned gpt-3.5 model with bert and demonstrated that gpt-3.5 achieved higher scoring accuracy. it showed a remarkable average increase of 9.1% compared to bert when applied to a single dataset of six assessment tasks. this finding emphasized the need for domain-specific fine-tuning llms to enhance their performance. on the other hand, many studies investigated the neural networks and machine learning models to build scoring tools. @xcite demonstrate how pretrained transformer models could be adapted for content scoring using an instance-based approach. by pooling token representations across all model layers, this approach achieved state-of-the-art performance on short answer scoring tasks. @xcite conduct a comparison between instance-based and similaritybased methods on multiple datasets. they investigated the influence of different training set sizes on the performance of these methods using learning curve experiments. it found that a fine-tuned sbert model does often yield the best results. overall, existing research offers limited insight into how llms perform in zero-shot and few-shot settings. ## experimental setup we tested the gpt family of models introduced by openai, specifically gpt-3.5 and gpt-4. 1 additionally, we tested meta ai's llama-2 models llama-7b, llama-13b, and llama-70b foot_1 . finally, google's bert model and svm were included as baselines for comparison. for testing, we randomly selected 20% of each task from the datasets. we observed that llms usually produce in addition to the score an explanation, or repeat the scores of the given shots, rather than providing only the score, so we applied a filtering function that retrieves only the last integer of the llms response, and if no integer was found, we assigned a randomly generated number between 0 and the maximum possible score of the current task. ## results & discussion ## conclusion this study explores the potential of llms in automated scoring tasks, specifically zero-and fewshot, and fine-tuned settings across three diverse datasets. overall, our findings reveal strong performance from zero-shot and few-shot models on general knowledge. gpt-4 achieved performance very close to the upper bound bert and outperformed svm model. llama models showed promising results; while not reaching gpt-4 levels, their performance remained consistent across different numbers of shots. in contrast, gpt-3.5 appeared overfitting as more shots introduced. this highlights the potential of few-shot llms for short answer scoring, especially on tasks involving general knowledge questions. however, llms face challenges when confronted with tasks that require complicated reasoning or domain-specific knowledge, as noticed by their poor performance in asap dataset. the complicated nature of the questions in these subjects appears to cause difficulties for llms, highlighting the need for further improvements in dealing with nuanced and specialized content within educational datasets. with regard to the fine-tuned model, our study revealed unexpected results as it failed to meet our performance expectations for automated scoring. it became clear that the model was overfitting at certain questions 'tasks', particularly noticeable when examining the performance of the powergrading dataset.