Unleashing Large Language Models’ Proficiency in Zero-shot Essay Scoring (2024)

Abstract: Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we show that our zero-shot prompting framework, Multi Trait Specialization (MTS), elicits LLMs’ ample potential for essay scoring. In particular, we automatically decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.

## introduction automated essay scoring (aes) aims at evaluating and scoring essays with machine learning @xcite . aes is a promising alternative to costly and laborious human assessment, greatly resolving rater fatigue and inter-rater inconsistency. aes systems have been widely deployed in classroom settings @xcite and high-stakes tests such as toefl @xcite . previous studies highly matched human ratings via developing supervised models tailored to a specific prompt @xcite * corresponding author. @xcite , assuming essays from the train and test sets belong to the same prompt. however, prompt-specific models struggled when confronted with essays written for unseen prompts @xcite . hence aes is progressing towards reflecting more real-world scenarios, exemplified by cross-prompt approaches @xcite which reinforce domain transferability of the supervised models. recently, advances in large language models (llms) @xcite @xcite have led to a paradigm shift in which llms excel across a wide range of downstream tasks via zero-shot or fewshot instructions @xcite . in many cases, careful prompt design plays a crucial role in unlocking llms' potential. for instance, chain-of-thought (cot) prompting @xcite improves llms' performance on complex reasoning benchmarks by externalizing the reasoning process. the development of llm-based chatbots aligned with human preferences @xcite has given rise to zeroshot aes, allowing us to move beyond the crossprompt setting. however, leveraging llms for zero-shot aes is less explored, in contrast to pro-liferating studies harnessing llms to serve as an evaluation metric for machine-generated text @xcite @xcite . initial works of zero-shot aes @xcite prompt llms to assign the overall score within a single step, which demonstrates suboptimal agreement with human raters. in this paper, we present mts (multi trait specialization), a zero-shot prompting framework to elicit essay scoring capabilities in llms, inspired by supervised models that explicitly predict trait scores and improve the overall scoring @xcite @xcite . in particular, we exploit chatgpt @xcite to decompose the writing quality into multiple traits and generate scoring criteria for each trait. next, an llm engages in several rounds of conversation, each round evaluating with respect to one of the traits. during the conversation, the llm is instructed to retrieve quotes to provide faithful evaluation on the essay, then assign a score based on the given scoring criteria. finally, the overall score is derived by averaging and min-max scaling the trait scores, in combination with the outlier clipping mechanism. we evaluate mts on asap @xcite and toefl11 @xcite with different llms, including chatgpt, llama 2 @xcite and mistral 7b @xcite . we take the vanilla approach as a primary baseline which asks llms to produce rationales followed by an overall score. as illustrated in figure 1 , mts consistently outperforms vanilla in average quadratic weighted kappa (qwk) across all combinations of llms and datasets, with maximum gains of 0.437 (0.025 → 0.462) on toefl11 and 0.355 (0.205 → 0.560) on asap. in addition, the small-sized llama2-13b-chat substantially surpasses chatgpt with the help of mts, enabling a more effective deployment. in essence, our contributions are as follows: ## method we formalize the definition of zero-shot aes as follows: given a dataset consisting of unlabelled essays @xmath0, the goal is to output an overall score ŷ(i) for every essay x (i) from a set of predefined scores y where ŷ(i) ∈ y. multi trait specialization encourages llms to assess the essay from diverse aspects of writing quality. it consists of three steps: (1) decomposing writing proficiency into multiple traits and generating scoring criteria; (2) assigning a trait-specific score by step-by-step evaluation specialized to the trait; (3) deriving the final score via trait aggregation and scaling. the overall architecture of mts is illustrated in figure 3 . ## main results by comparing different llms, we observe that mistral-7b-instruct achieves the most competitive performance under different prompting methods overall, reaching average qwk of 0.550 on asap and 0.587 on toefl11. interestingly, chatgpt underperforms the small-sized mistral-7b-instruct across all methods but rubric scoring, implying that the model size might not be a decisive factor in the performance. nevertheless, the larger model tends to perform better within the same model type, evidenced by the comparison between llama2-7bchat and llama2-13b-chat. in terms of the comparison with the supervised sota, the gap in average qwk between mts and npcr can be narrowed down to 0.171 on toefl11 and 0.242 on asap using mistral-7binstruct, as shown in figure 4 . this can be promising given that npcr consumes approximately a thousand labeled essays per prompt to reach the sota. we expect further reduction in the gap with more powerful llms. ## analysis we conduct experiments to derive insights into the success of mts by analyzing each of the modules in mts.