Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation (2024)

Abstract: Individual feedback can help students improve their essay writing skills. However, the manual effort required to provide such feedback limits individualization in practice. Automatically-generated essay feedback may serve as an alternative to guide students at their own pace, convenience, and desired frequency. Large language models (LLMs) have demonstrated strong performance in generating coherent and contextually relevant text. Yet, their ability to provide helpful essay feedback is unclear. This work explores several prompting strategies for LLM-based zero-shot and few-shot generation of essay feedback. Inspired by Chain-of-Thought prompting, we study how and to what extent automated essay scoring (AES) can benefit the quality of generated feedback. We evaluate both the AES performance that LLMs can achieve with prompting only and the helpfulness of the generated essay feedback. Our results suggest that tackling AES and feedback generation jointly improves AES performance. However, while our manual evaluation emphasizes the quality of the generated essay feedback, the impact of essay scoring on the generated feedback remains low ultimately.

## introduction essay writing is a valuable skill that empowers students to communicate effectively, to think critically, and to express their opinions clearly @xcite @xcite . however, learning to write essays of high quality can be challenging @xcite . individual feedback to essays is thus critical, so students may recognize and overcome their weaknesses @xcite . however, manually writing essay feedback is time-and laborintensive. given educators' limited time, this often renders real individual feedback infeasible in practice. as an alternative, automated essay writing support could benefit students by offering guidance at their own pace and convenience. ## approach this section describes our approach to essay feedback generation. we propose to tackle essay scoring and feedback generation jointly in order to study how and to what extent aes can benefit essay feedback generation and vice versa. by comparing different prompting strategies for large language models (llms), we explore how well the tasks can be solved using in-context learning. in particular, we test different prompting strategies by systematically varying three main aspects of the prompts, as visualized in figure 2 : (a) the prompt pattern, which defines the context and layout; (b) the task instruction type, which sets the ordering and phrasing of the tasks to be tackled; and (c) the in-context learning approach, which specifies the number of examples provided. ## data multiple aes datasets are available, with the automated student assessment prize's (asap) dataset @xcite being the most widely used. it comprises 12,980 essays written by school students in grades 7 to 10. all essays were scored manually by two raters. the essays are divided into eight essay sets. the essay sets differ by the essay prompt, i.e., the task description they were written for, the scoring range, and the rubric used by the raters as annotation guidelines. the rubrics provide a short description for essays of each quality level and typical elements of such essays. since for the introduced task of essay feedback generation, no parallel dataset is available yet, we use the asap dataset as input data and evaluate the generated feedback without supervision. ## evaluation we evaluate the performance of a large language model (llm) by comparing the proposed prompting strategies on the two tasks: essay scoring and feedback generation. first, we assess the scoring performance and, then, we both automatically and manually evaluate the generated feedback in terms of the helpfulness for the student writer. we aim to study the effects of tackling essay scoring and feedback generation jointly, as well as explore how well llms can solve both tasks using prompting. ## conclusion despite the strong text generation abilities of recent llms in various tasks, their effectiveness in generating essay feedback that helps student writers improve their essays has remained unclear until now. also, generating textual feedback that addresses the entire essay has previously only been tackled using one prompting strategy in a zero-shot learning setting. with this work, we go beyond existing work by comparing different llm prompting strategies for essay feedback generation. we propose tackling essay feedback generation and automated essay scoring (aes) jointly to study whether aes can benefit feedback generation and vice versa. our experiments suggest that aes can be solved competitively by prompting llms, benefitting from tackling feedback generation first. the generated feedback is deemed helpful for students by our automatic and manual evaluation. however, the impact of scoring on the feedback helpfulness remains low ultimately.