Using Large Language Models to Assess Young Students’ Writing Revisions (2024)

Abstract: Although effective revision is the crucial component of writing instruction, few automated writing evaluation (AWE) systems specifically focus on the quality of the revisions students undertake. In this study we investigate the use of a large language model (GPT-4) with Chain-of-Thought (CoT) prompting for assessing the quality of young students’ essay revisions aligned with the automated feedback messages they received. Results indicate that GPT-4 has significant potential for evaluating revision quality, particularly when detailed rubrics are included that describe common revision patterns shown by young writers. However, the addition of CoT prompting did not significantly improve performance. Further examination of GPT-4’s scoring performance across various levels of student writing proficiency revealed variable agreement with human ratings. The implications for improving AWE systems focusing on young students are discussed.

## introduction the ability to write is foundational to academic success. yet, national assessments show that nearly three-quarters of students in the united states are not proficient writers @xcite . a wellrecognized approach for improving students' writing skills is to engage students in cycles of revising their essays in response to formative feedback @xcite . however, students rarely receive substantive formative feedback on their writing for multiple reasons. first, teachers can be reluctant to assign writing tasks that require students to work across drafts because providing formative feedback is time-consuming @xcite . second, teachers can feel unsure about how to provide feedback to improve students' essay quality @xcite . finally, research shows that teachers are inconsistent in their feedback practices, and tend to focus on surface-level features of students' writing rather than the content of students' ideas and reasoning @xcite . automated writing evaluation (awe) systems are gaining prominence as one approach to increasing students' opportunity to receive formative feedback. while research suggests that teachers generally respond positively to awe systems and can see them as helpful time savers @xcite , evidence is modest that awe systems improve the quality of students' writing in the elementary and secondary grades @xcite . one reason why students' writing may not improve in response to automated feedback is that they often lack the skills necessary for effective revision @xcite . @xcite found that only 18% of students successfully implemented the feedback messages they received from an awe system. for example, when asked to provide more evidence for their claims, students commonly repeated the examples that they had cited before. this highlights the importance of providing students with feedback that builds their revision skills, in addition to feedback that improves their writing quality. given that formative assessment fosters writing skill development by establishing and reinforcing clear criteria for successful writing @xcite , it is notable that few assessments target students' revision skills. building on the previous discussion about the necessity of teaching students how to revise, we believe that formative assessments that precisely establish the criteria for effective revision can provide information to students and teachers about the extent to which revision goals are met and offer guidance for implementing revision feedback. to address this gap, our team developed a rubric for holistically assessing revision quality @xcite . by 'revision quality', we specifically examine whether revisions students made were aligned with the feedback provided, and the extent to which it improved the essay with respect to evidence use. this is in contrast to revisions that may improve essay quality in ways not aligned with the content of feedback messages. in the context of awe systems, automatically assessing the revision process is a necessary area of development. most systems have focused on assessing overall improvement in essay quality. although these systems can detect revisions, they tend to assign scores or provide feedback based on the overall essay quality, rather than attend to the quality of the revisions undertaken @xcite . recent advancements in large language models (llms) show significant promise for analyzing and evaluating student revision quality. gpt-4, standing out among these models, specifically has been shown to generate scores that are comparable to those given by human evaluators @xcite @xcite @xcite . while most of these studies have concentrated on gpt-4's ability to assess writing quality, our study extends previous research by investigating the effectiveness of gpt-4 for evaluating revision quality with different prompting strategies. given that students often find essay revision challenging, it is essential to provide a revision score that reflects diverse revision patterns. this study represents an initial step in exploring gpt-4's capability to score revisions, setting the stage for offering personalized feedback on students' revision practices in future research. in this study, we specifically explore gpt-4's performance in assessing the revision attempts of young students (ages 10 to 12) who often exhibit less structured and sophisticated writing styles. given that most existing research concentrates on evaluating essays by adolescents and adults (e.g., @xcite , it is of interest to explore how gpt-4 adapts to the writing of younger age groups. in addition, as students may display a wide range of writing proficiency, it is crucial to ensure that gpt-4 does not exhibit systematic biases that could compromise scoring accuracy. two research questions are addressed: ## data in this section we describe the dataset of students' essays, the rubric used for assessing students' revision quality, and the process for evaluating these revisions by human raters. ## discussion and conclusions revising is a very difficult skill to master, and many young students struggle to implement the feedback they receive @xcite . to foster the development of students' revision skills, assessing revision quality and identifying revision patterns across various levels of writing proficiency is essential for providing targeted feedback to students on their revision efforts. with this aim, this study explored the potential of using a large language model, specifically gpt-4, to evaluate the quality of essay revisions aligned with the feedback messages students received from an awe system. first, our results suggest that gpt-4 has a great deal of potential for effectively evaluating writing revision quality. we used a detailed rubric providing specific revision patterns in the zero-shot (baseline experiment) prompting and one-shot cot prompting and both approaches showed a moderate level of agreement between human raters and gpt-4. however, both cot prompting strategies implemented in the study did not improve gpt-4 baseline performance. it is not altogether clear why this was the case as other researchers have found that cot prompting tends to improve the accuracy of writing quality scores @xcite . we note, however, that evaluating the quality of revisions in younger students' essays may be a more complex task than assessing overall quality. it contains a series of evaluative steps beyond simply identifying revision patterns with a rubric. this includes interpreting feedback messages, identifying what was added in second drafts, and evaluating the alignment of those additions to the feedback. we recommend that future research explore additional prompting strategies to better address this complexity. for example, treeof-thoughts prompting, which encourages llms to explore various ideas and assess intermediate steps in order to provide an optimal response @xcite , could be a useful way forward for generating more accurate assessments of complex writing processes. secondly, unlike studies that focus on adult writers such as college students, our research provides insight into the capabilities of llms to assess the writing produced by young students. we observed that gpt-4 tended to assign lower scores to revisions than human raters. one reason for this might be that fifth and sixth graders are still in the midst of developing their language as well as reasoning skills. the changes they make to their essays are constrained then by their overall ability to elaborate and explain their thinking in writing. human raters took into account the age of students, and what they deemed reasonable to expect for revision at that age, and gave credit for effort (incremental changes) rather than only the quality of students' final product. unlike human raters then, gpt-4 may lack knowledge of developmentally appropriate expectations for student writing which potentially affects its scoring accuracy. therefore, llms would benefit from tailored training to adjust their criteria for "good" writing to be calibrated for differentaged students.