Closing the Loop: Learning to Generate Writing Feedback via Language Model Simulated Student Revisions (2024)

Abstract: Providing feedback is widely recognized as crucial for refining students’ writing skills. Recent advances in language models (LMs) have made it possible to automatically generate feedback that is actionable and well-aligned with human-specified attributes. However, it remains unclear whether the feedback generated by these models is truly effective in enhancing the quality of student revisions. Moreover, prompting LMs with a precise set of instructions to generate feedback is nontrivial due to the lack of consensus regarding the specific attributes that can lead to improved revising performance. To address these challenges, we propose PROF that PROduces Feedback via learning from LM simulated student revisions. PROF aims to iteratively optimize the feedback generator by directly maximizing the effectiveness of students’ overall revising performance as simulated by LMs. Focusing on an economic essay assignment, we empirically test the efficacy of PROF and observe that our approach not only surpasses a variety of baseline methods in effectiveness of improving students’ writing but also demonstrates enhanced pedagogical values, even though it was not explicitly trained for this aspect.

## introduction writing high-quality essays often requires subjectspecific and customized feedback from peers and experts, followed by multiple rounds of revisions @xcite @xcite . as students incorporate feedback into their writing, they not only improve the current piece but also advance the general writing skills, learn to critically self-assess their work @xcite , and gain a deeper understanding of the subject matter @xcite . recent advances in language models (lms) @xcite @xcite make it possible to develop automatic feedback generation systems to provide concrete and actionable comments in a timely manner @xcite , compared to the time-consuming process performed by humans. however, careful prompt engineering is necessary to incorporate precise instructions, ensuring that the generated feedback effectively guides students in improving the quality of their writing. more importantly, providing such detailed instructions is not a trivial task since there is still no general consensus about what attributes the feedback must entail to effectively contribute to students' learning outcomes @xcite . for example, @xcite show that including explanations in feedback can only improve the writing quality of specific revisions, and sometimes (e.g., if too lengthy) can negatively affect tenth-graders' overall writing performance @xcite . to this end, our goal is to build an automatic feedback generation system that can be directly optimized to maximize students' writing revision performance, to avoid the complexity of explicitly specifying the criterion for effective feedback. however, involving actual students at every stage of the system-building process is impractical due to the time required and the potential negative impact on participants from an immature system @xcite . to address this challenge, we first develop an lm-based student simulator that emulates the process of applying feedback to revise initial content, inspired by the recent efforts to simulate human processes @xcite @xcite . in our empirical evaluations of lm simulators, we discovered that by varying the temperature used in autoregressive decoding, we can effectively simulate a diverse array of behaviors to support a comprehensive testing of the feedback generator. we then propose a feedback generation model, prof foot_0 , that produces feedback via learning from lm-simulated student revisions. concretely, we use the lm student simulator to iteratively generate preference relations involving desirable and undesirable feedback. we then apply the direct preference optimization (dpo) objective @xcite @xcite along with the preference relations to update the feedback generator. importantly, the iterative process aims to enhance the effectiveness of the generated feedback, resulting in better implementation performance according to the student simulator. to evaluate the feedback generated by prof, we conduct a study on an essay assignment from an introductory economics course offered at a university in united states. in our experimental analyses, we compared the performance of our model with that of few-shot prompted gpt-3.5/gpt-4 models along with other nontrivial comparisons. our approach not only achieves a similar level of pedagogical alignment, but also outperforms these enterprise lms in terms of implementation performance of essay revising. notably, our model is significantly smaller in size (8 billion parameters), making it more efficient and can be easily adapted to other writing assignments. furthermore, in our empirical experiments involving the student simulators, we observed that the feedback generated by our model aligns well with the actual human revisions, demonstrating its faithfulness in implementation. to summarize, our work makes the following major contributions: ## data description we collected data from the essay assignments submitted by students enrolled in the economics 101 course at the university of michigan, ann arbor, united states. this assignment explores a scenario in which "an increase in the minimum wage in san francisco could lead to burgeoning adoption of automation". to discourage this outcome, two policies are proposed: a) a tax on automation and b) a ban on automation. the students are instructed to craft a persuasive letter explaining the economic consequences of a minimum wage increase. furthermore, they are tasked with presenting arguments against one of the aforementioned policies by using the tools and principles taught in the course. refer appendix a.1 to view the assignment prompt. thereafter, each student essay is reviewed by 3 peers to obtain a set of 3 feedback which identifies areas for improvement. this assignment uses scripted peer feedback @xcite wherein the peer reviewers are expected to provide feedback along a series of prompts. refer appendix a.2 to view the questions in the scripted peer feedback. finally, the author revises the original essay based on the received peer reviews. in total, we collected 363 datapoints, each comprising the initial writing, three peer feedback, and the revised essay. among these, we utilized 291 essay-revision pairs along with the 873 (291 × 3) feedback for our train split, while the remaining 72 datapoints were used for testing. as demonstrated later, our approach yields enhanced performance despite the limited number of datapoints and the absence of expert annotated feedback. this assignment includes a detailed rubric for evaluating and grading student-written essays. our approach utilizes this rubric to create an essay evaluation prompt for assessing revisions produced by lm student simulators. this prompt can be found in appendix a.3. initial writings generating feedback revisions from student simulator scoring revisions for preference pair selection preference relations dpo model for next iteration at each iteration t, the feedback generator m t generates multiple feedback samples, which are then evaluated for their effectiveness using the student simulator and then gpt-4 as a judge. these evaluations are used to establish preference relations over feedback using the quality of the corresponding revised essays , which are subsequently used to update the parameters of m t via dpo @xcite , resulting in the updated version m t+1 . ## prof: learning to generate feedback with simulated student revisions to directly optimize the feedback generation on student revising performance, we have two lms that function as feedback generator and student simulator respectively, as illustrated in figure 1 . in §3.1, we first describe the training of student simulators using the data from §2, to emulate how students integrate feedback into revisions. in §3.2 and §3.3, we present how the feedback generator is initialized and iteratively optimized by the proposed prof method based on desirable and undesirable feedback, as measured by simulated revisions' quality. ## analysis of student simulators in this section, we analyze the alignment between the properties of revision in student simulators and actual students, based on peer written feedback. concretely, we examine the impact of the temperature parameter on the number of modifications ( §4.1), revision performance ( §4.2), and faithfulness to feedback ( §4.3), and compare it with the real students' revision process. ## feedback generator setups for our iterative optimization approach, we use llama3-8b based student simulator for training and emphasize on gpt-3.5 based one for testing. we do this for the following three reasons: (1) gpt-3.5 based student simulator better aligns with actual student revision, as demonstrated in §4.2 and thus provides a more realistic testing environment to measure implementation performance. (2) using gpt-3.5 based student simulator is prohibitively expensive when used in conjunction with an iterative optimization approach. using llama3-8b as a student simulator makes our research more accessible due to lower training cost and easy access to the open-source models. (3) training and testing on the same student simulator would not provide conclusive evidence of the effectiveness of our approach, as it might learn to exploit one type of student simulator while performing poorly on others. for completeness, however, we also include the effectiveness of the generated feedback using the llama3-8b based student simulator. refer appendix b for more details. ## results for feedback generation we consider the following types of baselines: (1) gpt-3.5 / gpt-4: using enterprise llms as a few-shot feedback generator by sampling incontext examples from peer-written feedback from the train-split. (2) sft-from-human: fine-tuning llama3-8b on peer review feedback. our method variants are named as "prof, τ = x" where the llama3-8b based feedback generator is initialized with sft-from-human and trained using the iterative optimization framework green and blue represents best and second-best performance respectively. ## conclusion in this paper, we present an optimization technique called prof that focuses on maximizing students writing revision performance through lm simulation. we conducted extensive analysis to showcase the alignment between our student simulators and actual revisions, while also demonstrating the flexibility of adjusting temperature to elicit diverse behaviors from the models. through experiments, we show that our trained models not only exhibit better effectiveness but are more pedagogically aligned.