Predicting the Quality of Revisions in Argumentative Writing (2023)

Abstract: The ability to revise in response to feedback is critical to students’ writing success. In the case of argument writing in specific, identifying whether an argument revision (AR) is successful or not is a complex problem because AR quality is dependent on the overall content of an argument. For example, adding the same evidence sentence could strengthen or weaken existing claims in different argument contexts (ACs). To address this issue we developed Chain-of-Thought prompts to facilitate ChatGPT-generated ACs for AR quality predictions. The experiments on two corpora, our annotated elementary essays and existing college essays benchmark, demonstrate the superiority of the proposed ACs over baselines.

## introduction argumentative revision (ar) in response to feedback is important for improving the quality of students' written work. successful ars 1 usually include adding relevant evidence, deleting repeated evidence or reasoning, and elaborating relevant evidence examples to support claims @xcite . differentiating between successful versus unsuccessful ars, however, is a complex endeavor. for example, making the same ar in distinct argumentative contexts (acs) could differentially affect the quality of a student's essay. here the acs are defined as pieces of sentences that present reasons, evidence, and claims supporting or opposing arguments in argumentative writing (see sec. 4.2). for example, figure 1 shows two pieces of ars that added the same sentence "it was hard for them to concentrate though, as there was no midday meal" but caused opposite ar quality. recently developed automated writing evaluation (awe) systems have focused on assessing the content and structure of student essays to automatically provide students with formative feed1 @xcite use the term desirable revisions. ar #372: unsuccessful @xmath0. according to the text, many kids in sauri did not attend school because their parents could not afford school fees. </original @xmath1, as there was no midday meal. </@xmath2: successful @xmath3, paper, and pencils, but the students wanted to learn. all of them worked hard with few supplies they had. </original @xmath4, as there was no midday meal. </@xmath5: two pieces of ars in two student essays show that adding the same sentence "it was hard for them to concentrate though, as there was no midday meal" (bold in red) in different contexts caused opposite ar quality. ar #372 added a piece of evidence that already existed in the original draft (bold in blue) thus the attempted ar did not improve the essay quality. ar #592 improved the quality by adding a relevant piece of new evidence. ar #372 was unsuccessful while ar #592 was successful. ## experiments to answer the rqs, we implemented locationbased contexts as baselines, and a series of acs as comparable methods: we also evaluated the effectiveness of acs on the college essay benchmark. note that @xcite conducted data augmentation with a simple synonym replacement because they argued that it was impossible to obtain reasonable results without training on augmented data. we hypothesized that data augmentation will introduce noise but the limited data can yield reasonable results training with the distilroberta-based model. therefore, we did not do data augmentation and compared ac-based methods to our implemented standard and strong baselines on reasoning revi-revision #372: ac-@xmath6, including free medicine and bed net provision, as well as the positive impact of providing lunch to children, to argue that poverty can be reduced… </@xmath7: ac-@xmath8, updated hospitals to prevent the spread of diseases and access to education… </@xmath9, while #592 is a relevant adding toward their contexts in figure 1 . as a case study, we examine the effectiveness of acs in revision #372 and #592 presented in figure 1 . the chatgpt-generated ac-claim is shown in figure 4 , where the red bold sentence "providing lunch to children, to argue that poverty can be reduced" is helpful to identify that the added sentence, "it was hard for them to concentrate though, as there was no midday meal." in revision #372 is a already existed evidence, and thus it was an unsuccessful revision. however, ac-claim in revision #592 does not show repeated but relevant information, and thus the ar is regarded as successful. ## conclusion this work studies the relationship between argumentative contexts (acs) and argumentative revisions (ars) in argumentative writing. in particular, we use zero-shot-cot prompts to facilitate chatgpt-generated acs for ar quality predictions. the experiments on our elementary essays corpus and publicly available college essays benchmark demonstrate the superiority of the proposed acs over existing location-based context baselines, which proposes a new direction for predicting ar quality. the analysis suggests that most evidence ars need reasoning acs, and reasoning ars need a diverse set of claims, evidence, and reasoning acs to predict their quality.