Predicting Desirable Revisions of Evidence and Reasoning in Argumentative Writing (2023)

Abstract: We develop models to classify desirable evidence and desirable reasoning revisions in student argumentative writing. We explore two ways to improve classifier performance – using the essay context of the revision, and using the feedback students received before the revision. We perform both intrinsic and extrinsic evaluation for each of our models and report a qualitative analysis. Our results show that while a model using feedback information improves over a baseline model, models utilizing context - either alone or with feedback - are the most successful in identifying desirable revisions.

## introduction successful essay writing by students typically involves multiple rounds of revision and assistance from teachers, peers, or automated writing evaluation (awe) systems. natural language processing (nlp) has become a key component of awe systems, with nlp being used to assess the content and structure of student writing and to automatically provide formative feedback @xcite @xcite . while some students produce revised texts that are in line with the feedback automatically generated by a system or provided by other humans, other students either ignore the feedback or are unsuccessful in their feedback implementation attempts @xcite . hence, analyzing student revisions in terms of their desirability for improving essay quality is important. the development of awe systems that leverage nlp to analyze a revision's alignment to feedback messages is one approach to convey to students a sense of a good revision direction. our research focuses on the automatic classification of desirable and undesirable revisions of evidence use and reasoning 1 in argumentative writ1 such revisions of text content are generally considered most important in revising @xcite . in this paper, we first describe the labeling of desirable and undesirable revisions in three existing corpora of evidence and reasoning revisions. we then describe a baseline model and enhanced models using context and feedback information to predict revision desirability. finally, we present results from intrinsic and extrinsic evaluations to demonstrate the utility of our enhanced models. feedback message: "....explain how the evidence helps to make your point... ... tie the evidence not only to the point you are making within a paragraph, but to your overall argument..." ## data and resources our data consists of three corpora of paired drafts of argumentative essays, written in response to a prompt and revised in response to feedback. a comparison of the data is shown in the elementary school students wrote draft1 about an article on a project in kenya, then received awe system feedback focused on students' use of text evidence and reasoning (selected based on automatic scoring). an example of the feedback messages is shown in the high-school students wrote draft1 in response to a prompt about dante's inferno @xcite , then received peer feedback along 6 rubric dimensions (e.g., evidence, organization, etc.). we only utilize feedback about evidence in this work (shown in the college essays were written by 60 students on technology proliferation @xcite then revised to create draft2, then revised again without any further textual foot_1 feedback to create draft3. drafts 2 and 3 were later graded by experts based on a rubric. we create a binary improvement score for each essay pair, calculated as 1 if draft3 improved compared to draft2, -1 otherwise. for all corpora, sentences from the two drafts were aligned manually based on semantic similarity. aligned sentences represent one of four operations between drafts -no change, modification, sentence deleted from draft1, sentence added to draft2. each pair of changed aligned sentences was then extracted as a revision (rows 3, 5 and 9 in finally, to understand how students revise evidence and reasoning, whether their revisions were desirable, and whether desirable revisions relate to measures of essay improvement, we then applied the evidence and reasoning revision categorization scheme developed in @xcite . in this scheme, revisions related to evidence are characterized by five codes -relevant, irrelevant, repeat evidence, non-text based, and minimal. reasoning revisions are characterized by six codes -linked claim-evidence (lce), not lce, paraphrase evidence, generic, commentary, and minimal. the annotation was done by an expert familiar with the coding scheme (cohen's kappa in a previous study was 0.833 for evidence and 0.719 for reasoning). ## predicting revision desirability in this section, we describe the models for automatically classifying desirable revisions. first, we describe a data augmentation process to increase the training data. then we describe a model to identify revision desirability, and extend it to use context and the feedback information. we setup our models to answer the following research questions: rq1: is the context of the revision predictive of revision desirability? rq2: is the feedback received before revising the essay predictive of revision desirability? rq3: do the context and feedback together boost the identification of desirable revision? ## qualitative analysis in order to better understand the model predictions, in we think @xmath05 dollars isn't that much money but they live in situations where $5 is a weeks worth of money. d e d u u d u they had water, food, electricity, supplies, medicine, and simple things. u e d u u d u the second example is a desirable evidence predicted as undesirable by context and desirable by the feedback model. the awe feedback asked the student to use more evidence and add details. we think the feedback model tied the extra information in the modified sentence to what was asked for. the last example is an undesirable evidence predicted correctly only by the models using context information. although the example text resembles a desirable evidence, it is actually undesirable because it was repeated. obviously, the model needed context to identify that it is a repeated evidence. ## conclusion in this study, we presented new models for the automatic identification of desirable revisions in three corpora of argumentative writing varying in writer's level of expertise, source of feedback, and grading rubrics. we presented a new method of extracting context from essay revisions. using intrinsic and extrinsic evaluation we showed that models using the context information performed best in identifying desirable revisions. we also studied the use of feedback messages received by students to predict desirable revisions. to the best of our knowledge this is the first model to use feedback information to analyze student revision. our experiments showed that feedback information also helped improve classifier performance, particularly when used with context. we have released the college data annotated with revision desirability. it can be downloaded from this link: https://petal-cs-pitt.github.io/ data.html . the code is also available from here: https://github.com/tazin-afrin/ desirable-revision-classification sity of pittsburgh. we would like to thank the anonymous reviewers for taking the time to review our paper and provide us with detailed feedback. we would also like to thank the members of the petal lab for their valuable feedback. the opinions expressed are those of the authors and do not represent the views of the institute.