Annotation and Classification of Evidence and Reasoning Revisions in Argumentative Writing (2020)

Abstract: Automated writing evaluation systems can improve students’ writing insofar as students attend to the feedback provided and revise their essay drafts in ways aligned with such feedback. Existing research on revision of argumentative writing in such systems, however, has focused on the types of revisions students make (e.g., surface vs. content) rather than the extent to which revisions actually respond to the feedback provided and improve the essay. We introduce an annotation scheme to capture the nature of sentence-level revisions of evidence use and reasoning (the ‘RER’ scheme) and apply it to 5th- and 6th-grade students’ argumentative essays. We show that reliable manual annotation can be achieved and that revision annotations correlate with a holistic assessment of essay improvement in line with the feedback provided. Furthermore, we explore the feasibility of automatically classifying revisions according to our scheme.

## introduction automated writing evaluation (awe) systems are intended to help improve students' writing by providing formative feedback to guide students' essay revision. such systems are only effective if students attend to the feedback provided and revise their essays in ways aligned with such feedback. to date, few awe systems assess (and are assessed on) the extent to which students' revisions respond to the feedback provided and thus improve the essay in suggested ways. moreover, we know little about what students do when they do not revise in expected ways. for example, most natural language processing (nlp) work on writing revision focuses only on annotating and classifying revision purposes @xcite , rather than on assessing the quality of a revision in achieving its purpose. a few studies do focus on revision quality, but without relating revisions to feedback @xcite . in this study, we take a step towards advancing automated revision analysis capabilities. first, we develop a sentence-level revision scheme to annotate the nature of students' revision of evidence use and reasoning (hereafter, we refer to this as the 'rer scheme') in a text-based argumentative essay writing task. by evidence use, we refer to the selection of relevant and specific details from a source text to support an argument. by reasoning, we mean an explanation connecting the text evidence to the claim and overall argument. ## related work automated revision detection work has centered on classifying edits on largely non-content level features of writing, such as spelling and morphosyntactic revisions @xcite , error correction, paraphrase or vandalism detection @xcite , factual versus fluency edits @xcite , and document-versus word-level revisions @xcite . other research has focused on patterns of revision behavior, for example, the addition, deletion, substitution, and reorganization of information @xcite . however, these categories center on general writing features and behaviors. in the context of awe systems, this could be seen as a limitation because feedback is most useful to students and teachers alike when it is keyed to critical features of a genre -such as claims, reasons, and evidence use in argumentative writingthat are most challenging to teach and learn. some research has begun to take up the challenge of investigating student revision for argumentative writing @xcite . results show a high level of agreement for human annotation and some relationship to essay improvement, though not at the level of individual argument elements @xcite . existing schemes also lack in specificity, e.g., they do not distinguish between desirable and undesirable revisions for each argument element in terms of improving essay quality. prior work on assessing revision quality has evaluated revision in general terms (e.g., strength @xcite or overall improvement @xcite ), but without consideration of the feedback students were provided. we instead focus on analyzing revisions in response to feedback from an awe system. although prior studies have focused on all revision categories (e.g., claim, evidence, and word-usage @xcite )), we focus on only evidence and reasoning revisions that correspond to the scope of the awe system's feedback. also, we focus not only on why the student made a revision (e.g., add evidence) but also analyze if the revision was desirable or not (e.g., relevant versus irrelevant evidence). ## corpus our corpus consists of the first draft (draft1) and second draft (draft2) of 143 argumentative essays. the corpus draws from our effort to develop an automated writing evaluation system -erevise, to provide 5th-and 6th-grade students feedback on a response-to-text essay @xcite . the writing task administration involved teachers reading aloud a text while students followed along with their copy. then, students were given a writing prompt 1 to write an argumentative essay. as part of a prior exploration of students' implementation of the system's feedback, this corpus of 143 essays was coded holistically on a scale from 0 to 3 for the extent to which use of evidence and reasoning improved from draft1 to draft 2 in line with the feedback provided (wang et al., 2020) foot_1 . a code, or score, of 0 indicated no attempt to implement the feedback given; 1= no perceived improvement in evidence use or reasoning, 2= slight improvement; and 3= substantive improvement. note again that this score represents a subjective, holistic (i.e., not sentence-level) assessment of whether draft2 improved in evidence use and/or reasoning specifically in alignment with the feedback that a particular student received. we refer to this as 'improvement score' in the rest of the paper. ## revision categorization (rer scheme) we propose a new scheme for annotating revisions of evidence use and reasoning (rer scheme) that will be useful for assessing the improvement of the essay in line with the feedback provided. the initial set of codes drew from the qualitative exploration of students' implementation of feedback from our awe system @xcite , in which the authors inductively and holistically coded how students successfully and unsuccessfully revised their essays with respect to evidence use and reasoning. for example, students sometimes added evidence that repeated evidence they had already provided in draft1. or they successfully modified sentences to better link the evidence to the claim. both the initial set of codes and our awe system's feedback messages were informed by writing experts and research suggesting that strong argument writing generally features multiple pieces of specific evidence that are relevant to the argument and clear explication (or reasoning) of how the evidence connects to the claim and helps to support the argument (see, for example, @xcite @xcite ). for the present study, two annotators read through each extracted evidence or reasoningrelated revision in the context of the entire essay. they labeled each instance of revision with a code. the annotators iteratively expanded or refined the initial codes until they finalized a set of codes for evidence use revisions and another for reasoning revisions (see sections 4.1 and 4.2). together these ## evaluation of the rer scheme we evaluated our annotated corpus to answer the following research questions (rq): rq1: what is the inter-rater reliability for annotating revisions of evidence use and reasoning? rq2: is the number of each type of revision related to essay 'improvement score'? rq3: is there any difference in the 'improvement score' based on the kinds of revisions? rq4: is there a cumulative benefit to predicting essay 'improvement score' when students made multiple types of revisions? to answer rq1, we calculated cohen's kappa for inter-rater agreement on the 33 essays (23 percent) that were double-coded. our results show that we were able to achieve substantial inter-rater agreement on reasoning (@xmath0.719) and excellent inter-rater agreement for evidence use (@xmath1.833) (see, e.g., @xcite ). to answer rq2, we calculated the pearson correlation between the raw number of revisions per code to the 'improvement score' described in section 3. looking at the correlations for our proposed rer codes (which sub-categorize the evidence and reasoning codes @xcite ), we see that the rer codes yield more and generally stronger results. we found that, as hypothesized, adding relevant pieces of evidence was significantly positively correlated with the 'improvement score', while the addition of irrelevant evidence, non-text based evidence or repeating prior evidence were all unrelated to this score. similarly, we found that adding reasoning that linked evidence to the claim (lce) was significantly correlated with the 'improvement score' and so was paraphrasing evidence. other reasoning codes, as expected, were not significantly related to the 'improvement score'. we did not initially consider paraphrases as a desirable type of revision; yet, this code showed a significant positive correlation. while unexpected, we were not altogether surprised as two of the feedback messages (shown in to answer rq3, we performed one-way anovas for different levels of the 'improvement score' (0=no attempt, 1=no improvement, 2=slight improvement, 3=substantive improvement, aligned with feedback provided) comparing means of the number of revisions added, modified, or deleted. anovas showed overall significance for the categories shown in to answer rq4, we examined three stepwise linear regression models to understand whether adding more revision codes had a cumulative influence explaining more variance in 'improvement score'. m odel e included only revisions related to evidence use. m odel r included only revisions related to reasoning. m odel er included all evidence use and reasoning revisions. as shown in ## automatic rer classification the ultimate goal for developing the rer scheme is to implement it in an awe system to provide feedback to students about revision outcome not only at the essay-level but also at a more actionable, sentence-level. while the previous section demonstrated the utility of the rer scheme, this section explores its automatic classification. since our overall revision dataset is small, we focus on the simplified task of developing a binary classifier to predict whether an evidence revision is 'relevant' or not. 'relevant' is both the most frequent rer code and relates positively to the improvement score. the input is a revision sentence pair -the sentence from draft1 (s1) and its aligned sentence from draft2 (s2). the pair can have 3 variations: (null, s2) for added sentences, (s1, null) for deleted sentences, and (s1, s2) for modified sentences. since we are focusing on 'relevant' evidence prediction, and by our definition in section 4.1 'relevant' evidence supports the claim, we also consider the given source text (a) in extracting features. features. we explore word2vec as features for our classification task foot_5 . we extract representations of s1, s2, and a using the pre-trained glove word embedding @xcite . for each word representation (w) we use the vector of dimension 100, @xmath5, . . . , v 100 ]. then the sentence or document vector (d) is calculated as the average of all word vectors @xmath6, . . . , d 100 ], where d @xmath7, . . . , v ni ), for n words in the document. following this method we extract vectors d s1 , d s2 , and d a for s1, s2, and a respectively. finally, we take the average of those 3 vectors to represent the feature vector, @xmath8, . . . , f 100 ], where f @xmath9, d s2j , d aj ). for machine learning, we use off-the-shelf logistic regression (logr) from the scikit-learn toolkit. 9 we did not perform any parameter tuning or feature selection. in an intrinsic evaluation, we compare whether there are significant differences between the classifier's performance and a majority baseline in terms of average un-weighted precision, recall and f1, using paired sample t-tests over 10folds of cross-validation. in an extrinsic evaluation, we repeat the pearson correlation study in section 5 for the predicted code, 'relevant' evidence. intrinsic evaluation. extrinsic evaluation. ## discussion in our corpus, students revised only about half of the sentences from draft1 to draft2. among the revisions, only a small proportion focused on evidence or reasoning, despite feedback targeting these argument elements exclusively. this resonates with writing research (though not in the context of awe) showing that students often struggle to revise @xcite , and that novice writers -like our 5th-and 6th-graders -tend to focus on local word-and sentence level problems rather than content or structure @xcite . when novices do revise, their efforts frequently result in no improvement or improvement only in surface features @xcite . we knew of no revision schemes that assessed the extent to which evidence use and reasoningrelated revisions aligned with desirable features of argumentative writing (i.e., showed responsiveness to system feedback to use more relevant evidence, give more specific details, or provide explanations connecting evidence to the claim); hence, we developed the rer scheme. the scheme -along with the reliability we established and the positive correlations we demonstrated between its sentence-level application and a holistic assessment of essay improvement in line with provided feedback -is an important contribution because the codes are keyed to critical features of the argument writing genre. therefore, it is more useful than existing schemes that focus on general revision purposes (surface vs. content) or operations (addition, deletion, modification) for assessing the quality of students' revisions. this assessment capability is important for at least two related reasons. first, an awe system is arguably only effective if it helps to improve writing in line with any feedback provided. it is easier to attribute other types of revisions or improvements to the general opportunity to revisit the essay than to any inputs the system provides to students. for argument writing (and our awe system), then, it is necessary, to be able to identify specific revision behaviors related to evidence use and reasoning. with the rer scheme, we were able to distinguish among revision behaviors. on the whole, predictably undesirable revisions (e.g., deleting relevant evidence) were not correlated with the 'improvement score'. second, gaining insight into how students specifically revise evidence use and reasoning can help hone the content of awe feedback so that it better supports students to make desirable revisions that impact the overall argument quality. from our coding, we learned that students make deletion or modification revisions less frequently; rather they tend to make additions, even if they do not improve the essay. we also learned that repeating existing evidence accounted for about 19 percent of the evidence-use revisions. we could refine our feedback to preempt students from making these undesirable revisions. or, once automated revision detection is implemented, we could develop a finer-grained set of feedback messages to provide students to guide their second revision (i.e., production of draft 3). finally, our study takes a step towards advancing automated revision detection for awe by developing a simple machine learning algorithm for classifying relevant evidence. however, it is important to note that the classifier's input is currently based on the gold (i.e., human) alignments of the essay drafts and the gold revision purpose labels (e.g., evidence). an actual end-to-end system would have lower performance due to the propagation of errors from both alignment and revision purpose classification. in addition, due to the small size of our current corpus, our classification study was simplified to focus on evidence rather than both evidence and reasoning, and to focus on binary rather than 5-way classification. although our algorithm is thus limited to predicting only relevant evidence, the classifier nonetheless outperforms the baseline given little training data. ## conclusion and future work we developed the rer scheme as a step towards advancing automated revision detection capabilities of students' argument writing, which is critical to supporting students' writing development in awe systems. we demonstrated that reliable manual annotation can be achieved and that the rer scheme correlates in largely expected ways with a holistic assessment of the extent to which revisions address the feedback provided. we conclude that this scheme has promise in guiding the development of an automated revision classification tool. although the rer scheme was developed with a specific corpus and writing assignment, we believe some of the categories (e.g., reasoning linked to claim and evidence) can easily be adapted to data we have from other revision tasks. with more data, we also plan to improve the current classification method with state-of-the-art machine learning models, and extend the classification for all categories.