To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support (2023)

Abstract: Optimizing the phrasing of argumentative text is crucial in higher education and professional development. However, assessing whether and how the different claims in a text should be revised is a hard task, especially for novice writers. In this work, we explore the main challenges to identifying argumentative claims in need of specific revisions. By learning from collaborative editing behaviors in online debates, we seek to capture implicit revision patterns in order to develop approaches aimed at guiding writers in how to further improve their arguments. We systematically compare the ability of common word embedding models to capture the differences between different versions of the same text, and we analyze their impact on various types of writing issues. To deal with the noisy nature of revision-based corpora, we propose a new sampling strategy based on revision distance. Opposed to approaches from prior work, such sampling can be done without employing additional annotations and judgments. Moreover, we provide evidence that using contextual information and domain knowledge can further improve prediction results. How useful a certain type of context is, depends on the issue the claim is suffering from, though.

## introduction text revision is an essential part of professional writing and is typically a recursive process until a somehow optimal phrasing is achieved from the author's point of view. aside from proofreading and copyediting, text revision subsumes substantive and rhetorical changes not only at the lexical, syntactic, and semantic levels, but also some that may require knowledge about the topic of discussion or about conventions of the domain or genre. an optimal phrasing is especially important in argumentative writing, where it is considered a key component in academic and professional success: an argument's style directly affects its persuasive effect on the audience @xcite . ## related work foundational studies of writing specify two main revision sub-processes: evaluating (reading the produced text and detecting any problems) and editing (finding an optimal solution and executing the changes) @xcite . in this work, we focus on the former in the domain of argumentative texts. although considerable attention has been given to the computational assessment of the quality of such texts, very few works consider the effects of revision behaviors on quality. existing research largely focuses on the absolute and relative assessment of single quality dimensions, such as cogency and reasonableness @xcite . @xcite propose a unifying taxonomy that combines 15 quality dimensions. they clarify that some dimensions, such as acceptability and rhetorical effectiveness, depend on the social and cultural context of the writer and/or audience. a number of approaches exist that control for topic and stance @xcite , model the audience @xcite , or their beliefs @xcite , and similar. however, they all compare texts with different content and meaning in terms of the aspects of topics being discussed. while such comparisons help characterize good arguments, they are not geared towards identifying issues within them, let alone towards guiding writers on how to improve the quality of their arguments. the only works we are aware of that study revisions of argumentative texts are those of @xcite , @xcite , and @xcite . the first two suggest approaches that enable automatic assessments of whether a revision can be considered successful, that is, it improves the quality of the argumentative essay or claim. the third extends the corpus of @xcite to complement 86 essays with more fine-grained annotations, enabling the distinction of content-level from surface-level revision changes at different levels of granularity. all these approaches to characterizing the type of revision and its quality require two versions as input. in contrast, we seek to identify whether an argumentative text needs to be revised at all and, if so, what type of improvement should be undertaken. in such framing, the solutions to the tasks can also be used to support argument generation approaches, for example, by helping identify weak arguments for counter-argument generation @xcite , as well as automated revision approaches, for example, by providing required revision types or weak points as prompts @xcite . due to the lack of corpora where revisions are performed by the authors of texts themselves, researchers utilize collaborative online platforms. such platforms encourage users to revise and improve existing content, such as encyclopedias @xcite , how-to instructions (anthonio et al., 2020), q&a sites @xcite , and debate portals @xcite . studies have explored ways to automate content regulation, namely text simplification @xcite , detection of grammar errors @xcite , lack of citations @xcite , biased language @xcite , and vagueness @xcite . @xcite consider a task similar to ours -detecting sentences in need of revision in the domain of instructional texts -their findings do not fully transfer to argumentative texts, as different domains have different goals, different notions of quality, and, subsequently, different revision types performed. revision histories of peer-reviewed content help alleviate the shortcomings typical for self-revisions, where a writer may fail to improve a text for lack of expertise or skills @xcite . yet, they also introduce new challenges stemming from sociocultural aspects, such as opinion bias @xcite and exposure bias @xcite . approaches to filtering out true positive and negative samples have been suggested to tackle such issues. these include community quality assessments, where high quality content is determined based on editor or user ratings and upvotes @xcite , timestamp-based heuristics, where high-quality labels are assigned to content that has not been revised for a certain time period @xcite , and complementary crowdsourced annotation @xcite . however, all this requires domain-specific information which may not be available in general. in our experiments, we analyze the potential of sampling data solely based on revision characteristics, namely revision distance (the number of revisions between a certain claim version and its final version). moreover, studies have shown that writing expertise is domain-dependent, revealing commonalities within various professional and academic writing domains @xcite . although certain quality aspects can be defined and evaluated using explicit rules, norms, and guidelines typical for a domain, not all quality aspects can be encoded in such rules. this raises the need to develop approaches capable of capturing implicit revision behaviors and incorporating additional context relevant to the decision-making process @xcite . below, we outline the main challenges stemming from the noisy and biased nature of revision-based corpora as well as from the context dependence of certain argument quality aspects. we then establish potential data filtering and sampling strategies targeting said issues. ## tasks and challenges revision-based data provides many opportunities; yet, it also comes with several challenges that arise at different stages of the experiment design process. in the following, we define the tasks we deal with in this paper, summarize the main challenges, and outline our approaches to these challenges. ## data in our experiments, we use claimrev (skitalinskaya et al., 2021): a corpus of over 124,312 claim revision histories, collected from the online debating platform kialo, 2 which uses the structure of dialectical reasoning. each debate has a main thesis, which is elaborated through pro and con claims, which allows to consider each comment as a self-contained, relevant argument. each revision history is a set of claims in chronological order, where each claim represents a revised version of the preceding claim meant to improve a claim's quality, which holds in 93% of all cases according to an annotation study @xcite . we extend the corpus by extracting 86,482 unrevised claims from the same set of debates as in claimrev, which have been introduced before the reported date of data collection (june 26, 2020). since claims that have been introduced shortly before this date are still likely to receive revisions, we additionally filter out claims that have undergone a revision within six months after the initial data collection @xcite . we remove all revision histories, where claim versions have been reverted to exclude potential cases of edit wars. our final corpus is formed by 410,204 claims with 207,986 instances representing optimally phrased claims (positive class) and 202,218 instances requiring revisions (negative class). all claims in need of further refinement are also provided with labels indicating the specific type of improvement the claim could benefit from. in this work, we limit ourselves to the three most common types, covering 95% of all labels revisions in the claimrev dataset: clarification, typo/grammar correction, and adding/correcting links. specifically, clarification means to adjust/rephrase a claim to make it more clear, typo/grammar correction simply indicates linguistic edits, and adding/correcting links refers to the insertion or editing of evidence in the form of links that provide supporting information or external resources to the claim. statistics of the final dataset are shown in ## methods to study the two proposed tasks, we consider two experimental settings: (i) extracting claim representations by using embeddings as input to an svm @xcite , (ii) adding a classifier layer on top of pre-trained transformer models with further fine-tuning (ft). in our experiments, we consider the following approaches to generating claim representations: ## conclusion most approaches to argument quality assessment rate or compare argumentative texts that cover different aspects of a topic. while a few works studied which of two revisions of the same argumentative text is better, this does not suffice to decide whether a text actually needs revisions. in this paper, we have presented two tasks to learn when and how to improve a given argumentative claim. we have delineated the main challenges of revision-based data, covering issues related to the representativeness and reliability of data, topical bias in revision behaviors, appropriate model complexities and architectures, and the need for context when judging claims. in experiments, we have compared several methods based on their ability to capture quality differences between different versions of the same text. despite a number of limitations (discussed below), our results indicate that, in general, revision-based data can be employed effectively for the given tasks, contributing towards solutions for each of the considered challenges. specifically, our suggested sampling strategy revealed that training on claim versions with a higher revision distance between them improves the performance when detecting claims in need of improvement. moreover, we found that the impact of the available types of contextual information is not only task-dependent but also depends on the quality issue that a claim suffers from. we argue that the developed approaches can help assist automated argument analysis and guide writers in improving their argumentative texts. with our work, we seek to encourage further research on improving writing support not only in debate communities but in educational settings as well.