Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale (2021)

Abstract: Assessing the quality of arguments and of the claims the arguments are composed of has become a key task in computational argumentation. However, even if different claims share the same stance on the same topic, their assessment depends on the prior perception and weighting of the different aspects of the topic being discussed. This renders it difficult to learn topic-independent quality indicators. In this paper, we study claim quality assessment irrespective of discussed aspects by comparing different revisions of the same claim. We compile a large-scale corpus with over 377k claim revision pairs of various types from kialo.com, covering diverse topics from politics, ethics, entertainment, and others. We then propose two tasks: (a) assessing which claim of a revision pair is better, and (b) ranking all versions of a claim by quality. Our first experiments with embedding-based logistic regression and transformer-based neural networks show promising results, suggesting that learned indicators generalize well across topics. In a detailed error analysis, we give insights into what quality dimensions of claims can be assessed reliably. We provide the data and scripts needed to reproduce all results.

## introduction assessing argument quality is as important as it is questionable in nature. on the one hand, identifying the good and the bad claims and reasons for arguing on a given topic is key to convincingly support or attack a stance in debating technologies @xcite , argument search @xcite , and similar. on the other hand, argument quality can be considered on different granularity levels and from diverse perspectives, many of which are inherently subjective @xcite ; they depend on the prior beliefs and stance on a topic as well as on the personal weighting of different aspects of the topic @xcite . existing research largely ignores this limitation, by focusing on learning to predict argument quality based on subjective assessments of human annotators (see section 2 for examples). in contrast, @xcite control for topic and stance to compare the convincingness of arguments. @xcite abstract from an argument's text, assessing its relevance only structurally. @xcite and el @xcite focus on personality-specific and ideology-specific quality perception, respectively, whereas @xcite asked annotators to disregard their own stance in judging length-restricted arguments. however, none of these approaches controls for the concrete aspects of a topic that the arguments claim and reason about. this renders it difficult to learn what makes an argument and its building blocks good or bad in general. in this paper, we study quality in argumentation irrespective of the discussed topics, aspects, and stances by assessing different revisions of the basic building blocks of arguments, i.e., claims. such revisions are found in large quantities on online debate platforms such as kialo.com, where users post claims, other users suggest revisions to improve claim quality (in terms of clarity, grammaticality, grounding, etc.), and moderators approve or disapprove them. by comparing the quality of different revisions of the same instance, we argue that we can learn general quality characteristics of argumentative text and, to a wide extent, abstract from prior perceptions and weightings. to address the proposed problem, we present a new large-scale corpus, consisting of 124k unique claims from kialo.com spanning a diverse range of topics related to politics, ethics, and several others (section 3). using distant supervision, we derive a total number of 377k claim revision pairs from the ## related work in the recent years, there has been an increase of research on the quality of arguments and the claims and reasoning they are composed of. @xcite describe argumentation quality as a multidimensional concept that can be considered from a logical, rhetorical, and dialectical perspectives. to achieve a common understanding, the authors suggest a unified framework with 15 quality dimensions, which together give a holistic quality evaluation at a certain abstraction level. they point out, that several dimensions may be perceived differently depending on the target audience. in recent follow-up work, wachsmuth and werner (2020) examined how well each dimension can be assessed only based on plain text only. most existing quality assessment approaches target a single dimension. on mixed-topic student essays, @xcite learn to score the clarity of an argument's thesis, @xcite do the same for argument strength, and @xcite classify whether an argument's premises sufficiently support its conclusion. all these are trained on pointwise quality annotations in the form of scores or binary judgments. @xcite provide a corpus with crowdsourced quality annotations for 30,497 arguments, the largest to date for pointwise argument quality. the authors studied how their annotations correlate with the 15 dimensions from the framework of @xcite , finding that only global relevance and effectiveness are captured. similarly, @xcite built a new corpus based on the framework to then exploit interactions between the dimensions in a neural approach. we present a small related annotation study for our dataset below. however, we follow @xcite in that we cast argument quality assessment as a relation classification problem, where the goal is to identify the better among a pair of instances. in particular, @xcite created a dataset with argument convincingness pairs on 32 topics. to mitigate annotator bias, the arguments in a pair always have the same stance on the same topic. the more convincing argument is then predicted using a feature-rich svm and a simple bidirectional lstm. other approaches to the same task map passage representations to real-valued scores using gaussian process preference learning @xcite or represent arguments by the sum of their token embeddings @xcite , later extended by a feed forward neural network @xcite . recently, @xcite employed a siamese neural network to rank arguments by the convincingness of evidence. in our experiments below, we take on some of these ideas, but also explore the impact of transformer-based methods such as bert @xcite , which have been shown to predict argument quality well @xcite . @xcite observed that longer arguments tend to be judged better in existing corpora, a phenomenon we will also check for below. @xcite prevent such bias in their corpora for both pointwise and pairwise quality, by restricting the length of arguments to 8-36 words. the authors define quality as the level of preference for an argument over other arguments with the same stance, asking annotators to disregard their own stance. for a more objective assessment of argument relevance, @xcite abstract from content, ranking arguments only based on structural relations, but they employ majority human assessments for evaluation. @xcite take a different approach, including knowledge about the personality of the reader into the assessment, and el @xcite study the impact of argumentative texts on people depending on their political ideology. as can be seen, several approaches aim to control for length, stance, audience, or similar. however, all of them still compare argumentative texts with different content and meaning in terms of the aspects of topics being discussed. in this work, we assess quality based on different revisions of the same text. in this setting, the quality is primarily focused on how a text is formulated, which will help to better understand what influences argument quality in general, irrespective of the topic. to be able to do so, we refer to online debate portals. debate portals give users the opportunity to discuss their views on a wide range of topics. existing research has used the rich argumentative content and structure of different portals for argument mining, including createdebate.com @xcite , idebate.org @xcite , and others. also, large-scale debate portal datasets form the basis of applications such as argument search engines @xcite . unlike these works, we exploit debate portals for studying quality. @xcite predicted argument persuasiveness in the discussion forum change-myview from ground-truth labels given by opinion posters, and @xcite used user upvotes and downvotes for the same purpose. here, we resort to kialo.com, where users cannot only state argumentative claims and vote on the impact of claims submitted by others, but they can also help improve claims by suggesting revisions, which are approved or disapproved by moderators. while @xcite assessed quality based on the impact value of claims from kialo.com, we derive information on quality from the revision history of claims. the only work we are aware of that analyzes revision quality of argumentative texts is the study of @xcite . from the corpus of @xcite containing 60 student essays with three draft versions each, 940 sentence writing revision pairs were annotated for whether the revision improves essay quality or not. the authors then trained a random forest classifier for automatic revision quality classification. in contrast, instead of sentences, we shift our focus to claims. moreover, our dataset is orders of magnitude larger and includes notably longer revision chains, which enables deeper analyses and more reliable prediction of revision quality using data-intensive methods. ## data here, we present our corpus created based on claim revision histories collected from kialo.com. ## approaches to study the two proposed tasks, claim quality classification and claim quality ranking, on the given corpus, we consider the following approaches. ## experiments and discussion we now present empirical experiments with the approaches from section 4. the goal is to evaluate how hard it is to compare and rank the claim revisions in our corpus from section 3 by quality. ## conclusion and future work in this paper, we have proposed a new way of assessing quality in argumentation by considering different revisions of the same claim. this allows us to focus on characteristics of quality regardless of the discussed topics, aspects, and stances in argumentation. we provide a new corpus of web claims, which is the first large-scale corpus to target quality assessment and revision processes on a claim level. we have carried out initial experiments on this corpus using traditional and transformer-based models, yielding promising results but also pointing to limitations. in a detailed analysis we have studied different kinds of claim revisions and provided insights into the aspects of a claim that influence the users' perception of quality. such insights could help improve writing support in educational settings, or identify the best claims for debating technologies and argument search. we seek to encourage further research on how to help online debate platforms automate the process of quality control and design automatic quality assessment systems. such systems can be used to indicate if the suggested revisions increase the quality of an argument or recommend the type of revision needed. we leave it for future work to investigate whether the learned concepts of quality are transferable to content from other collaborative online platforms (such as idebate.org or wikipedia), or to data from other domains, such as student essays and forum discussions.