Intrinsic Quality Assessment of Arguments (2020)

Abstract: Several quality dimensions of natural language arguments have been investigated. Some are likely to be reflected in linguistic features (e.g., an argument’s arrangement), whereas others depend on context (e.g., relevance) or topic knowledge (e.g., acceptability). In this paper, we study the intrinsic computational assessment of 15 dimensions, i.e., only learning from an argument’s text. In systematic experiments with eight feature types on an existing corpus, we observe moderate but significant learning success for most dimensions. Rhetorical quality seems hardest to assess, and subjectivity features turn out strong, although length bias in the corpus impedes full validity. We also find that human assessors differ more clearly to each other than to our approach.

## introduction good arguments help to persuade people, to compromise, or to at least understand each other better. what quality dimension is meant by good depends on the setting, though @xcite . several dimensions may be assessed computationally, as we exemplify for an argument in favor of advancing the common good, taken from @xcite : the argument is well-organized @xcite , its premises are certainly largely acceptable @xcite and relevant to the topic @xcite . whether they suffice to draw the conclusion @xcite is another question, let alone how convincing the argument is @xcite . some dimensions may be reflected in linguistic features of an argument's text. others depend on context, require topic or background knowledge, or are inherently subjective. in this paper, we benchmark what quality dimensions of an argument can be assessed intrinsically, i.e., when analyzing the text of an argument only. given the corpus of @xcite with 304 english debate portal arguments on 16 topics scored for 15 logical, rhetorical, and dialectical quality dimensions by three experts, we carry out systematic leave-one-topic-out cross-validation experiments. in particular, we learn standard supervised score regression on various text features; from content and distributional semantics, to style, structure, and length, to text quality, evidence, and subjectivity. for 11 dimensions, we observe moderate but significant prediction gains of the features over a mean baseline. following intuition, rhetorical quality related to credibility and emotions seem hardest to assess. features capturing subjectivity (e.g., sentiment and pronoun usage) turn out particularly effective. even better performs the length feature, though, revealing bias in the corpus and matching previous findings on other corpora @xcite . follow-up experiments indicate that the experts strongly differ in their assessability, and that some beat our features only slightly in assessing quality. altogether, an intrinsic computational assessment of argument quality seems useful but not enough alone. ## related work soon after the rise of argument mining, argument quality assessment has come up as a task @xcite , due to its importance for applications such as project debater @xcite . it rests on extensive theoretical discussions about what good arguments @xcite and bad arguments are @xcite , and how to argue reasonably @xcite . several corpora and approaches were proposed for specific argument quality dimensions, first related to essay scoring @xcite , some of which modeling arguments explicitly @xcite . later approaches targeted arguments from debate portals @xcite , student essays @xcite , mixed web texts @xcite , and news editorials @xcite . we use the corpus of @xcite , as it is the only one annotated for diverse dimensions and is claimed to reflect argument quality comprehensively. in follow-up work, wachsmuth et al. (2017a) found correlations with the convincingness reasons of habernal and gurevych (2016a), and potthast et al. (2019) as well as gretz et al. (2020) have evaluated their annotations against the quality annotation scheme of the corpus. however, we do not know any previous assessment approach developed on the corpus, possibly due to its limited size (see section 3). we focus on features intrinsic to an argument's text. this complements the study of @xcite who employed external knowledge to assess convincingness. like them, we find that longer arguments tend to be judged better. @xcite limit arguments to at most 36 words, avoiding length bias but also preventing deeper reasoning. quality in their corpus reflects which argument is in doubt preferred. ultimately, such judgments remain subjective @xcite . to alleviate this, el @xcite encode the reader's ideology and personality, but such information is often not given in practice. ## data the corpus of @xcite is a subset of 320 debate portal posts from the dataset of @xcite , 20 each for 16 controversial topics. three human experts (all authors of the paper) scored all posts that they saw as arguments for the following 15 logical, rhetorical, and dialectical quality dimensions on a scale from 1 (low) to 3 (high). in line with the experiments of @xcite , we use only those 304 texts in section 5 that were seen as arguments by all three experts. ## approach this paper does not aim to propose a novel quality assessment approach, but to evaluate what features of a text help to assess which quality dimension. external knowledge is included only via lexicons and while striving to make advancements for the common good you can change the world forever. allot of people have succeded in doing so. ## experiments we now report on experiments with the features from section 4 on the corpus from section 3. in particular, we systematically study three research questions for the 15 given argument quality dimensions: q1. to what extent can each quality dimension be assessed only from an argument's text? q2. how dependent is the assessability on the subjective view of the experts? q3. how well do the considered features predict argument quality compared to humans? ## conclusion in this focused study, we have systematically benchmarked how well argument quality can be assessed computationally on an existing corpus annotated for several quality dimensions. modeling subjectiveness in terms of sentiment, pronoun usage, and similar seems useful on the debate portal arguments included in the corpus, at least for logical and dialectical dimensions. however, the limited corpus size naturally makes it hard to find more complex features that robustly predict argument quality. in addition, the correlation of quality and length in the corpus limits the generalizability of our findings. this calls for more large-scale and balanced argument quality corpora. first attempts in this direction have been made @xcite , but the comprehensive view on quality of the corpus used here has no equal so far.