Detecting Attackable Sentences in Arguments (2020)

Abstract: Finding attackable sentences in an argument is the first step toward successful refutation in argumentation. We present a first large-scale analysis of sentence attackability in online arguments. We analyze driving reasons for attacks in argumentation and identify relevant characteristics of sentences. We demonstrate that a sentence’s attackability is associated with many of these characteristics regarding the sentence’s content, proposition types, and tone, and that an external knowledge source can provide useful information about attackability. Building on these findings, we demonstrate that machine learning models can automatically detect attackable sentences in arguments, significantly better than several baselines and comparably well to laypeople.

## introduction effectively refuting an argument is an important skill in persuasion dialogue, and the first step is to find appropriate points to attack in the argument. prior work in nlp has studied argument quality @xcite and counterargument generation @xcite . but these studies mainly concern an argument's overall quality and making counterarguments toward the main claim, without investigating what parts of an argument are attackable for successful persuasion. nevertheless, attacking specific points of an argument is common and effective; in our data of online discussions, challengers who successfully change the original poster's view are 1.5 times more likely to quote specific sentences of the argument for attacks than unsuccessful challengers (figure 1 ). in this paper, we examine how to computationally ## background the strength of an argument is a long-studied topic, dating back to @xcite , who suggested three aspects of argument persuasiveness: ethos (the arguer's credibility), logos (logic), and pathos (appeal to the hearer's emotion). more recently, @xcite summarized various aspects of argument quality studied in argumentation theory and nlp, such as clarity, relevance, and arrangement. some research took empirical approaches and collected argument evaluation criteria from human evaluators @xcite . by adopting some of these aspects, computational models have been proposed to automatically evaluate argument quality in various settings, such as essays @xcite , online comments @xcite , and pairwise ranking @xcite . while these taxonomies help understand and evaluate the quality of an argument as a whole, little empirical analysis has been done in terms of what to attack in an argument to persuade the arguer. what can be attacked in an argument has been studied more in argumentation theory. particularly, @xcite present argumentation schemes and critical questions (cqs). argument schemes are reasoning types commonly used in daily argumentation. for instance, the scheme of argument from cause to effect has the conclusion "b will occur" supported by the premise "if a occurs, b will occur. in this case, a occurs". each scheme is associated with a set of cqs for judging the argument to be good or fallacious. cqs for the above scheme include "how strong is the causal generalization?" and "are there other factors that interfere with the causal effect?" unlike the general argument quality described in the previous paragraph, cqs serve as an evaluation tool that specify local attackable points in an argument. they have been adopted for grading essays @xcite and teaching argumentation skills @xcite . some of the sentence characteristics in our work are informed by argumentation schemes and cqs. nlp researchers have widely studied the effectiveness of counterarguments on persuasion @xcite @xcite @xcite and how to generate counterarguments @xcite . most of the work focuses on the characteristics of counterarguments with respect to topics and styles, without consideration of what points to attack. on the other hand, some studies aimed to model the salience of individual sentences in attacked arguments by paying different degrees of attention to sentences using attention mechanism @xcite . while their approaches helped to predict the success of persuasion, it was difficult to interpret what constitute the salience or attackability of sentences. to address this limitation, we quantify and analyze the characteristics of sentences that are attacked and lead to the arguer's view change. ## data here we describe how we collected and labeled our data. ## quantifying sentence characteristics as the first step for analyzing the characteristics of attackable sentences, we examine driving reasons for attacks and quantify relevant characteristics. ## task 1: attackability characteristics one of our goals in this paper is to analyze what characteristics of sentences are associated with a sentence's attackability. hence, in this section, we measure the effect size and statistical significance of each feature toward two labels: (i) whether a sentence is attacked or not, using the dev set of the "attacked" dataset (@xmath0,635), (ii) whether a sentence is attacked successfully or unsuccessfully, using all attacked sentences (@xmath1,417). 9 since the effects of characteristics may depend on the issue being discussed, the effect of each feature is estimated conditioned on the domain of each post using a logistic regression, and the statistical significance of the effect is assessed using the wald test. for interpretation purposes, we use odds ratio (or)-the exponent of the effect size. 10 ## task 2: attackability prediction now we examine how well computational models can detect attackable sentences in arguments. ## conclusion we studied how to detect attackable sentences in arguments for successful persuasion. using online arguments, we demonstrated that a sentence's attackability is associated with many of its characteristics regarding its content, proposition types, and tone, and that kialo provides useful information about attackability. based on these findings we demonstrated that machine learning models can automatically detect attackable sentences, comparably well to laypeople. our work contributes a new application to the growing literature on causal inference from text @xcite , in the setting of "text as a treatment". specifically, our findings in section 5 pave the way towards answering the causal question: would attacking a certain type of sentence (e.g., questions or expressions of confusion) in an argument increase the probability of persuading the opinion holder? while our findings suggest initial hypotheses about the characteristics of sentences that can be successfully attacked, establishing causality in a credible manner would require addressing confounders, such as the challenger's reputation @xcite and persuasive skill reflected in their attack @xcite . we leave this analysis to future work. our work could be improved also by including discourse properties (coherence, cohesiveness). further, argumentation structure (support relations between sentences or lack thereof) might provide useful information about each sentence's attackability. amparo elizabeth cano-basave and yulan he. 2016. a study of the impact of persuasive argumentation in political debates. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1405-1413, san diego, california. association for computational linguistics.