Classifying Argumentative Relations Using Logical Mechanisms and Argumentation Schemes (2021)

Abstract: While argument mining has achieved significant success in classifying argumentative relations between statements (support, attack, and neutral), we have a limited computational understanding of logical mechanisms that constitute those relations. Most recent studies rely on black-box models, which are not as linguistically insightful as desired. On the other hand, earlier studies use rather simple lexical features, missing logical relations between statements. To overcome these limitations, our work classifies argumentative relations based on four logical and theory-informed mechanisms between two statements, namely, (i) factual consistency, (ii) sentiment coherence, (iii) causal relation, and (iv) normative relation. We demonstrate that our operationalization of these logical mechanisms classifies argumentative relations without directly training on data labeled with the relations, significantly better than several unsupervised baselines. We further demonstrate that these mechanisms also improve supervised classifiers through representation learning.

## introduction there have been great advances in argument mining-classifying the argumentative relation between statements as support, attack, or neutral. recent research has focused on training complex neural networks on large labeled data. however, the behavior of such models remains obscure, and recent studies found evidence that those models may rely on spurious statistics of training data @xcite and superficial cues irrelevant to the meaning of statements, such as discourse markers @xcite . hence, in this work, we turn to an interpretable method to investigate logical relations between statements, such as causal relations and factual contradiction. such relations have been underemphasized in earlier studies @xcite , possibly because their operationalization was unreliable then. now that computational semantics is fast developing, our work takes a first step to computationally investigate how logical mechanisms contribute to building argumentative relations between statements and to classification accuracy with and without training on labeled data. to investigate what logical mechanisms govern argumentative relations, we hypothesize that governing mechanisms should be able to classify the relations without directly training on relationlabeled data. thus, we first compile a set of rules specifying logical and theory-informed mechanisms that signal the support and attack relations ( §3). the rules are grouped into four mechanisms: factual consistency, sentiment coherence, causal relation, and normative relation. these rules are combined via probabilistic soft logic (psl) @xcite to estimate the optimal argumentative relations between statements. we operationalize each mechanism by training semantic modules on public datasets so that the modules reflect real-world knowledge necessary for reasoning ( §4). for normative relation, we build a necessary dataset via rich annotation of the normative argumentation schemes argument from consequences and practical reasoning @xcite , by developing a novel and reliable annotation protocol ( §5). our evaluation is based on arguments from kialo.com and debatepedia.org. we first demonstrate that the four logical mechanisms explain the argumentative relations between statements effectively. psl with our operationalization of the mechanisms can classify the relations without direct training on relation-labeled data, outperforming several unsupervised baselines ( §7). we analyze the contribution and pitfalls of individual mechanisms in detail. next, to examine whether the mechanisms can further inform supervised models, we present a method to learn vector representations of arguments that are ''cognizant of'' the logical mechanisms ( §8). this method outperforms several supervised models trained without concerning the mechanisms, as well as models that incorporate the mechanisms in different ways. we illustrate how it makes a connection between logical mechanisms and argumentative relations. our contributions are: ## related work there has been active research in nlp to understand different mechanisms of argumentation computationally. argumentative relations have been found to be associated with various statistics, such as discourse markers @xcite , sentiment @xcite , and use of negating words @xcite . further, as framing plays an important role in debates @xcite , different stances for a topic emphasize different points, resulting in strong thematic correlations @xcite . such thematic associations have been exploited in stance detection and dis/agreement classification. stance detection @xcite @xcite aims to classify a statement as pro or con with respect to a topic, while dis/agreement classification @xcite @xcite aims to decide whether two statements are from the same or opposite stance(s) for a given topic. topics are usually discrete, and models often learn thematic correlations between a topic and a stance @xcite . our work is slightly different as we classify the direct support or attack relation between two natural statements. the aforementioned correlations, however, are byproducts rather than core mechanisms of argumentative relations. in order to decide whether a statement supports or attacks another, we cannot ignore the logical relation between them. textual entailment was found to inform argumentative relations @xcite and used to detect arguments @xcite . similarly, there is evidence that the opinions of two statements toward the same concept constitute their argumentative relations @xcite . causality between events also received attention, and causality graph construction was proposed for argument analysis @xcite . additionally, in argumentation theory, walton's argumentation schemes @xcite specify common reasoning patterns people use to form an argument. this motivates our work to investigate logical mechanisms in four categories: factual consistency, sentiment coherence, causal relation, and normative relation. ## rules we first compile rules that specify evidence for the support and attack relations between claim c and statement s ( ## modules in this section, we discuss individual modules for operationalizing the psl rules. for each module, we fine-tune the pretrained uncased bert-base @xcite . we use the transformers library v3.3.0 @xcite for high reproducibility and low development costs. but any other models could be used instead. each dataset used is randomly split with a ratio of 9:1 for training and test. cross-entropy and adam are used for optimization. to address the imbalance of classes and datasets, the loss for each training instance is scaled by a weight inversely proportional to the number of its class and dataset. ## annotation of normative argumentation schemes in this section, we discuss our annotation of the argumentation schemes argument from consequences and practical reasoning (figure 1 ). the resulting annotations are used to train the modules in §4.4 that compute the probability terms in r10-r13. for each pair of normative claim c and statement s, we annotate the following information: (1a) whether c advocates for or opposes its norm target, and (1b) the norm target t (figure 1 task 1); (2a) whether s uses a norm, consequence, or property for justification, and (2b) the justification j (figure 1 task 2); (3a) whether j's focus is on advocating for t or opposing t , and (3b) whether j is positive or negative (figure 1 task 3). 6 our annotation schema is richer than existing ones @xcite . due to the increased complexity, however, our annotation is split into three pipelined tasks. for this annotation, we randomly sampled 1,000 arguments from kialo whose claims are normative (see §6 and ## experiment 1. probabilistic soft logic the goal here is to see how well the logical mechanisms alone can explain argumentative relations. ## conclusion we examined four types of logical and theoryinformed mechanisms in argumentative relations: factual consistency, sentiment coherence, causal relation, and normative relation. to operationalize normative relation, we built rich annotation schema and data for the argumentation schemes argument from consequences and practical reasoning, too. evaluation on arguments from kialo and debatepedia revealed the importance of these mechanisms in argumentation, especially normative relation and sentiment coherence. their utility was further verified in a supervised setting via our representation learning method. our model learns argument representations that make strong correlations between logical relations and argumentative relations in intuitive ways. textual entailment was found to be particularly helpful in the supervised setting. some promising future directions are to probe fine-tuned bert to see if it naturally learns logical mechanisms and to improve psl with more rules.