Explainable Unsupervised Argument Similarity Rating with A bstract M eaning R epresentation and Conclusion Generation (2021)

Abstract: When assessing the similarity of arguments, researchers typically use approaches that do not provide interpretable evidence or justifications for their ratings. Hence, the features that determine argument similarity remain elusive. We address this issue by introducing novel argument similarity metrics that aim at high performance and explainability. We show that Abstract Meaning Representation (AMR) graphs can be useful for representing arguments, and that novel AMR graph metrics can offer explanations for argument similarity ratings. We start from the hypothesis that similar premises often lead to similar conclusions —and extend an approach for AMR-based argument similarity rating by estimating, in addition, the similarity of conclusions that we automatically infer from the arguments used as premises. We show that AMR similarity metrics make argument similarity judgements more interpretable and may even support argument quality judgements . Our approach provides significant performance improvements over strong baselines in a fully unsupervised setting. Finally, we make first steps to address the problem of reference-less evaluation of argumentative conclusion generations.

## introduction rating the similarity of arguments @xcite is a core task in argument mining and argument search @xcite @xcite . argument similarity ratings are also needed for (case-based) argument retrieval @xcite , data exploration via argument clustering, and even automated debaters @xcite : to counter an opponent's argument, one may retrieve an argument similar to theirs, but of opposite stance to the topic @xcite . typically, argument similarity ratings are computed over 'bag-of-word' argument representations, or else over argument representations inferred with language models such as bert @xcite or infersent @xcite . two key advantages of such approaches are due to their unsupervised setup: first, unsupervised methods do not rely on human annotations, which are expensive and can be subject to noise and biases. second, it has been shown for previous supervised methods that they have learned less about argumentation tasks than had been assumed, by exploiting spurious clues and artifacts from manually created data @xcite . this has led to a recent interest in solving argumentation tasks in an unsupervised manner, e.g., by logical reasoning @xcite . in this paper we will highlight that previous methods for rating argument similarity suffer from a common flaw: beyond shallow statistics (word matches in bag-of-word models, or word similarities in distributional space), they do not provide any rationale for their predictions, and the prediction process is in general not transparent. therefore, we know only little about the following question: in this work, we undertake a first attempt at answering this question, by testing two hypotheses: i) representing arguments with abstract meaning representations (amrs) and using amr graph metrics improves argument similarity rating and provides explanatory information. ii) extending arguments with inferred conclusions can improve argument similarity rating. in the following §2 we discuss related work. §3 introduces our two key hypotheses, and §4 presents our argument similarity rating model and its implementation. in §5 we compare our model against strong baselines from prior work. in §6 we conduct several analyses to show how our approach can contribute to a better understanding of arguments, their conclusions and argument similarity ratings: we i) assess predictors of human argument similarity ratings to investigate the criteria that correlate with human ratings of argument similarity; ii) discuss potential advantages of using amr for graph-based argumentation tasks in a concrete example, and iii) investigate how interpretable argument similarity computation can help assess the quality and usefulness of conclusions drawn from arguments in a reference-less conclusion evaluation setup. foot_0 2 related work argument similarity and search assessing argument similarity is a key task in argument mining @xcite and can enhance argument search @xcite @xcite @xcite ). yet, while delivering solid performance on benchmarks, current methods fail to provide any deeper rationale for their predictions. it is thus not clear whether and to what extent spurious clues or other artifacts may influence the similarity decision @xcite . in this paper, we aim at alleviating these issues by i) representing arguments with abstract meaning representation @xcite and conducting similarity assessment using well-defined graph metrics that provide explanatory amr structure alignments; and ii) by investigating to what extent argument similarity can be projected to inferred conclusions. ## hypotheses we base our models for explanatory argument similarity assessment on two hypotheses. hypothesis i: abstract meaning representation @xcite of arguments supports explainable argument similarity assessment amrs are directed, rooted and acyclic graphs that aim at capturing a sentence's meaning in a machine-readable format. edges are labeled with semantic relation types (e.g., negation, cause, etc.) and vertices denote either variables or concepts (variables are instances of concepts and allow us to capture coreferences) hence, the amr formalism captures various semantic phenomena that can play a role when assessing argument similarity. e.g., besides the obviously useful aspect of negation, amr captures semantic roles and predicate senses @xcite . while it is clear that similar arguments tend to involve similar predicates and predicate senses, semantic structure and role assignment may also play a role. for instance, the claims: consumption of alcohol leads to depression vs. depression leads to consumption of alcohol are clearly distinct, while sharing the same concepts. other amr facets may also be useful. e.g., amr captures coreferences and resolving them in different ways can induce significant meaning differences, finally, amr includes key semantic relations (location, cause, possession, etc.) that are often implicit or underspecified in language, hence their explicit representation in amr provides a rich basis for assessing arguments. arguments represented with amr can be compared with amr graph metrics @xcite @xcite that also induce an explicit alignment between two argument graphs. hypothesis ii: similar arguments lead to similar conclusions we hypothesize that a key feature of similar arguments is that they invite for similar conclusions. analogously, dissimilar arguments tend to lead to differing conclusions. consider the following two arguments: i) cannabis can have negative effects on brain development of teens. ii) smoking cannabis is harmful for the lungs. the arguments are dissimilar, even though they share the same (negative) stance and argue from a similar perspective (health). this dissimilarity is also reflected in the conclusions that can be inferred from them: from i) we can infer that, i.a., cannabis consumption should be strictly controlled for age or cannabis can have a negative impact on the brain-while from ii) we could infer that cannabis, if consumed, should not be smoked or cannabis smokers should get their lungs checked. as a complementary example, the similarity of two arguments may be reinforced by the similarity of their inferred conclusions, as shown below: i) fracking can contaminate water and water wells and suck towns dry. ii) as a water-poor state, fracking and its toxic wastewater presents a serious danger to our communities and ecosystems. arguments i) and ii) are rated as similar, presumably because they point at detrimental ramifications of fracking related to water issues. this similarity is likely to be reflected in conclusions drawn from them, such as: i) fracking can lead to water issues or ii) fracking poses dangers for water-poor states. ## argument similarity via amr metrics according hyp i, we represent arguments with amr graphs and rate their similarity with amr metrics. to test hyp ii we infer conclusions from arguments with language models and compute similarity on arguments extended with their conclusion. ## discussion explanation dimensions our argument similarity rating approach may provide explanations in various dimensions. i) first and foremost, the explicit alignment and similarity computation based on amr and amr graph metrics, by relating similar concepts between arguments and their conclusions, provides insight into which components of two argument amrs relate to each other, with individual alignment scores, and to what extent they congtribute to the overall score. especially in light of recent observations showing supervised models to be prone to superficial cues in data sets @xcite @xcite , this property is desirable. ii) we apply the fine-grained amr decomposition of @xcite in terms of semantic phenomena, such as negation or semantic roles. this can further illuminate in which ways an argument pair is similar/dissimilar. iii) by taking into account the similarity of automatically inferred conclusions, the similarity computed for premises may be re-adjusted in case the similarity of the inferred conclusions strongly differs. on a related note, the amr similarity statistics enabled us to gain some first indications of what could be considered a good conclusion (without even matching against a reference): e.g., our qualitative evaluations indicate that good conclusions tend to be neither very similar, nor very dissimilar to the premise. this seems plausible, since (too) high similarity may indicate a mere summary (reducing novelty), while (too) low similarity may indicate a lack of coherence (reducing validity). ## conclusion in this paper, we investigated two hypotheses: i) amr meaning representation and graph metrics help in assessing argument similarity, ii) automatically inferred conclusions can aid or reinforce the similarity assessment of arguments. we find solid evidence for the first hypothesis, especially when slightly adapting amr metrics to focus more on concrete concepts found in arguments. we find weak evidence that supports the second hypothesis, i.e., metrics improve consistently, but by small margins, when they are allowed to additionally consider the amrs of automatically inferred conclusions. we believe, however, that more substantial gains may be obtained in future work, by improving conclusion generation models such that they produce content that is both valid and novel. finally, we have made first steps towards a reference-less metric for assessing novelty and justification of generated conclusions.