Uncovering Implicit Inferences for Improved Relational Argument Mining (2023)

Abstract: Argument mining seeks to extract arguments and their structure from unstructured texts. Identifying relations between arguments (such as attack, support, and neutral) is a challenging task because two arguments may be related to each other via implicit inferences. This task often requires external commonsense knowledge to discover how one argument relates to another. State-of-the-art methods, however, rely on pre-defined knowledge graphs, and thus might not cover target argument pairs well. We introduce a new generative neuro-symbolic approach to finding inference chains that connect the argument pairs by making use of the Commonsense Transformer (COMET). We evaluate our approach on three datasets for both the two-label (attack/support) and three-label (attack/support/neutral) tasks. Our approach significantly outperforms the state-of-the-art, by 2-5% in F1 score, on all three datasets.

## introduction in argumentation, such as in a debate, it is important to understand the key arguments put forward and how the arguments relate to each other based on a specific context. ideally, we would like to empower systems to extract arguments and the relations between arguments automatically. such systems could be used to aggregate opinions @xcite , participate in debates @xcite , or they could assist humans in making informed decisions by considering different points of view. @xcite defines an argument as a claim backed by one or more grounds (or premises) based on a warrant (i.e., the inference link between the grounds and the claim). for example, consider the following two sentences: drunk driving hurts innocent people. therefore, drunk driving is wrong. based on toulmin's model, we have the claim drunk driving is wrong, supported on the grounds that drunk driving hurts innocent people. in what follows, we will call the claim and grounds argumentative units (aus). to justify the claim, the author relies on the reader's ability to uncover the implicit warrant that hurting innocent people is wrong which connects the grounds to the claim. while toulmin's warrants model only arguments in which the units are connected by support relations, we extend the definition to also allow for attacks and neutral relations. we do this to enable us to apply the notion of warrants to the relational argument mining task, as proposed by @xcite , to better identify attack/support/neutral relations between pairs of aus. finding the correct warrant is a challenging task that requires deep reasoning, which pre-trained language models struggle with @xcite . however, by iteratively generating knowledge, we can chain together inferences to get a series of causes and effects that connect two aus together similar to chain-of-thought reasoning @xcite . we propose the use of the commonsense transformer (comet) @xcite which is equipped with social commonsense to generate warrants as a series of inferences. the knowledge contained in comet is qualitatively different from other knowledge sources in that it contains normative commonsense. this depends upon the culture and beliefs of various groups compared with static commonsense knowledge graphs (kgs), such as conceptnet @xcite , which focus on globally accepted knowledge. for this reason, we suggest that comet is better able to identify the reasoning behind human arguments and that a generative approach based on comet is better suited to finding warrants compared to a static knowledge graph as proposed by @xcite . section 4 introduces our approach to test this hypothesis, where we articulate an algorithm, argcon (section 4.2), to generate warrants between a pair of argumentative units and use these warrants as additional input to the classifier of argument relations to evaluate whether this improves the model's ability to distinguish between attack, support or neutral relations. we then compare the use of three external knowledge sources: conceptnet + the commonsense knowledge extracted by paul et al., the knowledge generated by argcon, to obtain novel commonsense inferences (section 4.2), and argcon + link which enriches the generated knowledge with additional relations via a link prediction model (section 4.3). in section 5, we compare our work to @xcite , which is the only other work we are aware of to address the task of relational argument mining by uncovering commonsense relations. we show that our method for generating warrants with comet outperforms their method with a 1-2% increase in f1 scores on two datasets. we further evaluate our model on a three-class dataset, which includes an additional neutral relation, and witness a 5% improvement of the f1 score over a roberta baseline. hence, we set a new benchmark for the three-class task in relational argument mining. we share our findings in section 6 and conclude with our future directions in section 7. ## background in this paper, we investigate the effectiveness of comet @xcite for generating implicit warrants that link arguments together. comet is a generative model which has been finetuned on a human-authored commonsense knowledge graph atomic @xcite . by training on atomic, comet extends the knowledge captured by the knowledge graph to new domains and is able to synthesize new commonsense knowledge based on knowledge captured during pre-training. the model is able to generate knowledge along multiple relations-a few examples are shown in some successful applications of comet include the generation of potential outcomes of events for abductive reasoning @xcite , question answering @xcite ) and text generation @xcite . closely related to our work, @xcite shows that comet can be used to generate logically sound premises for incomplete arguments. while previous works have considered using comet for single-hop inference, we found that for our task of relational argument mining, often multiple inference steps are needed to get from the first argumentative unit to the second; we address this challenge in section 4.2. ## related work various approaches have been proposed to infer argumentative structure from unstructured text. these include long-short term memory (lstm) models @xcite , pre-trained transformers @xcite and logic programming @xcite . pretrained transformers, in particular, have been shown to perform exceptionally well on this task with no additional feature engineering @xcite ; this suggests that the introduction of external knowledge encoded within the transformers due to pre-training on large corpora is necessary to make significant progress on this task. of these works, only @xcite makes use of external knowledge to identify the stance of arguments towards a given topic. this differs from our task in that we wish to identify the relationship between pairs of aus and so have the added challenge of finding paths that link the aus together. commonsense knowledge has proven repeatedly to aid in tasks such as natural language inference @xcite ) and question answering @xcite . the majority of these works focus mainly on deriving knowledge from concept-net @xcite ; however, a key limitation of knowledge graphs @xcite is their inherent incompleteness. recent trends in argument mining have shown a move towards incorporating external knowledge for various argument-mining tasks. two works explore the use of automatically constructed knowledge graphs extracted from wikidata and google search, which, however, fail to achieve better results than bert with no additional knowledge @xcite . on the other hand, a work using lstm-based models for knowledge-enhanced argument mining with conceptnet + , shows that the introduction of knowledge indeed improves performance upon their baselines @xcite . one of the key limitations of existing works is their use of static knowledge graphs which are unable to handle arbitrary pairs of aus due to their limited coverage. to address this issue, we use comet to generate conmmonsense inferences on-the-fly to connect aus and therefore generalize to new domains. ## our approach in this section, we describe the design of our model as well as our procedures for obtaining knowledge on the fly with comet and enriching the knowledge with additional links. ## evaluation in this section, we describe the datasets we have chosen, our choices of parameters for knowledge extraction, our training setup, and the results of our experiments on three datasets. ## analysis and discussion in order to interpret the results better, we analyze the inference chains generated by argcon, and we also investigate the performance of the models across different topics. in ## conclusion in this paper, we introduced a new method, argcon, to investigate the role of the commonsense knowledge encoded in comet for the task of relational argument mining. in particular, argcon can reveal implicit commonsense reasoning chains (i.e., warrants) and these chains provide useful information for classifying argument relations such as attack, support, and neutral. our experiments on three different datasets show that the chains generated by this method outperform existing approaches. our analysis shows that the performance of our models varies given the topic under discussion and in some cases argcon creates noisy connections as a result of injected knowledge. we also observe that our model performs well in its ability to handle complex causal relationships and reason about social norms and biases. in this work, we only consider pairs of aus without taking into account any additional context. however, incorporating the text surrounding the pair of arguments is often necessary to be able to determine their relationship. in future work, we would like to incorporate such contextual information into our approach and explore the impact this has on the final performance. another direction that we want to explore is considering other relations of comet knowledge such as intents, needs, and wants to investigate whether this kind of social commonsense improves classification accuracy. another avenue of investigation we would like to pursue is the role that external knowledge plays in increasing explainability and improving the diagnosis of end-to-end model predictions. beyond argument mining, we believe that argcon is useful for other inference tasks such as recognizing textual entailment and question answering, we hope to experiment with other datasets to test this.