Beyond Recognising Entailment: Formalising Natural Language Inference from an Argumentative Perspective (2024)

Abstract: In argumentation theory, argument schemes are a characterisation of stereotypical patterns of inference. There has been little work done to develop computational approaches to identify these schemes in natural language. Moreover, advancements in recognizing textual entailment lack a standardized definition of inference, which makes it challenging to compare methods trained on different datasets and rely on the generalisability of their results. In this work, we propose a rigorous approach to align entailment recognition with argumentation theory. Wagemans’ Periodic Table of Arguments (PTA), a taxonomy of argument schemes, provides the appropriate framework to unify these two fields. To operationalise the theoretical model, we introduce a tool to assist humans in annotating arguments according to the PTA. Beyond providing insights into non-expert annotator training, we present Kialo-PTA24, the first multi-topic dataset for the PTA. Finally, we benchmark the performance of pre-trained language models on various aspects of argument analysis. Our experiments show that the task of argument canonicalisation poses a significant challenge for state-of-the-art models, suggesting an inability to represent argumentative reasoning and a direction for future investigation.

## introduction when engaging in critical discussion, people often employ stereotypical rules of inference to justify their claims. one of the main areas of study in argumentation theory is categorising these rules as templates to capture commonly employed patterns of reasoning. to this end, various taxonomies have been proposed, the most prevalent being walton's argumentation schemes @xcite ). walton's schemes are conceived of as templates for constructing arguments, with each scheme consisting of a list of (major and minor) premises that support a conclusion. alongside the premises, argument schemes are associated with several critical questions that serve to test the validity of the argument and identify fallacious reasoning @xcite . while walton conceived of his taxonomy as essential to computational argumentation, little work has been done on the automatic classification of argument schemes given a natural language text. in other words, most of the existing research starts with the assumption that the premises and conclusions and their function as part of a scheme are known a priori. similarly, researchers in natural language inference (nli) have made advances in the task of recognising textual entailment (rte). various models appear to achieve remarkable success in capturing textual entailment relationships. despite these strides, the absence of a standardized definition of entailment hampers the comparability and interpretability of models across different rte datasets and evaluation metrics @xcite . datasets designed for recognising textual entailment often imply some notion of defeasibility: ". . . in principle, the hypothesis must be fully entailed by the text. judgment would be false if the hypothesis includes parts that cannot be inferred from the text. however, cases in which inference is very probable (but not completely certain) are still considered true." @xcite . yet in the works we have surveyed, this notion remains to be ill-defined. this ambiguity in defining entailment not only impedes progress, but also raises questions about the reliability and generalisability of the models developed for these tasks. entailment in natural language involves defeasible inferences that draw on normative and commonsense knowledge. we posit that recognising entailment in natural language text can be more rigorously formulated as the identification of the scheme of inference being employed, and determining whether the hypothesis faithfully applies the scheme to justify the conclusion. in argumentation theoretic terms, recognising entailment involves classifying the argument scheme and then applying the appropriate critical questions to test the argument's validity. this goes beyond simple entailment and provides a measure of the extent to which one statement entails another. in an attempt to operationalise argumentation theoretic notions of natural language inference, we draw from wagemans (2016) which describes a taxonomy of argument types named the periodic table of arguments (pta). accordingly, we make the following contributions: ## background: structuring arguments the theory of argumentation seeks to provide systematic methods for the analysis, reconstruction, and evaluation of arguments. throughout history, philosophers have developed many models of argumentation that emphasise different hermeneutical frameworks. the atomic construct studied by all these theories is the argument, which is often defined as an inference in which a conclusion is supported by a set of, possibly implicit, premises @xcite . ## related work annotating argument schemes experimental work on classifying argument schemes is extremely limited and highly understudied. @xcite and @xcite both describe classification procedures for identifying schemes; however, few works systematically apply these procedures to real-world data. the periodic table's type identification procedure (atip) (wagemans, 2021) has gone through several iterations of refinement, with an earlier version being compared alongside walton schemes to measure the ability of annotators to agree on the classification of arguments according to each taxonomy respectively. @xcite annotate a set of us presidential debates from 2016 with argument schemes, both from walton's taxonomy as well as the periodic table. however, they fail to provide the intermediary steps that are required by the pta and only give the final classification. feng and hirst (2011) uses a small dataset of annotated arguments to train a decision tree to classify arguments in one of five walton schemes with moderate results. the dataset they used is no longer accessible. @xcite is the only existing, multi-topic dataset of argument schemes that is publically available, this dataset consists of arguments generated to match walton's argument schemes. defeasible textual entailment defeasibility in non-monotonic logic describes the notion of an inference that is valid on the basis of currently available evidence. unlike in classical logic, defeasible inference in non-monotonic logic allows the inference to be retracted based on further evidence or in the case of an exception to a rule. @xcite recognise the fact that the majority of natural language inference is defeasible and construct a defeasible nli dataset of defeasible inferences in english across a range of everyday topics. however, their focus is on generating defeasible hypotheses based on a premise rather than determining the nature of the defeasible inference as we focus on here. identifying enthymemes arguments in which the inference relies on implicit premises are known as enthymemes. the identification and disambiguation of enthymemes is quite closely related to that of classifying argument schemes since the argument scheme determines the auxiliary premises that are required for the argument to work. habernal et al. (2018) make progress towards the automatic reconstruction of implicit warrants by training a model to choose the correct warrant from a list of confounding options. beyond classification, @xcite generate implicit premises that support a given conclusion, while @xcite generate sequences of commonsense reasoning that connect the premises to the conclusion. we believe that the automatic classification of arguments will help researchers to achieve better results in this task, since identifying the structure of an argument automatically can assist models in discovering implicit premises. argument substance the detection of argument substance is an existing and well-known task in argument mining @xcite . the function of the substance in determining the argument type and its role in downstream tasks is, however, unique to the periodic table. ## data there are no publicly available datasets that researchers could benefit from to work on the automatic classification of argument schemes (section 3) according to the periodic table of arguments. to construct a new dataset, we began with the annotated data from @xcite . this dataset consists of a scrape from kialo foot_0 . kialo is a structured debating website that allows users to provide pro and con claims for various topics. each topic has a main claim for which pros and cons can be provided. each pro and con is then viewed as a new claim which can have its own sub-pros and sub-cons. this creates a tree-like debate structure for each topic as in figure 2 . we chose this dataset due to its structured nature, allowing us to consider claims as conclusions with pros as premises. additionally, each claim in kialo is relatively self-contained and can usually be understood without taking into consideration the rest of the discussion. this allows us to focus our study on inferential structures that are explicitly present in the text rather than being implied by context. a datasheet according to @xcite is provided in appendix b. ## experiments in this section, we will evaluate various approaches to learning the steps in the annotation process. in doing so, we wish to obtain a set of benchmarks that characterise the difficulty of the various tasks. 4 while we used cohen's kappa to measure agreement during training, we were only measuring agreement between two annotators, the non-experts and the expert. additionally, we were not aware of the class imbalance as the data had not been annotated yet, hence the change of agreement statistic. ## conclusion we constructed the first multi-topic dataset of argument types by using the theory of the periodic table of arguments. this dataset was created as a result of an annotation study, in which the annotators used our developed tool (argnotator). the focus of our experiments was on the canonical representation and classification of argument forms and substances. we showed that substance classification can be done effectively using bert-based models, while fine-tuned llms provide a promising approach to argument canonicalisation. the next steps involve detecting and classifying the argument lever, considering both its form and substance, to contribute to a more nuanced machine understanding of different argument types. the research also aims to include a critical evaluation of argument quality by posing relevant critical questions that assess the robustness and coherence of identified arguments. having the means to measure the quality of an argument systematically opens the doors to automatically constructing weighted argumentation frameworks @xcite ) that allow computational methods to be used to evaluate the strength of arguments within the context of a discourse/debate. this work provides some initial steps towards bridging the gap between natural language inference and symbolic approaches to computational argumentation.