Argument-based Detection and Classification of Fallacies in Political Debates (2023)

Abstract: Fallacies are arguments that employ faulty reasoning. Given their persuasive and seemingly valid nature, fallacious arguments are often used in political debates. Employing these misleading arguments in politics can have detrimental consequences for society, since they can lead to inaccurate conclusions and invalid inferences from the public opinion and the policymakers. Automatically detecting and classifying fallacious arguments represents therefore a crucial challenge to limit the spread of misleading or manipulative claims and promote a more informed and healthier political discourse. Our contribution to address this challenging task is twofold. First, we extend the ElecDeb60To16 dataset of U.S. presidential debates annotated with fallacious arguments, by incorporating the most recent Trump-Biden presidential debate. We include updated token-level annotations, incorporating argumentative components (i.e., claims and premises), the relations between these components (i.e., support and attack), and six categories of fallacious arguments (i.e., Ad Hominem, Appeal to Authority, Appeal to Emotion, False Cause, Slippery Slope, and Slogans). Second, we perform the twofold task of fallacious argument detection and classification by defining neural network architectures based on Transformers models, combining text, argumentative features, and engineered features. Our results show the advantages of complementing transformer-generated text representations with non-text features.

## introduction fallacious arguments have been firstly defined as defective inferences, i.e., logically invalid types of arguments @xcite . more recently, in a more pragmatic perspective, fallacious arguments have been defined as infringements of performance rules characteristic of a particular ideal type of argumentative engagement @xcite and as illicit dialectical shifts across different dialogue types, highlighting that the attempted move is inappropriate with respect to its pragmatic application context @xcite . despite their employment in many scenarios (e.g., online discussion platforms and blogs, tv roundtables), a natural testbed of this form of misleading argumentation is political debate. for instance, the ad hominem fallacy, where the plausibility of the argument depends on the credentials, personal background and past actions of the speaker, is probably one of the fallacy labels that is most often thrown around in political debate. this kind of arguments may sound convincing and has the goal to mislead the audience, persuading it about the validity of the argument. given the potential nefarious impact of these misleading arguments on the society, identifying and classifying fallacious arguments is therefore a main open challenge in argument mining (am) @xcite @xcite , and in nlp in general. existing approaches in the literature @xcite @xcite @xcite @xcite mainly concentrated on the classification of fallacious text snippets over a finite set of labels, leaving the challenging issue of identifying the fallacious text snippet and its boundaries under-investigated. in this paper, we tackle this open research question on a dataset of political debates from the u.s. presidential campaigns from 1960 to 2020. more precisely, the contribution of this paper is twofold. first, we extend an existing resource of u.s. political debates from the presidential campaigns annotated both with argument components and relations, and six fallacy categories (namely, ad hominem, appeal to authority, appeal to emotion, false cause, slippery slope and slogans). this new resource, named the elecdeb60to20 dataset, includes now also the debates of the 2020 presidential campaign (trump-biden) with all the related annotations. second, we propose a new approach, based on transformers, to detect the fallacious text snippets in these debates, and then classify them along the six fallacy categories. this approach encodes the argument components (i.e., premise, claim), the argument relations (i.e., support, attack) and the pos tags to successfully identify and classify fallacious arguments. experimental results show that the proposed approach outperforms standard baselines and concurrent approaches with an average f1-score of 0.74, with our proposed model named multifusion bert, on the task of fallacious argument detection and classification. whilst most of the computational approaches targeting fallacious argumentation focus on the pure classification of such nefarious content @xcite @xcite @xcite , the originality of our contribution is that it proposes, to the best of our knowledge, the first neural architecture to both detect fallacious arguments and classify them in political debates, and one of the very few approaches to tackle this task in general, outperforming competing approaches @xcite . the urgency to study fallacies in political discourse is crucial both from the philosophical and the political perspective. it emphasizes the need to scrutinize political arguments for sound reasoning rather than deception. understanding logical flaws is a main issue for informed decision-making, enabling the recognition and assessment of fallacious arguments in political discourse. moreover, examining how fallacies are employed in political debates reveals their strategic role in influencing opinion and diverting attention. this strategic use of fallacies mirrors the subtleties of language, emphasizing the interplay between rhetoric, philosophy, and political communication. furthermore, it offers valuable insights into the dynamics of public discourse and the need for critical analysis @xcite . ## related work over the years, there has been a growing interest in the field of nlp to the detection of fallacies and related phenomena, including misinformation and propaganda @xcite . the pioneering work @xcite on fallacies in newspaper news has been a signif-icant source of inspiration in this area. recently, researchers have made significant progress in identifying and classifying fallacies within discourse. in this section, we discuss the approaches proposed in the literature on the two tasks of fallacy classification (section 2.1) and fallacy detection (section 2.2). ## elecdeb60to20 dataset to effectively address the task of detecting and classifying fallacious arguments within political debates, we decided to rely on the elecdeb60to16 dataset @xcite . it comprises televised debates from u.s. presidential election campaigns spanning from 1960 to 2016. these debates were sourced from the website of the commission on presidential debates 1 , which openly provides transcripts of debates broadcasted on television and featuring the prominent candidates for presidential and vicepresidential nominations in the united states. all information on this website is accessible to the public. considering the most recent presidential election between trump and biden occurred in 2020, we expanded the dataset with the transcripts of the debates of this election campaign to include updated annotations, incorporating argumentative components such as claims and premises, as well as the relations between these components, i.e., support and attack. as a result of this annotation update, the dataset is renamed as elecdeb60to20 2 , reflecting the coverage of debates spanning from 1960 to 2020. this updated dataset provides a more comprehensive and contemporary collection of fallacies in political debates, enhancing the relevance and applicability of the data for further analysis and research in the field. this resource is a valuable benchmark for investigating potential connections between specific argument components and relations that underlie the occurrence of fallacious arguments. ## fallacy detection we cast the fallacy detection task as an information extraction problem, where the goal is to identify and classify in the debates the textual snippets corresponding to the six categories of fallacies annotated in the context of a political debate (see section 3.1 for the list of fallacies and their description). we rely on the bio/iob data format, and specific tags are assigned to annotate the fallacies, i.e., b-adhominem, i-adhominem, b-appealtoauthority, i-appealtoauthority, b-appealtoemotion, i-appealtoemotion, b-falsecause, i-falsecause, b-slipperyslope, i-slipperyslope, b-slogans, i-slogans, o. the fallacy detection and classification tasks consist therefore in assigning one of these thirteen predefined labels to each token. to have a richer representation of fallacy annotations, we build a contextual framework that includes the sentence containing the fallacy, as well as the preceding and following sentences. when the fallacious sentence is the first or last in the dialogue, the preceding or following sentence is excluded. ## evaluation ## conclusion existing argumentation schemes @xcite to identify flawed and invalid forms of reasoning often fall short when applied to fallacious arguments employed in real-world contexts like political debates. to tackle this challenge, the contribution of this paper is twofold. first, we extended the elecdeb60to16 dataset by incorporating the trump vs. biden 2020 presidential debate along with argumentative annotations and fallacies. second, we proposed and evaluated multifusion bert, a transformer-based architecture that combines the debate text, the argumentative features (i.e., components and relations), and engineered features to perform the fallacy detection and classification task. our results highlight the main role of argumentative features in the correct identification and classification of fallacious arguments. this approach yields an average performance improvement of 2.12% compared to baseline methods and competing approaches. as future research, we intend to delve deeper into fallacious argumentation by integrating knowledge in order to address more challenging fallacy categories like causal ones, where reasoning and knowledge-based features are required to identify the fallacy. our further objective is to generate valid arguments from identified fallacious ones and their context. additionally, a challenge we aim to tackle is to explore ways to counter the formal invalidity of fallacious arguments through the generation of new arguments.