Multimodal Fallacy Classification in Political Debates (2024)

Abstract: Recent advances in NLP suggest that some tasks, such as argument detection and relation classification, are better framed in a multimodal perspective. We propose multimodal argument mining for argumentative fallacy classification in political debates. To this end, we release the first corpus for multimodal fallacy classification. Our experiments show that the integration of the audio modality leads to superior classification performance. Our findings confirm that framing fallacy classification as a multimodal task is essential to capture paralinguistic aspects of fallacious arguments.

## introduction recent studies in argument mining (am) mainly focus on semantic textual analysis @xcite . however, a different line of research has shown the importance of including paralinguistic features in argumentative discourse analysis across a wide variety of domains, including advertisements, news coverage, and legal analytics @xcite and in cognate tasks such as fake news detection @xcite . to evaluate these findings, multimodal argument mining (mam) emerged to gain a more comprehensive understanding of argumentative discourse via integrating multiple modalities. so far, mam applications include argument detection, argument component classification, and relation classification @xcite @xcite . in contrast, argumentative fallacy classification @xcite ) has yet to be explored. while not covering all fallacy types comprehensively, @xcite analyzes political discourse to show the connection between human sound, the paralinguistic component of fallacious arguments, and their verbal content. in particular, they link stereotypes on accents to the ad hominem fallacy, as politicians use accents with negative stereotypes to mock or discredit opponents during election campaigns. likewise, a staccato speech rhythm can be linked to the appeal to authority fallacy since it is associated with strictness, authority, and dominance, and prosodic elements emphasizing anger can increase the persuasive impact of appeal to emotion fallacies @xcite . these observations lead us to believe that argumentative fallacy classification should be formulated as a multimodal task in the context of political debates. to tackle this new task, we introduce mm-used-fallacy, the first corpus for multimodal argumentative fallacy classification. we extend the used-fallacy foot_0 corpus @xcite by integrating the audio modality. we follow the methodology described in @xcite to align audio recordings to annotated debate transcripts. the new corpus contains 1,891 annotated text-audio pairs. in our evaluation, we extend the multimodal architecture of @xcite by including state-of-the-art unimodal encoding models, such as pre-trained transformers, that are suitable for low-resource scenarios. our findings confirm that the combination of text and audio leads to superior classification performance for several models, corroborating our initial hypothesis on argumentative fallacy classification. we release our corpus and code in a public repository. foot_1 2 related work several studies have targeted political debates in am @xcite @xcite , inspired by the recent findings on the importance of paralinguistic components like prosodic features for argument detection @xcite . most notably, @xcite and @xcite introduced two independent extensions of used @xcite , the us presidential election corpus. these extensions represent the largest to-date multimodal corpora for am. similar efforts have been conducted in the context of fake news detection. in particular, @xcite observed superior classification performance in several tasks, such as the detection of check-worthy claims, when following a multimodal problem formulation. while the existing studies on political debates have targeted a wide set of am tasks, the automated analysis of argumentative fallacies has never been explored from a multimodal perspective. in the context of multimodal deep learning, it is a standard approach to combine unimodal models via fusion techniques @xcite . this is also the case for mam @xcite . in this work, we extend the methodology introduced by @xcite to include state-of-the-art models for text encoding, such as roberta @xcite and sbert @xcite , and audio encoding like clap @xcite . ## experimental setup we frame argumentative fallacy classification as a multi-class sentence classification task. we evaluate models via leave-one-out cross-validation, totaling 28 individual model runs. in particular, we build folds such that, at each iteration, each debate is either in the left-out test split or in the remaining splits. we experiment with the multimodal architecture presented in @xcite (figure 1 ). the text module comprises a pre-trained text embedding model and a dropout layer on top of it. the audio module consists of a pre-trained audio embedding model, a bilstm layer, and a dropout layer. the output of the text and audio modules is concatenated and fed to the classification module, defined as a stack of dense layers. we extend @xcite by exploring two audio signal encoding and three text encoding models. for audio, we consider wav2vec @xcite and clap @xcite , while for text, we use pretrained bert @xcite , roberta @xcite , and sbert @xcite . we also experimented with pre-trained text embeddings without fine-tuning but obtained unsatisfactory results (not reported). we train all models using cross-entropy loss and adam (kingma and ba, 2015) for optimization. see appendix e from more details. to assess the contribution of each modality, we consider three input configurations: text-only, audio-only, and text-audio. we address main fallacy categories only, leaving sub-categories for future work. to limit model overfitting on certain fallacy categories, we remove duplicate snippets, totaling 1063 unique dialogue-snippet pairs. ## results the observed performance improvements are not equally distributed over fallacy types. for instance, text-only models achieve superior or equal f1-score for the slogans (s) category. one reason for that could be the low number of examples in this category. another reason could be that, according to linguistic analyses, slogans primarily rely on figurative language @xcite , whereas prosodic features have minimal impact on them @xcite . classifying fallacies at the sentence level may represent an additional challenge for the audio modality. as shown in our investigation into the sensitivity of our results to individual speakers and the influence of audio features on fallacy classification revealed insightful nuances across different debates. while our dataset structure, comprising pairs of candidates per debate, prevented us from pinpointing specific speakers benefiting most from audio features, we observed significant variations across folds and debates. notably, recent debates tended to show a more pronounced benefit from audio features, possibly due to improved audio quality or the predominance of newer audio files in the training data for audio feature extractors. specifically, debates such as 1980-carter/reagan, 2004-bush/kerry, and 2008-mccain/obama notably benefited from the inclusion of audio features. in other debates, like 1976-carter/ford and 1996-clinton/dole, integrating audio features led to decreased classification accuracy, while debates like 1960-kennedy/nixon yielded mixed results. these findings underscore the complex interplay between speaker characteristics, debate context, and the utility of audio features in enhancing fallacy classification, highlighting the need for further investigation. ## conclusion we posit that argumentative fallacy classification should be framed as a multimodal task. to empirically evaluate our hypothesis, we build the first dataset for multimodal argumentative fallacy detection, mm-used-fallacy. our results show that the integration of audio modality is indeed beneficial, observing significant performance improvement (4-8 f1-score percentage points) in a variety of model architectures. our findings are coherent with recent studies in argument mining @xcite and fake news detection @xcite . we believe that a multimodal formulation should affect the experimental setting starting from data collection, if possible, so as to capture several audio properties like non-verbal features @xcite in addition to prosodic ones. we believe that the multimodal resource that we provide has significant potential for enabling further experimentation. some possibilities are addressing fallacy subcategories, experimenting with other tasks like argumentative fallacy detection, evaluating the importance of argumentative components @xcite , and employing novel multimodal architectures. moreover, to gain a deeper understanding of our results, employing interpretability techniques designed to emphasize the significance of paralinguistic elements over linguistic elements in prediction may be beneficial.