MAFALDA : A Benchmark and Comprehensive Study of Fallacy Detection and Classification (2024)

Abstract: We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies.

## introduction a fallacy is an erroneous or invalid way of reasoning. consider, e.g., the argument "you must either support my presidential candidacy or be against america!". this argument is a false dilemma fallacy: it wrongly assumes no other alternatives. fallacies can be found in various forms of communication, including speeches, advertisements @xcite , twitter/x posts @xcite , and political debates @xcite . they are also part of propaganda techniques employed to shape public opinion and promote specific agendas. most notably, fallacies played a role in the 2016 brexit referendum @xcite , and the debate about covid-19 vaccinations @xcite , where fake news spread on news outlets and in social networks @xcite @xcite . detecting and identifying these fallacies is thus a task of broad importance. the recent advances in deep learning and the availability of more data have given rise to approaches for detecting and classifying fallacies in text automatically @xcite @xcite @xcite . and yet, this work is fragmented: most approaches focus on specific types of corpora (e.g., only speeches) or specific types of fallacies (e.g., only ad hominem fallacies). furthermore, not all works use the same types of fallacies, there is no consensus on a common terminology (hansen, 2020), and fallacies come at different levels of granularity: an appeal to emotion can be, for instance, an appeal to anger, fear, pride, or pity. most importantly, annotating fallacies is an inherently subjective task. while previous works acknowledge the subjectivity, none explicitly embraces it. on the contrary, the annotators typically aim for a unique annotation -by discussion or vote. additionally, existing works do not give human performances on the benchmarks and evaluate only models. this paper addresses these drawbacks by introducing the multi-level annotated fallacy dataset mafalda -a manually created fallacy classification benchmark. our contributions are: 1. a taxonomy of fallacies that aligns, consolidates, and unifies existing public fallacy collections (section 3). 2. a new annotation scheme -coined disjunctive annotation scheme -that accounts for the inherent subjectivity of fallacy annotation by permitting several correct annotations (section 4). 3. a corpus that merges existing corpora, with 9,545 non-annotated texts and 200 manually annotated texts with 260 instances of fallacies, each with a manual justification (section 5). 4. a study of the performance of state-of-the-art language models and humans on our benchmark (section 6). all our code and data are publicly available under a cc-by-sa license 1 at https://github.com/ chadihelwe/mafalda , allowing our study to be reproduced and built upon. we start our paper by discussing related work in section 2. while all of these works advanced the understanding of fallacy detection, the studied fallacies are not the same across different works and are sometimes outright disjoint. the only work that creates a comprehensive taxonomy of fallacies is the (not yet peer-reviewed) work of @xcite . however, this work enumerates 232 fallacies, which is clearly too many to be handled by a human. and indeed, their dataset is composed only of toy examples generated by gpt-4. in this paper, we propose a benchmark that not only unifies public datasets on fallacy detection in a handy yet all-embracing taxonomy, but also comes with human annotations, human explanations, and evaluations for both language models and humans. ## experiments we will now evaluate the ability of state-of-theart llm to detect the fallacies in our benchmark. our benchmark is not intended for training or finetuning, and hence, we study a zero-shot setting with basic prompts. we are interested in the task of fallacy detection and classification of a given text, i.e., the input is a text, and the output is a list of annotated spans. ## conclusion we have presented mafalda, a unified dataset designed for fallacy detection and classification. this dataset integrates four pre-existing datasets into a cohesive whole, achieved through developing a new, comprehensive taxonomy. this taxonomy aligns publicly available taxonomies dedicated to fallacy detection. we manually annotated 200 texts from our dataset and provided an explanation in the form of a completed template for each of them. the disjunctive annotation scheme we proposed embraces the subjectivity of the task and allows for several alternative annotations for the same span. we have further demonstrated the capabilities of various large language models in zero-shot fallacy detection and classification at the span level. while level 0 classification shows good results, levels 1 and 2 are largely out of reach of llms in zeroshot settings. we hope that our benchmark will enable researchers to improve the results of this challenging task. future work includes expanding into few-shot settings and exploring advanced prompting techniques, such as chain-of-thought, using the template-based definitions of fallacy and the taxonomy we provided. furthermore, we believe that using a top-down approach, i.e., from level 0 to level 2 of our taxonomy, may provide better results than the bottom-up approach we used in our experiments. regarding our disjunctive annotation scheme, we are interested in exploring its use in other nlp domains. lastly, we plan to enrich the dataset with more annotated examples for model fine-tuning.