Fine-grained Fallacy Detection with Human Label Variation (2025)

Abstract: We introduce FAINA, the first dataset for fallacy detection that embraces multiple plausible answers and natural disagreement. FAINA includes over 11K span-level annotations with overlaps across 20 fallacy types on social media posts in Italian about migration, climate change, and public health given by two expert annotators. Through an extensive annotation study that allowed discussion over multiple rounds, we minimize annotation errors whilst keeping signals of human label variation. Moreover, we devise a framework that goes beyond “single ground truth” evaluation and simultaneously accounts for multiple (equally reliable) test sets and the peculiarities of the task, i.e., partial span matches, overlaps, and the varying severity of labeling errors. Our experiments across four fallacy detection setups show that multi-task and multi-label transformer-based approaches are strong baselines across all settings. We release our data, code, and annotation guidelines to foster research on fallacy detection and human label variation more broadly.

## introduction fallacies are traditionally defined as types of reasoning that seem valid but are not @xcite . they occur when someone commits an error in the argumentation, either with the purpose of persuading the audience or unintentionally. disentangling the object of the discussion from the way in which it is expressed is paramount because even true statements can be invalid due to their faulty composition. social media is a perfect ground for studying fallacies, which have moved beyond the aristotle's realm of two-person debates into the vast domain of the internet. fallacious social media content can mislead a large audience, in some cases leading to the proliferation of misinformation on highly-debated topics @xcite . recognizing fallacies in everyday argumentation may not only limit the spread of harmful content but also plays a key role in developing individuals' critical thinking skills, ultimately contributing to mitigate faulty argumentation at its root and promoting democratic debate @xcite . fallacy detection is an open challenge in nlp and has shown to be intrinsically difficult for both humans and machines @xcite . although some fallacy detection datasets have been proposed in recent years, they either contain coarsegrained annotations (e.g., post-level; @xcite or assume that no more than one fallacy can be expressed in a given text segment @xcite . however, multiple fallacies may overlap in text @xcite and knowing where a fallacy occurs is central for educational purposes. moreover, current datasets encode a single "ground truth" for fallacies through label aggregation. this cancels out human label variation @xcite that naturally occurs in fallacy annotation due to multiple plausible answers and genuine disagreement, in turn affecting modeling and evaluation. we introduce faina (fallacy detection with individual annotations), a dataset for fallacy detection with human label variation (figure 1 ), along work langs genres and domains/topics scope #c #i multi hlv argotario (en) @xcite en general pair 5 † 909 ✗ ✗ argotario (de) @xcite de general pair 5 † 335 ✗ ✗ changemyview @xcite en various topics (discussions) text 1 † 3,622 ✗ ✗ informalfallacies* @xcite en politics; religion; veganism; fallacies span 8 1,708 ✗ ✗ adhomintweets @xcite en blm; metoo; veganism; remote work pair 6 † 2,400 ‡ ✗ ✗ languageofpopulism* @xcite en; it; pt political discourse (4 political leaders) text 9 † 1,919 ✓ ✗ logic @xcite en general snippet 13 2,449 ✗ ✗ logicclimate @xcite en climate change snippet 13 1,079 ✗ ✗ covid-19 @xcite en covid-19 snippet 10 526 ✗ ✗ climate @xcite en climate change snippet 10 477 ✗ ✗ elecdeb60to16 @xcite en political debates (us pres. campaigns) snippet 14 1,628 ✗ ✗ elecdeb60to20 @xcite en political debates (us pres. campaigns) span 6 1,989 ✗ ✗ rufal @xcite en with experiments across setups of increasing complexity and thorough data analyses. the faina dataset is the first annotated resource for fallacy detection embracing multiple plausible answers and natural disagreement. it is also the first dataset providing annotations at the fine-grained level of text segments with potential overlaps, accounting for over 11k annotated fallacies by two expert annotators across an inventory of 20 fallacy types. faina covers public discourse on migration, climate change, and public health issues in social media posts from twitter, and focuses on italian, a currently overlooked language in fallacy detection. to account for the complexity axes of the task, we design an evaluation framework that embraces the presence of multiple gold standards that are equally valid, partial span matches with overlaps, and the varying severity of fallacy classification errors. we conduct experiments across four setups of increasing complexity, from post-level to spanlevel fallacy detection and using either fallacy macro-categories or the full inventory. our results and analyses show that multi-task and multi-label transformer-based classifiers are strong baselines for fallacy detection tasks and that current large language models (llms) in zero-shot settings are still far from achieving satisfactory performance. we also provide thorough data analyses, including insights on our multi-round annotation procedure which minimizes annotation errors whilst keeping label variation, and a manual audit of the outputs generated by llms. to foster research on fine-grained fallacy detection and broadly on hu-man label variation in span-level tasks, we make our materials available to the nlp community. ## related work the increasing interest in fallacy detection has led to the creation of datasets with different characteristics in recent years ( regarding annotation, current datasets mainly provide coarse-grained labels at the text (post, paragraph), snippet, or text pair level. the only exceptions are span-level datasets by @xcite and @xcite which, however, do not foresee overlapping annotations. inspired by work on propaganda detection @xcite and persuasion techniques detection @xcite , our dataset in-stead provides span-level annotations with overlaps for a better model transparency. previous datasets include up to 3.6k human-annotated instances @xcite ) across 14 fallacies @xcite . instead, ours provides 11k human-labeled spans across 20 fallacy types. recent work highlighted the importance of considering human label variation @xcite , i.e., genuine disagreement @xcite , subjectivity and perspectives @xcite , and multiple plausible answers @xcite as signal rather than noise. yet, in fallacy detection all the labels from multiple annotators are typically aggregated @xcite , selectively chosen @xcite , or adjudicated through discussion or by an expert @xcite @xcite . more broadly, label variation has been mostly studied at the post or token level, with only few exceptions analyzing span-level disagreement in argument annotation @xcite . in our work, we resolve annotation errors while keeping genuine disagreement and multiple plausible answers, propose the first fallacy detection dataset at the span level with parallel annotations, and use human label variation during evaluation. ## harmonization of fallacies given that different sets of fallacies have been proposed in the literature, we selected our tagset by reviewing fallacy types from previous work (section 2) and harmonizing them. we thus compiled a list of 41 fallacies and conducted a pilot annotation on 15% of our data (section 4). this exploratory phase allowed us to homogenize fallacy names and unify those with similar definitions under the same label (e.g., {post hoc, false cause} → causal oversimplification), leading to a total of 20 fallacy types. most fallacy definitions are derived from musi et al. (2022) but also from da san martino et al. (2019b), reflecting the persuasive nature of posts in our data. fallacy definitions are provided below (extended definitions and examples are in appendix c.2): 1. ad hominem (ah): an excessive attack on an individual or a group; 2. appeal to authority (aa): appealing to an authority figure or a group consensus to support a thesis; 3. appeal to emotion (ae): manipulation of the recipient's emotions in order to win an argument; 4. causal oversimplification (co): the attributed causal relation is simplified and fallacious; 5. cherry picking (cp): choosing evidence which supports a given position, dismissing findings which do not; 6. circular reasoning (cr): the end of an argument comes back to the beginning without having proven itself; 7. doubt (do): questioning the credibility of someone or something; 8. evading the burden of proof (ep): a position is advanced without any support as if it was self-evident; 9. false analogy (fa): two different things or situations are treated equally; 10. false dilemma (fd): a claim presenting only two options or sides when there are many; 11. flag waving (fw): playing on strong national feeling (or with respect to a group) to justify or promote an action or idea; 12. hasty generalization (hg): a generalization is drawn from a sample which is not representative or not applicable to the whole situation; 13. loaded language (ll): using words/phrases with strong emotional implications (positive or negative) to influence an audience; 14. name calling or labeling (nc): labeling the object of the propaganda campaign as either something the audience fears, hates, finds undesirable or otherwise loves or praises; 15. red herring (rh): the argument supporting the claim diverges the attention to issues which are irrelevant for the claim at hand; 16. slippery slope (ss): implies that an improbable or exaggerated consequence could result from a particular action; 17. slogan (sl): a brief and striking phrase used to provoke excitement of the audience; 18. strawman (st): the arguer misinterprets an opponent's argument for the purpose of more easily attacking it, demolishes it, and then concludes that the opponent's real argument has been demolished; 19. thought-terminating cliché (tc): short and generic phrase that discourages meaningful discussion; 20. vagueness (va): words which are ambiguous are shifted in meaning in the process of arguing or are left vague, being potentially subject to skewed interpretations. we further organize these fallacy types into a taxonomy grouped around three macro-categories which include all the others: insufficient proof, simplification, and distraction, following similar efforts @xcite . the full taxonomy is presented in figure 4 and serves for evaluation purposes (section 5). ## faina dataset in this section, we describe the creation of faina, from data collection (section 4.1) to data annotation (section 4.2). we then provide detailed dataset statistics (section 4.3). data statements @xcite ) are available in appendix a. ## experiments in this section, we present experiments on fallacy detection using the faina dataset. we first introduce our experimental setup (section 5.1) and the approaches we employed (section 5.2). we then provide results and a thorough discussion, includ-ing insights for future work (section 5.3). ## conclusions we introduced faina, the first fallacy detection dataset embracing multiple plausible answers and natural disagreement at the fine-grained level of text segments. faina advances research on human label variation in nlp and opens new avenues for research. given its multi-topic and multi-year nature, it can be used to benchmark fallacy detection approaches in out-of-domain scenarios and across time. moreover, our annotation paradigm and guidelines can be applied to cover new languages, topics, and additional annotators. lastly, faina can be used for novel work on cross-lingual annotation transfer in which source and target languages are shifted, e.g., from italian to english.