Are LLM s Good Zero-Shot Fallacy Classifiers? (2024)

Abstract: Fallacies are defective arguments with faulty reasoning. Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions. However, existing fallacy classifiers are limited by the requirement for sufficient labeled data for training, which hinders their out-of-distribution (OOD) generalization abilities. In this paper, we focus on leveraging Large Language Models (LLMs) for zero-shot fallacy classification. To elicit fallacy-related knowledge and reasoning abilities of LLMs, we propose diverse single-round and multi-round prompting schemes, applying different taskspecific instructions such as extraction, summarization, and Chain-of-Thought reasoning. With comprehensive experiments on benchmark datasets, we suggest that LLMs could be potential zero-shot fallacy classifiers. In general, LLMs under single-round prompting schemes have achieved acceptable zeroshot performances compared to the best fullshot baselines and can outperform them in all OOD inference scenarios and some opendomain tasks. Our novel multi-round prompting schemes can effectively bring about more improvements, especially for small LLMs. Our analysis further underlines the future research on zero-shot fallacy classification. Codes and data are available at: https://github.com/panFJCharlotte98/Fallacy_Detection.

## introduction a fallacy is a defective argument derived from erroneous or invalid reasoning that may appear to be reasonable but are, in fact, logically unsound or faulty @xcite @xcite . figure 1 illustrates examples of different fallacy types, for instance, a fallacy type of circular reasoning: i am a great leader because i make great leadership decisions. fallacies commonly appear in various scenarios, such as news articles @xcite , advertisements @xcite , propaganda @xcite , politics @xcite , and social media @xcite . they could be intentionally exploited to disseminate misinformation @xcite , manipulate public opinions, undermine rational discussions, and influence critical decision-making @xcite . in consequence, detecting and classifying fallacies becomes an imperative challenge. however, existing fallacy classification methods typically follow a full-shot supervised fine-tuning manner, limited by three issues: (i) they require sufficient labeled data for training @xcite , but collecting these data is often time-consuming and expensive. this is because annotating fallacy data typically rely on expert knowledge, due to the complex and esoteric nature of fallacies @xcite . (ii) they cannot well generalize to out-of-distribution (ood) fallacies and discourse types @xcite due to the inherent limitation of supervised learning. once given an unseen fallacy class or discourse from other domains, they have to retrain a new model from scratch, which costs considerable computational and time resources. (iii) they are susceptible to imbalanced data @xcite . they could reach high performance on the dominant fallacy types while low on infrequent ones. as a result, full-shot fallacy classifiers cannot fulfill real-world application scenarios. motivated by the above issues, in this paper, we explore zero-shot fallacy classification, i.e., classifies fallacies without training data. specifically, we concentrate on large language models (llms) as they have been extensively pretrained and possess wide knowledge and strong reasoning abilities. to investigate llms' performance on this task, we consider two kinds of prompting schemes. first, we employ a single-round prompting scheme. we simply prompt llms to classify fallacies either with or without manually crafted fallacy type definitions. second, to elicit the inherent fallacy-related knowledge and reasoning abilities of llms, we further propose diverse multi-round prompting schemes: we instruct llms to analyze and classify fallacies through definition generation, general fallacy analysis (with warm-up), premise & conclusion analysis, and chain-of-thought. under these prompting schemes, we observe that zero-shot prompted llms can outperform or achieve comparable performances with sota full-shot fine-tuned t5 baselines on some open-domain benchmark datasets and can generally achieve sub-optimal performances on hard domain-specific datasets. more importantly, llms consistently demonstrate advantage over full-shot trained baselines on ood inferences on low-resource fallacy classification tasks. besides, our novel multi-round prompting schemes can effectively improve llms' classification performance compared to the single-round prompts, especially for small llms, e.g., llama3, qwen2.5 and mistral. we conclude our main contributions as below: zero-shot single-round baselines zero-shot multi-round promptings (a) with definitions & without definitions fallacy type (b) definition generation definitions ? fallacy type list definitions text fallacy type (c) general fallacy analysis fallacy type text fallacious ? fallacy analysis fallacy type list (f) zero-shot cot fallacy analysis fallacy type list text step-by-step ? conclusion? fallacy type (e) premises & conclusion premises: … conclusion: … fallacy analysis fallacy type text fallacious ? fallacy type list premises? conclusion? (d) general fallacy analysis with warm-up large language models the ability of llms in logical reasoning has been quantitatively or qualitatively evaluated, e.g., commonsense causality reasoning @xcite @xcite and abstract reasoning @xcite @xcite . particularly, evidence @xcite has verified the existence of inconsistency in its reasoning process (same for the latest llms like gpt-4). this motivates us to guide llms through the proposed multi-round prompting schemes. our work distinguishes from the previous in two points: (i) instead of a general reasoning task, we focus on the fallacy classification, a more advanced reasoning task that requires not only expert knowledge of but also rigorous logical reasoning. (ii) apart from the common single-round prompting, we propose the novel multi-round prompting schemes that bring along noticeable improvements for small llms. ## methodology in this section, we apply llms for zero-shot fallacy classification by two kinds of prompting schemes: zero-shot single-round prompting and zeroshot multi-round prompting. ## conclusion in this paper, we explore llms' performance on zero-shot fallacy classification. we propose both single-round and multi-round prompting schemes to fully elicit llms' fallacy classification ability. through extensive experiments on benchmark datasets, we demonstrate that llms with a zeroshot single-round prompting scheme can outperform or achieve highly comparable performances with sota full-shot fine-tuned t5 baselines on some open-domain benchmark datasets and can generally achieve sub-optimal performances on hard domain-specific datasets. besides, our multiround prompting schemes can effectively enhance the performances, especially for small llms. the performance gaps between zero-shot prompted llms and full-shot fine-tuned baselines are acceptable, indicating llms' potential for further improvements for this task. our detailed analysis moreover highlights the future research for zeroshot fallacy classification.