Logical Fallacy Detection (2022)

Abstract: Reasoning is central to human intelligence. However, fallacious arguments are common, and some exacerbate problems such as spreading misinformation about climate change. In this paper, we propose the task of logical fallacy detection, and provide a new dataset (Logic) of logical fallacies generally found in text, together with an additional challenge set for detecting logical fallacies in climate change claims (LogicClimate). Detecting logical fallacies is a hard problem as the model must understand the underlying logical structure of the argument. We find that existing pretrained large language models perform poorly on this task. In contrast, we show that a simple structure-aware classifier outperforms the best language model by 5.46% F1 scores on Logic and 4.51% on LogicClimate. We encourage future work to explore this task since (a) it can serve as a new reasoning challenge for language models, and (b) it can have potential applications in tackling the spread of misinformation. Our dataset and code are available at https://github.com/causalNLP/logical-fallacy

## introduction reasoning is the process of using existing knowledge to make inferences, create explanations, and generally assess things rationally by using logic @xcite . human reasoning is, however, often marred with logical fallacies. fallacious reasoning leads to disagreements, conflicts, endless debates, and a lack of consensus. in daily life, fallacious arguments can be as harmless as "all tall people like cheese" (faulty generalization) or "she is the best because she is better than anyone else" (circular claim). however, logical fallacies are also intentionally used to spread misinformation, for instance "today is so cold, so i don't believe in global * equal contribution. † done during the research internship at eth zürich. 1 our dataset and code are available at https:// github.com/causalnlp/logical-fallacy . warming" (faulty generalization) or "global warming doesn't exist because the earth is not getting warmer" (circular claim). in order to detect such fallacious arguments, we propose the task of logical fallacy detection. logical fallacy detection methods can be helpful to tackle important social problems. for instance, these methods can be combined with fact-checkers @xcite for misinformation detection as many claims can be factually correct but still fallacious. however, logical fallacy detection is challenging as it requires a model to discover egregious patterns of reasoning @xcite . to address this pressing need and encourage more work to detect reasoning flaws, we construct a dataset of logical fallacies consisting of general logical fallacies (logic), and a challenging extrapolation set of climate claims (logicclimate), as shown in figure 1 . we find that this task is challenging for 12 pretrained large language models, whose performances range from 8.62% to 53.31% micro f 1 scores on the logic dataset. logical fallacy examples faulty generalization (18.01%) "i met a tall man who loved to eat cheese. now i believe that all tall people like cheese." "sometimes flu vaccines don't work; therefore vaccines are useless." ad hominem ## logical fallacy dataset first, we introduce our data. our logical fallacy dataset consists of two parts: a) a set of common logical fallacies (logic), and b) an additional challenge set of logically fallacious claims about climate change (logicclimate). the entire logic dataset contains 2,449 logical fallacy instances across 13 logical fallacy types. we randomly split the data into train, dev, and test sets; dataset statistics are shown in comparison with existing datasets. due to the challenges of data collection, all previous existing datasets on argument quality are of limited size. in dataset # claims # classes purpose arg. suff. 1,029 binary detect insufficiency ad homi. 2,085 binary detect name calling logic 2,449 multiple detect all fallacy types compared to the existing datasets, our dataset has two advantages: (1) we have a larger number of claims in our dataset, and (2) our task serves the more general purpose of detecting all fallacy types instead of a single fallacy type. these two characteristics make our dataset significantly more challenging. ## a structure-aware model the task of logical fallacy classification is unique in that logical fallacies are not just about the content words (such as the sentiment-carrying words in a this example matches the following logical form: "if [msk1], then [msk2]" leads to "if [msk2], then [msk1]". ## limitations and future work some limitations of the current proposed model is that it can be effective for text with clear spans of paraphrases, but does not always work for more complicated natural text, such as the journalistic style in the climate change news articles. another limitation is that, in the scope of this work, we only explored one logical form for each fallacy type. since there could be multiple ways to verbalize each fallacy, future work can explore if the models can match the input text to several candidate logical forms, and create a multi-way voting system to decide the most suitable logical fallacy type. orthogonal to model development, future work can also explore other socially meaningful applications of this work, in line with the nlp for social good initiative @xcite , foot_1 logical fallacy detection can be used in various settings: to validate information and help fight misinformation along with fact-checkers @xcite , to check whether cognitive distortions @xcite @xcite are correlated with some types of logical fallacies, to check whether some logical fallacies are commonly used as political devices of persuasion in politicians' social media accounts, among many other possible application cases. logic is at the center of research on argumentation theory, an active research field in both the linguistics community @xcite @xcite , and nlp community @xcite @xcite . the most relevant nlp works include classification of argument sufficiency @xcite , ad hominem fallacies from reddit posts @xcite and dialogs @xcite , as well as automatic detection of logical fallacies using a rule parser (nakpih and santini, 2020). to the best of our knowledge, our work is the first to formulate logical fallacy classification with deep learning models, and also the first to propose logical fallacy detection for climate change news. ## conclusion this work proposed logical fallacy detection as a novel task, and constructed a dataset of common logical fallacies and a challenge set of fallacious climate claims. using this dataset, we tested the performance of 12 existing pretrained language models, which all have limited performance when identifying logical fallacies. we further proposed a structure-aware classifier which surpasses the best language model on the dataset and the challenge set. this dataset provides a ground for future work to explore the reasoning ability of nlp models.