Reason from Fallacy: Enhancing Large Language Models’ Logical Reasoning through Logical Fallacy Understanding (2024)

Abstract: Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but they still struggle with some complicated reasoning tasks including logical reasoning. One non-negligible reason for LLMs’ suboptimal performance on logical reasoning is their overlooking of understanding logical fallacies correctly. To evaluate LLMs’ capability of logical fallacy understanding (LFU), we propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW in this paper. Towards these LFU tasks, we have successfully constructed a new dataset LFUD based on GPT-4 accompanied by a little human effort. Our extensive experiments justify that our LFUD can be used not only to evaluate LLMs’ LFU capability, but also to fine-tune LLMs to obtain significantly enhanced performance on logical reasoning.

## introduction as a cognitive process, logical reasoning plays an important role in many intellectual activities, such as problem solving, decision making and planning @xcite . up to now, a lot of efforts have been dedicated to logical reasoning based on language models @xcite @xcite . more recently, the popularity of large language models (llms) such as chatgpt @xcite and @xcite stimulates the growth of research on llm-based logical reasoning. compared to traditional small language models, llms have demonstrated better performance in many reasoning tasks. however, llms still struggle with some more complex reasoning tasks including logical reasoning. one non-negligible reason for llms' suboptimal performance on logical reasoning is their overlooking of understanding logical fallacies correctly. as early as 350 bc, aristotle first proposed the premise: my french colleague is very romantic. hypothesis: i think all french people are romantic. before entailment. llm after llm a has b a in c c has b d in e e in f f in d fallacy knowledge …… contradiction ! this is faulty generalization. figure 1 : llms have deficiencies in logical reasoning. once they understand logical fallacies, they know how to avoid logical fallacies, and thus improve their performance in various logical reasoning tasks. chen et al. have observed that, llms often commit logical fallacies in logical reasoning, such as "either protect the environment or develop the economy." (false dilemma) and "some roses are not red because not all roses are red." (circular reasoning). it has been found that language models could avoid mistakes only when they understand what mistakes are @xcite , which justifies the ancient greek philosopher epi-curus's saying "the mistake is the first step to save yourself." based on our empirical studies, we have also found that the logical reasoning capability of llms is closely related to their understanding of logical fallacies. the previous studies related to logical fallacy @xcite @xcite only focus on logical fallacy detection, i.e., the identification and classification of logical fallacies, rather than systematically evaluating llms' capability of logical fallacy understanding (lfu), not to mention improving llms' lfu capability. moreover, they have not explored the relationships between lfu and logical reasoning, which is crucial to improve llms' capability of logical reasoning through enhancing their lfu capability. to address this problem, we focus on evaluating and enhancing llms' lfu capability in this paper, so as to enhance their capability of logical reasoning. nonetheless, our work has to face several challenges as follows. first, we need to formalize the concrete tasks for lfu, since no previous studies focus on this problem. second, we need a new dataset specific to lfu, as the previous datasets of logical fallacies @xcite only contain the logical fallacy types presenting in the sentences. to this end, we should propose a framework of constructing the lfu dataset towards the concrete lfu tasks, and then truthfully evaluating llms' lfu capability with the dataset. to overcome these challenges, we primarily focus on constructing a dataset for lfu in this paper, of which the samples are generated to evaluate models' achievement on the following five lfu tasks corresponding to three cognitive dimensions of what, why, and how @xcite . our proposed lfu tasks simulate the human understanding process of logical fallacies. towards these tasks, we design a pipeline framework to automatically generate and synthesize a high-quality dataset, namely logical fallacy understanding dataset (lfud), based on gpt-4 accompanied by a little human effort. specifically, we first collect some sentences as the propositions (statements) which are the basic logic units and used to generate the sentences containing logical fallacies. then, with the help of gpt-4, we generate sentences based on the propositions with twelve typical logical fallacy types @xcite . and for each lfu task we propose, the instances of each fallacy type are synthesized. then, we use our lfud to evaluate the lfu capability of some representative llms. for the ultimate objective of our work, i.e., enhancing llms' capability of logical reasoning, we further fine-tune these llms with the instances in lfud. our extensive experiments reveal that fine-tuning llms with lfud can significantly enhance their logical reasoning capability. in summary, our main contributions in this paper include: ## methodology of dataset construction in this section, we present the pipeline of constructing our lfud, of which the overall framework is depicted in figure 2 . starting from the propositions, we detail the steps of synthesizing the samples towards five lfu tasks and the twelve representative logical fallacy types. ## conclusion to evaluate llms' lfu performance, we propose five concrete tasks from three cognition dimensions what, why, and how. towards these tasks, we constructed a high quality dataset lfud, which has been proven helpful by our extensive experiments to enhance llms' capability of logical reasoning. we hope our work in this paper is instructive and our lfud becomes a valuable resource for further research on lfu.