Boosting Logical Fallacy Reasoning in LLM s via Logical Structure Tree (2024)

Abstract: Logical fallacy uses invalid or faulty reasoning in the construction of a statement. Despite the prevalence and harmfulness of logical fallacies, detecting and classifying logical fallacies still remains a challenging task. We observe that logical fallacies often use connective words to indicate an intended logical relation between two arguments, while the argument semantics does not actually support the logical relation. Inspired by this observation, we propose to build a logical structure tree to explicitly represent and track the hierarchical logic flow among relation connectives and their arguments in a statement. Specifically, this logical structure tree is constructed in an unsupervised manner guided by the constituency tree and a taxonomy of connectives for ten common logical relations, with relation connectives as non-terminal nodes and textual arguments as terminal nodes, and the latter are mostly elementary discourse units. We further develop two strategies to incorporate the logical structure tree into LLMs for fallacy reasoning. Firstly, we transform the tree into natural language descriptions and feed the textualized tree into LLMs as a part of the hard text prompt. Secondly, we derive a relation-aware tree embedding and insert the tree embedding into LLMs as a soft prompt. Experiments on benchmark datasets demonstrate that our approach based on logical structure tree significantly improves precision and recall for both fallacy detection and fallacy classification.

## introduction logical fallacy refers to the use of invalid or flawed reasoning in an argumentation @xcite @xcite . logical fallacy can occur as unintentional mistakes or deliberate persuasions in a variety of human communications, such as news media @xcite , educational essay @xcite , political debates @xcite , or online discussions @xcite . logical fallacies can lead to harmful consequences for society, such as spreading misinformation @xcite , raising public health risks @xcite , manipulating public opinions @xcite @xcite , introducing societal bias and polarization @xcite . despite their prevalence and harmfulness, understanding logical fallacies still remains a challenging task, which requires both semantics understanding and logical reasoning @xcite . in this paper, we focus on fallacy detection and classification, and aim to develop an approach that generalizes across different domains and genres. the key observation is that logical fallacies heavily rely on connective phrases to indicate an intended logical relation between two textual arguments, while the semantics of the arguments do not actually support the claimed logical relation. figure 1 shows two examples where the connective phrases were bolded. the first example uses the connective words therefore and cause to suggest a causal relation between vaccinations and increasing flu cases, however, the temporal relation between the two events as stated in the first half of the statement does not necessarily entail a causal relation between them, and indeed, their semantics do not actually support the suggested causal relation. recognizing this discrepancy undermines the credibility of the whole statement. similarly in the second example, the connective word likewise is commonly used to indicate an analogy relation, however, the second argument is clearly a specific case of the general condition stated in the first argument and therefore there is no analogy relation between them, and recognizing this mismatch between the suggested logical relation and the real relation enables us to detect this fallacy. as the logical relation indicated by a connective phrase may not be supported by semantics of its arguments in the context, we identify the purposefully indicated logical relations in a context-free unsupervised manner by matching a connective phrase with a taxonomy of connectives compiled for ten common logical relations (conjunction, alternative, restatement, instantiation, contrast, concession, analogy, temporal, condition, causal). to construct a logical structure tree, we first construct a constituency tree for a statement and then search in the constituency tree for connective phrases in the top-down left to right order, and the first found connective phrase will be the root node of the logical structure tree. next, we identify the text spans of its two arguments using rules and recursively build the left and right sub-trees by applying the same procedure to constituency tree segments corresponding to the two arguments. the logical structure tree is integrated into llms for fallacy reasoning using two strategies. the first considers textualized tree, where we convert the tree into natural language descriptions, making the tree readable by llms. particularly, we describe the relations and arguments in a bottom-up manner, providing the llms with insight into logical relations from a local to global perspective. we then concatenate the textualized tree with the instruction prompt, and input them into llms as a hard prompt. the second considers tree-based soft prompt, where we derive a relation-aware tree embedding. specifically, we design relation-specific encoders to process each type of relation and incrementally derive the tree embedding from bottom up to the root node. we then insert the tree embedding into llms as a soft prompt for further tuning. experiments on benchmark datasets across various domains and genres validate that our approach based on logical structure tree effectively improve precision and recall for both fallacy detection and fallacy classification tasks. our main contributions are summarized as follows: habernal et al. (2017) developed a software that deals with fallacies in question-answering. sheng et al. (2021) investigated ad hominem fallacy in dialogue responses. habernal et al. (2018) explored the ad hominem fallacy from web argumentations. stab and gurevych (2017) recognized insufficient arguments in argumentation essays. goffredo et al. (2022) categorized fallacies in political debates. nakpih and santini (2020) focused on fallacies in legal argumentations. musi et al. (2022) researched fallacies about pandemics on social medias. (alhindi et al., 2022) proposed a multi-task prompting approach to learn the fallacies from multiple datasets jointly. jin et al. (2022) proposed a structure-aware method to classify fallacies. different from jin et al. 2022 that masked out content words to form a sequencebased pattern, our paper proposes a tree-based hierarchical logical structure to unify both relation connectives and content arguments together. logical reasoning abilities of large language models are gaining increasing research attention @xcite @xcite @xcite @xcite . @xcite combined large language models with first-order logic. pan et al. (2023); zhang et al. (2023) empowered large language models with symbolic solvers. pi et al. (2022) presented an adversarial pre-training framework to improve logical reasoning. zhao et al. (2023) incorporated multistep explicit planning into the inference procedure. jiao et al. (2022) proposed a contrastive learning approach to improve logical question-answering. different from these previous work, we particularly focus on logical fallacy reasoning, aiming to detect and classify fallacies. misinformation refers to the unverified or false information (guess and lyons, 2020; @xcite @xcite . misinformation detection was studied for years, such as fake news @xcite @xcite , rumor @xcite , satire @xcite , political bias @xcite @xcite , propaganda @xcite @xcite . logical fallacies are often employed within misinformation to present invalid claim as credible, facilitating the spread of misinformation @xcite @xcite . developing automatic models to detect logical fallacies can also benefit the identification and mitigation of misinformation. ## logical structure tree the logical structure tree consists of relation connectives as non-terminal nodes, and textual arguments as terminal nodes. the relation connectives serve as parent nodes, and the two corresponding arguments are linked as left and right children nodes. figure 1 illustrates examples of the logical structure tree. the logical structure tree is constructed in an unsupervised manner, guided by the constituency tree and a taxonomy of connectives complied for ten common logical relations. ## logical fallacy reasoning we further design a framework to incorporate the logical structure tree into llms for fallacy detection and classification. this framework consists of two main components. the first is textualized tree, where we convert the logical structure tree into natural language descriptions, and feed it into llms as a hard text prompt. the second is tree-based soft prompt, where we derive a relation-aware tree embedding, and insert it into llms as a soft prompt for additional tuning. the hard and soft prompts are complementary: the hard prompt enriches the instruction with logical structure information, while the soft prompt facilitates direct tuning on tree embeddings. figure 2 shows an illustration. ## limitations we have compiled a set of connective words and phrases for the ten logical relations, as detailed in ## conclusion this paper detects and classifies fallacies. we propose a logical structure tree to explicitly represent and track the hierarchical logic flow among relation connectives and their arguments. we also design two strategies to incorporate this logical structure tree into llms for fallacy reasoning. extensive experiments demonstrate the effectiveness of our approach based on the logical structure tree.