Hybrid Intelligence for Logical Fallacy Detection (2025)

Abstract: This study investigates the impact of Hybrid Intelligence (HI) on improving the detection of logical fallacies, addressing the pressing challenge of misinformation prevalent across communication platforms. Employing a between-subjects experimental design, the research compares the performance of two groups: one relying exclusively on human judgment and another supported by an AI assistant. Participants evaluated a series of statements, with the AI-assisted group utilizing a custom ChatGPT-based chatbot that provided real-time hints and clarifications. The findings reveal a significant improvement in fallacy detection with AI support, increasing from an F1-score of 0.76 in the human-only group to 0.90 in the AI-assisted group. Despite this enhancement, both groups struggled to accurately identify non-fallacious statements, highlighting the need to further refine how AI assistance is leveraged.

## introduction the increasing prevalence of misleading information has created an urgent need to improve our ability to detect deceptive content. faulty reasoning (i.e., fallacies), which drives the spread of misinformation across various discourse domains, poses significant risks to informed decision-making and public discourse @xcite . some researchers even liken this problem to an epidemic @xcite . despite its importance, fallacy detection remains an understudied challenge, with current computational argumentation methods struggling to capture the complexity of deceptive arguments @xcite . van @xcite note that fallacies have received limited attention in both formal and computational argumentation research. recent studies have attempted to bridge this gap by constructing fallacy datasets and developing automatic fallacy identification methods. however, none have explored a hybrid intelligence (hi) approach, to the best of our knowledge. hi, which integrates human and ai capabilities, is considered highly promising but requires further empirical research to evaluate its effectiveness across different tasks and domains @xcite . the concept of hi is based on the view that while ai excels in data processing and pattern recognition, it lacks the creativity, empathy, and contextual understanding that humans bring to cognitive tasks. @xcite emphasize that hi systems are designed to leverage these complementary strengths, enabling humans and ai to learn from each other and improve over time. this continuous adaptation is particularly crucial for complex tasks such as fallacy detection. @xcite also note that while machine learning and hi are advancing toward realworld applications, the next step is to enhance their problem-solving capabilities. since hi is still a relatively new concept, further empirical research is necessary to assess its effectiveness across various domains and tasks. furthermore, although theoretical frameworks for hi exist, practical guidelines for integrating human intuition and creativity with ai's computational power remain unclear. this gap highlights the need for studies that not only develop theoretical insights but also provide empirical validation of the advantages of human-ai collaboration. this study aims to deepen our understanding of hi in fallacy detection by addressing several key questions: how can existing ai models be effectively integrated into a hybrid intelligent system to assist in this process? how do individuals identify fallacies with ai assistance compared to doing so without ai support? what challenges and limitations arise when employing hi systems for fallacy detection? by addressing these questions, this research seeks to evaluate the effectiveness of hi in fal-lacy detection and to provide practical insights into its real-world application. the findings have broader implications for decision-making in education, healthcare, and other fields. the results showed a substantial increase in performance with ai support, rising from an f1-score of 0.76 in the human-only study to 0.90 in the hi study. this highlights the potential of hi to enhance and support human cognitive abilities in complex tasks such as argumentation analysis. all resources developed in this paper are publicly available foot_0 . ## related work this section reviews recent advancements in computational argumentation and fallacy detection, introduces hybrid intelligence (hi) and its potential to enhance decision-making and problem-solving, and discusses the application of hi in improving fallacy detection and argument analysis. ## methodology the methodology of this study is primarily exploratory but incorporates experimental elements, including a between-subjects design inspired by @xcite , with two treatment groups: one where participants use an ai assistant (hi) and one without ai assistance. this design was chosen to effectively assess the impact of ai support on fallacy detection when combined with human reasoning. to evaluate participants' ability to identify logical fallacies, the study employed a range of materials and instruments, including surveys, a custom chatgpt-based chatbot, and various data analysis techniques. specifically, two user studies were conducted to collect detailed responses. in the human-only study, participants were asked to identify logical fallacies in given statements, with their performance and reasoning captured through openended responses. the hi study followed a similar structure but incorporated interactive elements, allowing participants to engage with a chatgptbased chatbot for fallacy detection. this setup enabled the evaluation of ai assistance in supporting reasoning and improving performance. figure 1 illustrates the overall methodology of this paper. task the fallacy detection task involved presenting participants with a statement as input and asking them to determine whether it was logically sound or fallacious. ## results this section presents the findings from the two studies conducted to evaluate the effectiveness of human-only fallacy detection and the impact of ai assistance through a chatgpt-based hi system. the first study established a baseline for human performance in identifying logical fallacies without ai support, while the second study explored how ai assistance could enhance performance and reasoning in fallacy detection. both studies utilized the same set of arguments, allowing for a direct comparison of results. the analysis focuses on performance scores, response patterns, and the influence of ai-generated hints, providing insights into the strengths and limitations of human reasoning and the potential of hi in improving fallacy detection. ## comparative analysis of human-only and hi studies this section compares the results of the humanonly fallacy detection study with the hybrid intelligence (hi) study, focusing on performance, additional variables such as hint usage, and the potential influence of the placebo effect. in addition, we report the results of a complementary experiment that directly evaluated large language models (llms) on the same fallacy detection tasks, providing a benchmark for comparison against both humanonly and human-ai studies. together, these analyses highlight the strengths and limitations of each approach and provide insights into the role of ai in enhancing human reasoning and decision-making. ## conclusion this study examined the effectiveness of hybrid intelligence (hi) in enhancing fallacy detection by comparing the performance of participants with and without ai assistance. the results demonstrated an improvement in performance when ai support was introduced, increasing from an f1score of 0.76 in the human-only study to 0.90 in the hi study. these findings highlight the potential of hi to complement and augment human cognitive capabilities, particularly in complex domains such as argumentation analysis. however, the study also revealed persistent challenges in distinguishing logically sound arguments, which remained difficult for participants in both conditions. this suggests a need for further advancements in critical thinking and argumentation training to better differentiate between logically sound and fallacious reasoning. the successful integration of a chatgpt-based ai assistant into the hi framework showcased the feasibility of ai-supported fallacy detection. nevertheless, certain limitations, such as occasional inconsistencies in ai behavior and variability in the quality of hints, indicate the necessity for further refinement in ai design and user interaction. despite these challenges, the study provides valuable insights into the potential applications of hi systems across various fields, including education and decision-making, where enhanced cognitive support is essential. future research should focus on expanding participant samples to include larger and more diverse populations, refining ai systems to improve reliability and usability, and exploring the long-term impact of hi on cognitive performance and reasoning tasks. by addressing these areas, the field can better harness the potential of hi to support and enhance human decision-making and problem-solving.