Missci: Reconstructing Fallacies in Misrepresented Science (2024)

Abstract: Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Such misinformation often misrepresents scientific publications and cites them as “proof” to gain perceived credibility. To effectively counter such claims automatically, a system must explain how the claim was falsely derived from the cited publication. Current methods for automated fact-checking or fallacy detection neglect to assess the (mis)used evidence in relation to misinformation claims, which is required to detect the mismatch between them. To address this gap, we introduce Missci, a novel argumentation theoretical model for fallacious reasoning together with a new dataset for real-world misinformation detection that misrepresents biomedical publications. Unlike previous fallacy detection datasets, Missci (i) focuses on implicit fallacies between the relevant content of the cited publication and the inaccurate claim, and (ii) requires models to verbalize the fallacious reasoning in addition to classifying it. We present Missci as a dataset to test the critical reasoning abilities of large language models (LLMs), that are required to reconstruct real-world fallacious arguments, in a zero-shot setting. We evaluate two representative LLMs and the impact of different levels of detail about the fallacy classes provided to the LLM via prompts. Our experiments and human evaluation show promising results for GPT 4, while also demonstrating the difficulty of this task.

## introduction false or misleading narratives spread rapidly on social media @xcite , posing challenges for non-experts in discerning credible information, and exceeding the capabilities of human fact-checkers (hfc). the need to support hfc has accelerated the research in automated fact-checking (afc) and related tasks @xcite ). yet, the real-world applicability of afc systems is limited due to the lack of trustworthiness @xcite , or their reliance on counter-evidence, which may not exist @xcite . often, misinformation distorts genuine information rather than creating new content @xcite . for example, the claim in figure 1 that "hydroxychloroquine is a cure for covid-19" contains a kernel of truth and relies on some content (referred to as accurate premise) of the cited study that found "chloroquine reduced infection of the coronavirus." however, further content from the study shows that it did not conduct human experiments (s 1 ) and focused on sars-cov-1 (s 2 ), different from the virus causing covid-19. only when knowing this additional information (s 1 and s 2 ) about the cited study, one can detect the applied fallacies (fallacy of composition and false equivalence). in this work, we focus on inaccurate claims that misrepresent scientific publications. we assume that the misrepresented publication is presented alongside the claim as a "proof" amplifying the claim's impact through increased perceived credibility. with this assumption, we can access the cited sources, which is essential for detecting fallacious reasoning and implements human verification strategies @xcite . our goal is to automatically outline the fallacious reasoning and explain why the claim is incorrect, a crucial aspect of debunking misinformation @xcite . earlier work on fallacy detection focused on surface-level fallacies like ad hominem or loaded language. more recent work @xcite also included logical fallacies like false cause or hasty generalization. yet, they did not adapt the task definition to account for the fact that these fallacies may need information beyond what is explicitly stated in the text. this hinders their applicability to real-world fallacies that rely on external information. to bridge this gap, we introduce missci, a new argumentation theoretical model for fallacious reasoning, accompanied by a new fallacy detection dataset derived from real-world misinformation. unlike prior work, we (i) treat inaccurate claims as the conclusion of a logical argument, encompassing all necessary information to detect the applied fallacies (green in figure 1 ). we (ii) formulate a distinct fallacy inventory drawn from literature to express fallacies when misrepresenting scientific publications. finally, inspired by @xcite , we (iii) explicitly verbalize the fallacious reasoning via premises (orange) that only implicitly exist based on the claim to reconstruct the fallacious argument. our focus lies on the reasoning abilities to reconstruct fallacious arguments, and we manually paraphrase the relevant publication content (green) based on hfc articles, rather than using the original misrepresented document. motivated by recent progress in zero-shot performance of large language models (llms) @xcite , we evaluate the reasoning abilities of two state-of-theart llms in reconstructing the fallacious reasoning on missci and define the task in a zero-shot setup, exemplified in figure 1 . given the claim, an accurate premise, and the publication contexts, the model must verbalize the fallacious reasoning and assign fallacy classes. our contributions are: previous work has focused on surface-level fallacies for propaganda detection @xcite @xcite , for online discussions @xcite , or for gamified settings @xcite . the addressed fallacies typically do not require information beyond what is stated explicitly in the text. other work targeted specific fallacies such as non sequitur in law (nakpih and santini, 2020) or ad hominem in social media @xcite . more similar to our work, some research @xcite @xcite focused on logical fallacies from the real world. yet, they neither verbalized the implicitly applied fallacies, nor considered the underlying sources beyond what is explicitly stated in the text. moreover, our task design to generate fallacious premises differs from implicit premise detection work @xcite in that the premises in missci are inherently invalid and linked to applied fallacy classes (see p 1 and p 2 in figure 1 ; orange). existing work on fallacy generation focused on data generation @xcite rather than on articulating applied fallacious reasoning within real-world fallacious arguments. scientific afc work @xcite @xcite @xcite ) considered external sources, like us, but did not identify and articulate the nuanced fallacies when concluding a claim from the cited study. detecting distortions in scientific communication is part of science communication research @xcite , where studies have examined exaggerations in news articles @xcite @xcite , analyzed reporting certainty in scientific publications @xcite , or quantified information mismatches between reported and actual scientific findings @xcite . in parallel work, @xcite quantify and compare the original paper with other reporting of their findings across fine-grained dimensions such as certainty or generalization. in addition to our distinct task setup, our problem space differs as we focus on harmful misinformation that comprises more severe distortions, which are not necessarily bound to a study's main findings. ## dataset our main annotator was a m.sc. student in biology, covering early pilot studies and post-annotation consolidation. additionally, two more m.sc. students, one in biology and one in linguistics, were employed during annotation. the annotators received a pay of 12.26 eur per hour. we used surge ai foot_0 as the annotation tool. weekly meetings involving all annotators and one of the authors were held throughout the project to provide feedback and to refine the guidelines as needed, in line with the recommendations of @xcite . to create missci, we (1) collected hfc articles and pre-selected links that may point to a misrepresented publication, (2) manually selected all links that pointed to a misrepresented publication, and ## experiments for each input (c, p 0 , s i ), comprising the incorrect claim, the accurate premise, and the publication context linked to a fallacy, the model must generate at least one fallacious premise pi together with the applied fallacy class fi . we only experiment in a zero-shot setting, since the dataset construction depends on high-quality hfc articles, which limits size and scalability. however, we separate 30 arguments as a validation split 5 to allow for a prompt selection without compromising the evaluation on the unseen test split, which comprises the remaining 154 arguments with 363 fallacious reasoning lines r i bridging different reasoning gaps. ## analysis 6.1 fallacy-level performance we assess the performances per fallacy class for the best prompts (d) in §5.3 for both llms in 3 , analyzed in §d.4, and primarily stems from a high recall for detecting this fallacy. for all other fallacy classes, gpt 4 achieves a substantially higher recall, leading to an overall higher performance in terms of f1-score, given the mostly similar precision across fallacy classes (cf. §e.2; figure 16 ). we observe the biggest difference for ambiguity and impossible expectations, which are frequently detected by gpt 4, but never by llama 2. the same was observed when prompting llms to predict fallacy classes applied by the gold fallacious premises (cf. §d.4; figures 13 & 14 ), suggesting that the differences were inherent to the llms. interestingly, both llms perform best on the most frequent fallacy classes despite no fine-tuning involved. ## discussion following suggestions in schlichtkrull et al. for responsible applications, we emphasize that data owners should ideally have domain expertise, recognizing that any system will inevitably be imperfect. we strictly did not ask the models to assign an overall rating of the claim's veracity. instead, we kept the user in the loop for decision-making and only assisted by outlining the fallacious discrepancies between the cited publication and the claim. clearly communicating the inaccuracies behind a claim is important for effective debunking @xcite and can help to increase digital literacy, which is important for building resilience against misinformation @xcite . while previous approaches taught digital literacy using serious games @xcite that require active participation, we envision a system that supports passive consumers of social media. ## conclusion and future work we introduced missci, a novel dataset to combat real-world misinformation that misrepresents scientific publications. we proposed a novel task formulation to automatically reconstruct the fallacious reasoning through logical arguments based on the cited publication's content. we showcased missci as a testbed for evaluating the reasoning abilities of llms. our experiments on two llms demonstrated the potential for reconstructing fallacious arguments. in future work, we plan to use missci with different llms, domains, and languages.