CHECKWHY : Causal Fact Verification via Argument Structure (2024)

Abstract: With the growing complexity of fact verification tasks, the concern with “thoughtful” reasoning capabilities is increasing. However, recent fact verification benchmarks mainly focus on checking a narrow scope of semantic factoids within claims and lack an explicit logical reasoning process. In this paper, we introduce CHECKWHY, a challenging dataset tailored to a novel causal fact verification task: checking the truthfulness of the causal relation within claims through rigorous reasoning steps. CHECKWHY consists of over 19K “why” claim-evidence- argument structure triplets with supports, refutes, and not enough info labels. Each argument structure is composed of connected evidence, representing the reasoning process that begins with foundational evidence and progresses toward claim establishment. Through extensive experiments on state-of-the-art models, we validate the importance of incorporating the argument structure for causal fact verification. Moreover, the automated and human evaluation of argument structure generation reveals the difficulty in producing satisfying argument structure by fine-tuned models or Chain-of-Thought prompted LLMs, leaving considerable room for future improvements.

## introduction fact verification is a crucial debunking task that entails verifying the truthfulness of claims by cross-referencing them with reliable evidence drawn from established resources @xcite , which prevents the proliferation of erroneous information online and fosters public trust @xcite . however, with the multi-step reasoning capability in fact verification models remaining uncertain @xcite @xcite , existing research reflects the deficiency in in-depth understanding of the explicit reasoning mechanisms when performing inference on multi-hop evidence. this prompts us to develop a strong benchmark that incorporates the interpretable "thought" process to assess the logical reasoning capabilities of models. currently, substantial progress has been made on common fact verification benchmarks, e.g., hover @xcite , feverous @xcite , and scitab @xcite . nevertheless, existing resources have inherent limitations. classic datasets primarily focus on verifying the semantic factoids of "who", "what", "when", and "where" within the claim @xcite . for example, verifying the claim "john lennon was born before the astronaut who drank the first coffee in space." can be decomposed into verifying factoids such as "who, and when the astronaut drank the first coffee in space." and "when the man was born.". however, these semantic factoids can be answered individually by straightforward factoidsmatching between the claim and distinct independent evidence @xcite , e.g., word overlapping or proof matching @xcite , thereby limiting its significant potential to summarize the correlational evidence with "thought" steps. heretofore, noticeably absent in prior datasets are "why" claims: containing causal relations that need to be verified. these claims prompt for not simple factoid matching, but an explicit logical reasoning path for verification @xcite . more specifically, figure 1 presents a "why" claim featuring a cause-effect pair where "military crises" causes the "decrement of the purity of denarius silver.". verifying such causal relations is quite challenging, which necessitates complex logical reasoning and context information beyond claim: the military crisis confronting the empire forced marcus aurelius to reduce the purity of the denarius silver. label: supports ## preliminaries our checkwhy framework is inspired by the argument structures theorized in the literature on logical theory @xcite @xcite . thus, we begin by outlining the standard argumentation structure and then present a concise overview of our argument structure. ## the checkwhy dataset we adopt a human-model collaboration annotation approach to construct our checkwhy, as shown in figure 2 , which including data preparation ( § 3.1), generation with llms ( § 3.2), and manual validation ( § 3.3). ## conclusion in this paper, we introduce checkwhy, a challenging dataset and benchmark built around a novel causal fact verification task, annotated by a humanmodel collaborative approach. by drawing inspiration from the logical theory, we incorporate the argument structure to represent the explicit logical reasoning process when assessing the causal relation within the "why" claim, which may prove valuable for developing multi-step reasoning skills across various scenarios. extensive experiments on various state-of-the-art models validate the importance of incorporating argument structures and the difficulty of generating them. with the baselines achieving limited results, we believe that check-why is a challenging yet attractive benchmark for the development of fact verification systems.