Assessing the Reasoning Capabilities of LLM s in the context of Evidence-based Claim Verification (2025)

Abstract: Although LLMs have shown great performance on Mathematics and Coding related reasoning tasks, the reasoning capabilities of LLMs regarding other forms of reasoning are still an open problem. Here, we examine the issue of reasoning from the perspective of claim verification. We propose a framework designed to break down any claim paired with evidence into atomic reasoning types that are necessary for verification. We use this framework to create RECV, the first claim verification benchmark, incorporating real-world claims, to assess the deductive and abductive reasoning capabilities of LLMs. The benchmark comprises of three datasets, covering reasoning problems of in creasing complexity. We evaluate three state of-the-art proprietary LLMs under multiple prompt settings. Our results show that while LLMs can address deductive reasoning prob lems, they consistently fail in cases of abductive reasoning. Moreover, we observe that enhancing LLMs with rationale generation is not always beneficial. Nonetheless, we find that generated rationales are semantically similar to those provided by humans, especially in deduc tive reasoning cases.

## introduction large language models (llms) have shown remarkable proficiency in complex tasks where reasoning capabilities, such as logical deduction and semantic comparison, are paramount. notable examples include solving mba exams @xcite , passing professional medical tests @xcite , performing quantitative reasoning @xcite , and communication games @xcite @xcite . however, there is ongoing debate about whether such proficiency is due to llms manifesting reasoning capabilities or rather pattern matching and semantic similarity via memorization. for example, earlier claims that llms posses theory of mind (tom) capabilities @xcite were shown to be inaccurate @xcite . in particular, despite appearing to manifest some form of tom capabilities, llms mostly rely on shallow heuristics and spurious correlations @xcite . additionally, preliminary observations of emergent reasoning capabilities @xcite were subsequently attributed to metric choice @xcite , in-context learning @xcite , and shortcuts @xcite . these findings motivate the need for further research on the reasoning capabilities of llms, especially in high-stake real-world applications, where research on this topic is in its infancy. a notable example is fact-checking, where llms are considered to hold great potential for increased productivity even if at the same time they also facilitate bad actors in the proliferation of misinformation @xcite verifying information is challenging since models require both accurate veracity classification and strong rationale generation to be effective @xcite . it is thus essential to understand the reasoning capabilities and limitations of llms in the context of fact-checking. in particular, we extend the current discussion around the reasoning abilities of llms, focusing on their ability to verify real-world claims. in this work, we first propose a framework for breaking down complex claims into atomic reasoning steps. the motivation behind this is the lack of uniform terminology around reasoning evaluation. most prominent evaluation datasets for reasoning are based on mathematics and coding, which involve deductive reasoning, even though the use of deduction is not made explicit @xcite . our framework is rooted in existing philosophy literature concerning logical reasoning that aligns well with nlp @xcite . we use our framework to create reasoning in evidence-based claim verification (recv), the first reasoning benchmark for claimverification. the benchmark comprises three datasets, curated from existing resources targeting different domains: vitaminc @xcite from wikipedia, climate-fever @xcite ) from online claims and wikipedia, and phemeplus @xcite from rumours circulating on social media and associated evidence from news articles. the claims involve increasing levels of complexity as we move from vitaminc to phemeplus, often requiring deductive and/or abductive reasoning. we use recv to evaluate three state-of-theart proprietary llms that have shown impressive performance on various reasoning and language benchmarks @xcite . these models are: claude v3 sonnet @xcite , @xcite , and @xcite . in particular, we prompt models with and without chain-of-thought (cot) @xcite rationale generation to assess if and how the latter influences reasoning. in alignment with previous work @xcite @xcite we find that llms are capable of deductive reasoning. however, they consistently fail at claim verification when presented with evidence that requires abductive reasoning. furthermore, we observe conflicting results when prompting llms with cot strategies. in particular, cot leads to performance improvements for simple claim verification as in vitaminc, but is detrimental in the case of complex claims such as those found in climate-fever and phemeplus. lastly, we carry out a qualitative analysis of generated rationales and observe high semantic similarity with human explanations, especially in deductive reasoning cases. in summary, we make the following contributions: ## terminology to avoid ambiguity, we provide a brief overview of the main terminology used in the paper. we first introduce fact-checking and rumour verification specific terminology. we define a claim as a checkworthy statement, i.e., a piece of information that has to be verified. similarly, a rumour is a widely circulating claim of unknown veracity @xcite ). an evidence is a piece of information, here textual, retrieved from a document, that can be used to verify the veracity of a claim. we also provide an overview of reasoningspecific terminology. we denote conclusion to be a statement that derives from the direct elaboration of some observations. an observation is a piece of factual information. in the case of a reasoning task for claim verification, observations coincide with evidence, as reasoning is performed on factual information with the intent of verifying a given claim. lastly, we denote explanations as natural language descriptions of the reasoning process that leads to a conclusion from a given set of observations. ## recv logical framework reasoning is often used interchangeably to denote critical thinking, decision-making, and logical reasoning. following @xcite and @xcite , we define reasoning as the process of logical steps that result in some form of decision-making or conclusion. thus we define reasoning as a series of inference steps linking claims and evidence to reach a conclusion. in particular we consider that reasoning consists of the interplay of three interrelated components: types, processes, and tasks. this is the basis of our recv framework. reasoning types are different forms of logical inference that we can use to reach a conclusion from a set of observations or premises. we distinguish between atomic and compound reasoning types. atomic types denote basic forms of logical inference and include deductive, abductive, inductive, and analogical reasoning. a reasoning task is any task that requires multiple reasoning types, often in complex interaction with each other. for example claim verification is a composite reasoning task. a reasoning process is the method of interaction between reasoning types or even tasks within complex reasoning tasks. notable examples of reasoning processes are multi-hop or multi-step inference, where individual steps or hops can be of different reasoning types. in this paper we focus particularly on atomic reasoning types. ## methodology we discuss our methodology for reasoning in claim verification. we first showcase the application of our recv framework and then motivate our focus on deduction and abduction via a preliminary study. ## recv benchmark we discuss the creation of recv, in particular, our sample selection strategy and data annotation. see appendix b for details regarding the three datasets. ## claim verification with llms our objective here is to assess the capabilities of llms in performing deductive and abductive reasoning to determine the veracity of a claim. setup: we formulate claim verification as a prediction task. given a claim-evidence pair, we prompt llms to predict whether the evidence supports or refutes the claim (figure a2 (bottom)). we consider two settings: no-exp and exp. in no-exp, we prompt llms to predict the claim veracity without any rationale generation. in exp, we first prompt llms to produce a rationale and then use the generated information to predict claim veracity. for each setting, we consider two different prompt strategies: zero-shot (zs), and manual chain-of-thought (m-cot) @xcite . in addition, in exp, we also consider zero-shot chain-of-thought (zs cot) @xcite . zs cot was applied only under exp as rationale generation is integral to zs cot prompting. in all the prompts, we provide dataset specific personas and instructions in the system prompt and cot examples in the user prompt. we report the prompts in appendix f. ## explanation evaluation providing reasonable explanations to support predicted veracity labels is a crucial aspect of claim verification systems. in particular, an automated system needs to be both convincing and trustworthy to convince users in practice @xcite . therefore, we evaluate the llms generated rationales in the exp setting. this is crucial considering that llms tend to hallucinate @xcite and be self-contradictory at times @xcite . we randomly selected 100 samples from each dataset in recv and compared generated rationales against those provided by human annotators. we note that the provided human rationales explain the reasons for the chosen veracity and do not explicitly have any mention of why the given rationale corresponds to a reasoning type. in particular, the main goal of this study is to evaluate the models' reasoning capabilities through implicit measurement and without biasing the models with the mention of reasoning modes. this is also equivalent to how models are prompted, where we do mention reasoning types. we instruct a third annotator to evaluate the quality of provided human rationales. see appendix g for more details. we restricted sample selection to those where at least three models predicted wrong veracity labels. we follow @xcite and compute factual consistency (fc), evidence appropriateness (ea), bartscore @xcite , and perplexity (ppl) to assess the quality of the generated explanations. we provide additional details about metrics in appendix h. ## findings we discuss the main findings of our work, including task complexity, the effectiveness of prompting strategies, and rationale generation. reasoning and task complexity. our results show that abductive reasoning is consistently more challenging than deductive reasoning. in particular, the performance gap between the two cases is around three times on average. this is mainly motivated by llms failing in performing uncertainty reasoning, often leading to erroneous assertive conclusions. nonetheless, this is not the only issue that makes recv challenging; task complexity plays a crucial role in reasoning performance. for instance, phemeplus represents a more complex setting than vitaminc where news articles can contain extensive amount of information compared to wikipedia pages. as shown in our qualitative analysis, llms tend to focus only on specific parts of input claim-evidence pairs, leading to suboptimal performance. for example, in the following claim from vitaminc, claim: peking university is a unitary sovereign state that's located in east asia. evidence: peking university abbreviated pku is a major chinese research university located in beijing and a member of the c9 league. the veracity of is deductively refuted. however, all the models labelled this pair as evidence supporting the claim. the rationale provided was that china was a sovereign country, ignoring the claim completely. this shows the over-reliance on specific parts of the evidence by the models while ignoring others. overall, our findings suggest that llms' reasoning capabilities are domain and task dependent. thus, we believe recv represents a valuable resource to assess reasoning capabilities since it covers a wide spectrum of settings concerning claim verification. prompting settings and strategies our experiments show that prompting strategies like zs cot and m-cot do not lead to systematic performance improvements, but are rather specific to datasets (e.g., vitaminc) and models. these results align with recent findings about cot being beneficial mainly for math-and code-related tasks @xcite . this is likely derived by divergent reasoning paths within the models during inference that lead to reduction in performance @xcite @xcite . fur-thermore, we also observe that internal alignment can hinder reasoning capabilities when it comes to abductive reasoning. models are averse to provide predictions when evidence is incomplete. yet abductive reasoning is often required for more complex tasks such as legal reasoning, just in time factchecking, and other diverse forms of composite reasoning tasks. hence, in order to achieve good results on these type of reasoning tasks, llms need to improve in the direction of abductive reasoning. explanation quality our evaluation of generated explanations shows that these are on average consistent with human rationales. in particular, zs cot rationales are more convincing due to their verbosity, whereas m-cot rationales are more concise. moreover, we observe that rationales generated for abductive reasoning cases resemble assertions as models disprefer generating uncertain rationales. nonetheless, considering that our results are limited to macro performance results and given the limited number of abductive cases, we believe our estimates to decrease as dataset size increases. we leave a fine-grained analysis on generated rationales concerning an extended version of recv as future work. llms for claim verification. early work with llms focused on verifying simple facts @xcite . more recently, llms for claim verification have been augmented with external knowledge @xcite , promptbased reasoning @xcite @xcite , claim decomposition for fine-grained search into text chunks @xcite or firstorder logic terms @xcite , and dataaugmentation @xcite . a close work to ours is @xcite . contrary to @xcite , we evaluate models based on real-world claims and evidence. additionally, in @xcite , models are prompted to generate outputs using specific reasoning modes. this likely implicitly biases the models and increases performance that might not measure proper understanding of the task. by contrast, we evaluate the models by de-coupling these implicit signals and base our evaluation on understanding of the tasks. lastly, we also provide a framework for breaking down claim-evidence pairs into different atomic reasoning types. while llms have been extensively applied in fact-checking, the question of which reasoning capabilities are needed to verify claims remains unaddressed. thus, we are the first to propose a reasoning benchmark for claim verification. although all three datasets used in our study can be used for fact checking, we do not explicitly cast our problem as improving fact checking performance. for instance, @xcite evaluates the models' fact checking capabilities based on models' internal knowledge using claims only. by contrast, we provide the model with claim and evidence pairs. therefore, our focus centers more around the models' capabilities of understanding the evidence and implicitly reasoning to come to a conclusion in order to verify a claim. other examples are @xcite , @xcite , and @xcite , where a significant difference with our work is our choice of claims. in particular, all our claims are realworld claims and evidence. additionally, we focus on news articles as we build on the phemeplus dataset rather than on wikipedia sources. furthermore, we also provide a reasoning framework to facilitate the decomposition of claims into atomic reasoning types. ## conclusion we propose a novel extendable logical reasoning framework for deconstructing claim-evidence pairs into reasoning steps, required to determine the veracity of a claim. we use our framework to create recv, the first reasoning benchmark for claim verification focussed on deductive and abductive reasoning. our results show that llms notably struggle with abductive reasoning, while performing better in deductive cases. our findings show that llms reasoning capabilities are domain and task dependent. in particular, no specific prompting strategy, including rationale generation, is systematically beneficial across all datasets and models. nevertheless, rationales generated by llms for deductive reasoning are on average consistent with human ones. overall, these results highlight that recv represents a challenging reasoning setting for llms and further research is required to reach satisfying performance.