Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities (2025)

Abstract: This study intends to systematically disentangle pure logic reasoning and text understanding by investigating the contrast across abstract and contextualized logical problems from a comprehensive set of domains. We explore whether LLMs demonstrate genuine reasoning capabilities across various domains when the underlying logical structure remains constant. We focus on two main questions (1) Can abstract logical problems alone accurately benchmark LLMs’ reasoning ability in real-world scenarios, disentangled from contextual support in practical settings? (2) Does fine-tuning LLMs on abstract logic problems generalize to contextualized logic problems and vice versa? To investigate these questions, we focus on standard propositional logic, specifically propositional deductive and abductive logic reasoning. We construct datasets for both reasoning types with four difficulty levels across 12 distinct domains based on the Wikipedia categorization in addition to those with purely abstract variables. Our experiments aim to provide insights into disentangling context in logical reasoning, the genuine reasoning capabilities of LLMs, and their generalization potential. Coda and data are available at https://anonymous.4open.science/r/ContextHub-957E .

## introduction large language models (llms) @xcite @xcite have demonstrated significant potential in reasoning capabilities across a variety of reasoning benchmarks @xcite @xcite @xcite @xcite , broadening their potential applications in fields such as psychology, education, and social * wenyue hua and kaijie zhu contribute equally. work done at rutgers university, new brunswick. correspondence at wenyuehua@ucsb.edu. sciences @xcite @xcite . the widespread use of llms necessitates rigorous evaluation of their reasoning abilities, particularly in context-rich scenarios that reflect real-world complexities. while assessments on abstract logical problems @xcite @xcite @xcite showcase llms' theoretical reasoning capacities, they do not entirely capture their practical utility in reallife applications where context drastically affects outcomes. conversely, focusing exclusively on context-specific tasks @xcite @xcite @xcite ) may conceal the fundamental mechanisms that empower llms to process and reason with information. thus, exploring the disparity between contextualized and abstract reasoning @xcite is vital for advancing llms and ensuring their effectiveness across different domains. to this end, we introduce contexthub, a benchmark designed to systematically and meticulously disentangle and evaluate the core reasoning capabilities of llms from the influences of contextual information. by leveraging a dual-assessment framework, contexthub compares llms' performance on identical logical constructs within both abstract and richly contextualized settings (a contextualized example can be seen in step 3 of figure 2 ). this approach not only identifies the significant impacts of context on reasoning but also provides a scalable and flexible methodology adaptable across various domains. contexthub aims to address two main questions: (1) evaluation disentanglement: how accurate and reliable are abstract logic problems versus contextualized problems in evaluating llms' reasoning abilities? we examine this by comparing llm performance across both problem types to understand context's role in reasoning. (2) fine-tuning we employ a dynamic graph-based construction method @xcite to generate formal logic templates at four levels of difficulty. these templates are then instantiated in 13 distinct domains, comprising 12 contexts with specific settings and 1 using purely abstract variables for comparison. every instantiated question undergoes a rigorous two-step quality control process to ensure reliability. we evaluate these datasets using a variety of llms, including gpt-4o, gpt-3.5-turbo, qwen series, yi-series, llama-2, and llama-3.1 series. in the fine-tuning phase, we compare three settings: models fine-tuned on abstract data only, on mixed contextualized data from all domains, and on single domain data. these settings enable us to investigate how abstract and contextualized logical problems affect llms' generalization abilities. our key findings are: (1) the relative performance of llms on abstract logic problems and corresponding contextualized logic problems is dependent on model size or general model performance. stronger models tend to perform better on abstract logic, while smaller models typically rely on contextual cues. (2) the domain of contextualization has a statistically significant impact on model performance, suggesting the choice of contextualization domain can affect the accuracy and reliability of llms for logical reasoning tasks. (3) the generalization power of abstract logic data is limited compared with that of contextualized logic data. this indicates that llms fine-tuned on contextualized logic data may be better equipped to handle a wider range of real-world logical reasoning tasks. ## related work evaluating the reasoning abilities of llms has garnered significant attention across various disciplines, from biomedical informatics @xcite @xcite and humanities @xcite @xcite to social sciences @xcite @xcite . research has predominantly concentrated on diverse logical reasoning tasks, including deduction, induction, and abduction, addressed through neural models @xcite @xcite @xcite . logicbench @xcite focuses on natural language logical reasoning questions. folio @xcite , ruletaker @xcite , and fld @xcite build logic questions based on deduction rules focusing only on validity without semantics. nevertheless, no existing benchmark offers a systematic, finegrained investigation into how additional contextual detail or scenario-specific variations can affect the stability and reliability of llms' logical reasoning-particularly in terms of how prior knowledge might bolster performance or, conversely, how the absence of such knowledge may lead to greater degradation than performance in a purely abstract context. while llms have demonstrated notable successes in certain reasoning tasks @xcite , their generalizable logical reasoning capabilities remain uncertain @xcite . there are also many logic reasoning benchmarks such as logicbench @xcite and fo-lio @xcite using contextualized language over logic templates. benchmarks such as ruletaker @xcite and fld @xcite build synthetic logic questions based on logic templates without coherence semantics. however, none of these benchmarks systematically examine the influence of contextual factors on reasoning performance or explore the stability of logical reasoning skills across diverse scenarios, including those that are purely abstract. valmeekam et al. @xcite ) argued that llms often struggle with common planning and reasoning tasks, which are typically straightforward for humans. similarly, wei et al. @xcite noted that while chain-ofthought (cot) techniques stimulate human-like thought processes, they do not necessarily indicate genuine neural reasoning. further illustrating these limitations, tang et al. @xcite reported that llama-2 predominantly relied on template matching for reasoning tasks and lacked the ability to generalize beyond learned logic rules, a challenge exemplified in their studies using symbolic trees and proofwriter. they questioned whether llms truly possess human-like inductive, deductive, and abductive reasoning capabilities. adding to this discourse, saparov and he @xcite introduced a synthetic question-answering dataset, prontoqa, to assess the logical reasoning abilities of llms. their findings indicated that while llms can correctly perform individual deduction steps, they struggle with complex scenarios requiring the exploration of multiple valid deduction pathways. ## benchmark construction this section demonstrates the construction process of contexthub on deductive logic and abductive logic. deductive reasoning infers a logically certain conclusion from general statements, whereas abductive reasoning hypothesizes the most likely explanation based on observed data. as illustrated in figure 1 , constructing instantiated logical reasoning benchmarks involves three steps: ## experiment our experimental setup defines 4 levels of difficulty for logical reasoning tasks, with tree depths of (2, 3, 4, 5) and a uniform width of 2 across nodes. the dataset comprises 10 deductive and 6 abductive formal logic templates at difficulty level 1, the maximum number of distinct templates of reasoning that can be generated at this level. we then include 40 deductive and 40 abductive reasoning formal logic templates at levels 2, 3, and 4, respectively. to ensure balanced data distribution, we assign equal counts of 'true', 'false', and 'n/a' truth values across all questions. the dataset spans 12 domains for contextualization, generating 5 unique instances per domain, resulting in 18, 240 total datapoints across various levels and types of logic. the experiments are structured into two phases: benchmarking and fine-tuning. in the benchmarking phase, we assess model performance across various domains and compare between contextualized and abstract logic, to explore whether llms consistently grasp the underlying logical structures. the fine-tuning phase uses the generated data points to investigate how different instantiation types (abstract vs. contextualized) and data domains influence model generalization. this setup allows us to scrutinize the effects of model scaling and domain specificity on performance. benchmarking we benchmark the logical reasoning performance of several advanced models, including gpt series, qwen-1.5 series @xcite , llama-2 series @xcite , llama-3.1 series @xcite , and yi-1.5 series @xcite . this evaluates whether models understand the underlying logic structures consistently, irrespective of contextual variations. we use an average weighted f 1 score for model evaluation, detailed in appendix ??. fine-tuning in the fine-tuning phase, we examine how different data types affect model generalization in logic reasoning. models are fine-tuned using three settings: (1) solely with abstract logic data to test base logical reasoning capabilities. (2) with a sample of contextualized data across all domains to evaluate generalization across varied contexts. (3) with contextualized data from single domains to investigate the impacts of domain specificity and diversity. we utilize models ranging from qwen1.5-0.5b to gpt-3.5-turbo. we leverage qlora @xcite for finetuning on open-source models. other relevant hyperparameters are: @xmath0, warmup @xmath1.01, learning @xmath2, weight @xmath3.01, lora @xmath4, lora @xmath5.05, lora @xmath6, batch @xmath7, accumulate gradient @xmath8. evaluation metrics to assess the reasoning capabilities of llms, we employ the average f1 score. the calculation of the average f1 score involves determining the average of the f1 scores for data points (d) that possess the same truth values. for datapoints with identical ground truth (gt) truth value t , the f1 score is computed by first ascertaining the true positive (t t p ), false positive(f t p ), and false negative(f t n ): the average f1 score for the entire dataset is calculated by determining the average of the f1 t scores for all possible truth values t . ## conclusion this paper provides a comprehensive investigation into the logic reasoning abilities of llms through the contexthub benchmark. our approach effectively separates logical reasoning from textual comprehension, enabling a focused analysis of models' reasoning capabilities. the study reveals that a model's performance on reasoning tasks is substantially influenced by the context and domain-specific variables involved. also, finetuning on instantiated data enhances the models' ability to generalize across various logic reasoning tasks, irrespective of the domain complexity or the diversity of domains.