D eterm LR : Augmenting LLM -based Logical Reasoning from Indeterminacy to Determinacy (2024)

Abstract: Recent advances in large language models (LLMs) have revolutionized the landscape of reasoning tasks. To enhance the capabilities of LLMs to emulate human reasoning, prior studies have focused on modeling reasoning steps using various thought structures like chains, trees, or graphs. However, LLM-based reasoning still encounters the following challenges: (1) Limited adaptability of preset structures to diverse tasks; (2) Insufficient precision in exploiting known conditions to derive new ones; and (3) Inadequate consideration of historical reasoning experiences for subsequent reasoning steps. To this end, we propose DetermLR, a novel perspective that rethinks the reasoning process as an evolution from indeterminacy to determinacy. First, we categorize known conditions into two types: determinate and indeterminate premises, facilitating the transformation process. Subsequently, we leverage quantitative measurements to prioritize more relevant premises to explore new insights. Furthermore, we automate the storage and extraction of available premises and reasoning paths with reasoning memory, preserving historical reasoning details for subsequent reasoning steps. Comprehensive experimental results demonstrate that DetermLR surpasses all baselines on various logical reasoning benchmarks: LogiQA, ProofWriter, FOLIO, PrOntoQA, and LogicalDeduction. Compared to previous multi-step reasoning methods, DetermLR achieves higher accuracy with fewer reasoning steps, highlighting its superior efficiency and effectiveness in solving logical reasoning tasks.

## introduction the emergence of large language models (llms) has instigated a transformative wave within the realm of artificial intelligence @xcite . the series models of gpt @xcite @xcite and palm @xcite have exhibited remarkable proficiency in natural language reasoning, contributing to the advancement of research and applications of cognitive intelligence @xcite . however, even the current state-of-the-art (sota) llms still grapple with a key limitation: the lack of humanlike advanced reasoning skills to rationally analyze known conditions and draw conclusions @xcite . this leaves a substantial gap between llm-based reasoning and the cognitive process of human reasoning. to alleviate this limitation, existing studies employ enhanced prompt engineering techniques to guide llms in eliciting intermediate thinking steps to ensure reliable conclusions @xcite @xcite . building upon this foundation, recent works have focused on introducing more intricate reasoning structures, such as multiple chains @xcite , trees @xcite or graphs @xcite , to tackle increasingly complex reasoning tasks. however, llm-based reasoning continues to encounter three challenges: (1) limited adaptability of preset structures to diverse tasks: since the task complexity cannot be solely inferred from the problem context, relying on a certain preset structure to solve a variety of reasoning problems may create deficiencies in reasoning effectiveness or efficiency @xcite . this approach contrasts with human problem-solving techniques, which are not dependent on preset reasoning structures. ideally, the reasoning structure should be the result of a manual review after solving the problem. (2) insufficient precision in exploiting known conditions to derive new ones: the literature on human cognitive reasoning provides valuable insights and emphasizes the importance of integrating available information for informed decision-making @xcite @xcite . this motivates cumulative reasoning (cr) @xcite , which uses llms to iteratively generate new propositions based on available premises. however, cr still cannot approach the human thought process, as it relies on the random combination of existing premises without a well-defined criterion. @xcite inadequate consideration of historical reasoning experiences for future reasoning: previous works @xcite often overlook historical reasoning details, resulting in the lack of necessary information for subsequent phases. to address these challenges and augment llms to grasp more human-like advanced reasoning skills, we need to consider three key factors: (1) refine the formulation of the essence of the reasoning process; @xcite prioritize relevant premises for efficiently exploring new information; @xcite memorize historical reasoning details to guide the direction of the subsequent reasoning steps. to this end, we propose determlr, a novel reasoning framework to align llm-based reasoning more closely with human thinking. first, we formulate the logical reasoning process as an evolution from indeterminacy to determinacy. since premises exhibit varying descriptions and associations with the target, we initiate the reasoning process with premise identification to finely categorize premises into two distinct types: determinate and indeterminate. determinate premises are defined as simple statements, which can definitively contribute to conclusion derivation. conversely, indeterminate premises may contain complex rules governing the relationships among multiple propositions. regardless of the problem complexity, the reasoning process consistently involves the continuous accumulation of determinate information, steering the conclusion toward greater clarity. subsequently, human reasoning typically aims for a "breakingthrough" from known conditions to deduce new insights, indicating the necessity to distinguish the priority of premises. therefore, we propose quantitative measurements to facilitate premise prioritization, which involves identifying the most relevant premise to the conclusion and screening supplementary premises likely to interact with this primary premise. this guides llms to exclude irrelevant premises and focus on more pertinent information for premise exploration. furthermore, we introduce a reasoning memo-rization module to automate the storage and extraction of available premises and reasoning paths. in this way, historical reasoning details are preserved in the reasoning memory to update reasoning states, and they are incorporated into future reasoning steps to refer to inherent experiences and avoid repeating similar mistakes. to verify the capability of llms to engage in rigorous logical reasoning, we conduct extensive experiments on various challenging logical reasoning benchmarks: logiqa, proofwriter, folio, prontoqa, and logicaldeduction. the experimental results show that determlr achieves the best performance on reasoning accuracy, coupled with superior efficiency of requiring fewer steps than previous multi-step reasoning methods. notably, in more intricate tasks like logiqa, determlr exhibits even more pronounced advancements, mirroring human-like reasoning skills to a greater extent. our technical contributions to advancing llmbased reasoning can be summarized as follows: ## conclusion in this work, we propose determlr, a novel reasoning framework to align llm-based reasoning more closely resemble human cognitive reasoning. first, we propose a novel perspective that formulates the reasoning process as an evolution from indeterminacy to determinacy. second, we employ quantitative measurements for premise prioritization and exploration, allowing llms to prioritize premises more conducive to exploring new insights. furthermore, we introduce a reasoning memorization module to preserve historical details during iterative reasoning. experimental results show that determlr can achieve the highest accuracy while requiring fewer reasoning steps. notably, in more intricate tasks like logiqa, determlr exhibits even more pronounced advancements, mirroring human-like reasoning skills to a greater extent.