Faithful Logical Reasoning via Symbolic Chain-of-Thought (2024)

Abstract: While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rigid deducing rules. To strengthen the logical reasoning capability of LLMs, we propose a novel Symbolic Chain-of-Thought, namely SymbCoT, a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting. Technically, building upon an LLM, SymbCoT 1) first translates the natural language context into the symbolic format, and then 2) derives a step-by-step plan to solve the problem with symbolic logical rules, 3) followed by a verifier to check the translation and reasoning chain. Via thorough evaluations on 5 standard datasets with both First-Order Logic and Constraint Optimization symbolic expressions, SymbCoT shows striking improvements over the CoT method consistently, meanwhile refreshing the current state-of-the-art performances. We further demonstrate that our system advances in more faithful, flexible, and explainable logical reasoning. To our knowledge, this is the first attempt at combining symbolic expressions and rules into CoT for logical reasoning with LLMs. Code is open at https://github.com/Aiden0526/SymbCoT.

## introduction achieving human-like logical reasoning capabilities is crucial for realizing agi, which plays a pivotal role in enabling intelligent systems to engage in problem-solving, decision-making, and critical thinking. recently, llms @xcite have demonstrated unprecedented capabilities in semantic understanding, casting a beacon of hope toward achieving agi. further enhancing llms to achieve human-level question is the following statement true, false, or uncertain? descampe is in the six-way tie in the leaderboard of the 1992 du maurier classic. the context states that there was one six-way tie on the leaderboard and one person in the six-way tie was from belgium. descampe is from belgium and is on the leaderboard of the 1992 du maurier classic. therefore, it can be inferred that descampe is in the sixway tie in the leaderboard of the 1992 du maurier classic. evaluate the l ink u sing first-order logic f rom the step from ∃x (tie(x, sixway) ∧ from(x, belgium)) to tie(descampe, sixway) requires a premise directly stating descampe's involvement in the six-way tie, which is absent such as tie(descampe, sixway). however, s u c h information is not given. thus, the truth of the statement "tie(descampe, sixway)" remains unknown. ## related work recent achievements in reasoning research powered by llms have shown promising results @xcite , bringing llms closer to human-level reasoning capabilities due to their profound semantic understanding @xcite . among these, the cot series methodology @xcite has garnered increasing attention for its emulation of human discrete chain reasoning. by considering more intermediate steps and the rationales behind decisionmaking, cot has significantly enhanced overall reasoning performance on many downstream applications @xcite . subsequent technologies have introduced more advanced reasoning frameworks, incorporating mechanisms such as self-consistency and non-linear, multidimensional topological structures, e.g., tree-of-thought @xcite , graph-of-thought @xcite , and other variants @xcite . however, research has also highlighted limitations within cot due to its reliance on natural language rationales, which may not always be advantageous in certain scenarios. studies have found that representing cot's intermediate steps in a structured manner, reflecting the task's intrinsic structure, can bolster reasoning capabilities for specific tasks @xcite . for instance, using pseudo-code to describe intermediate reasoning processes has been shown to enhance outcomes in code generation tasks @xcite , while adopting mathematical equations for cot's steps has proven beneficial in solving mathematical problems @xcite . focusing on logical reasoning, it becomes evident that solely using natural language formats for intermediate reasoning steps inevitably leads to significant information loss, especially when tackling complex logical reasoning jobs. this paper, therefore, proposes a symbolic-oriented cot approach tailored for logical reasoning. logical reasoning @xcite , a paramount aspect of the reasoning domain, demands models that can precisely grasp and manipulate complex logical structures. previous works have explored rule-based @xcite and neural-based solving @xcite methods for interpreting symbolic representations. the latest trend involves integrating llms into the symbolic reasoning process @xcite . for example, logic-lm @xcite and linc @xcite consider using llms as translators to convert natural language into symbolic syntax such as fol, which is then processed by external reasoning tools to enhance reasoning performance. these approaches maintain that llms cannot parse symbolic expressions as reliably as external rule-based reasoners. nonetheless, merely utilizing llms as translators does not inherently enhance their logical reasoning capabilities. this work pioneers the development of the first symbolic cot specifically designed for logical reasoning, fully utilizing llms. in contrast to approaches like logic-lm and linc, our method demonstrates several advancements: first, external reasoners require strict formatting, where any translation error by llms can lead to failure in parsing and reasoning. our reasoning steps, facilitated by the llm, exhibit greater robustness against syntax errors. second, the entire reasoning process is conducted by the llm, providing rationales that ensure a more human-friendly explanation throughout. third, we propose a blend of symbolic forms and natural language expressions within the logical reasoning process, achieving precise reasoning calculations while fully interpreting implicit information inherent in natural language. finally, we introduce a plan-then-solve cot reasoning architecture and a verification mechanism, ensuring the faithfulness of the reasoning process. ## analysis and discussion we now delve into our system further and try to explore why it advances. ## conclusion this study presents the symbolic chain-of-thought (symbcot) framework, which innovatively integrates symbolic expressions and logical rules with cot prompting to improve the logical reasoning abilities of llms. specifically, based on the backbone llm, symbcot translates the natural language context into the symbolic format and then derives a step-by-step plan to solve the logical reasoning problem with symbolic logical rules, followed by a verifier to check the translation and reasoning chain. experimental results on 5 standard benchmarks with both fol and co symbolic expressions show that symbcot strikingly enhances the vanilla cot on logical reasoning.