C o T - RAG : Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models (2025)

Abstract: Chain-of-thought (CoT) reasoning boosts large language models’ (LLMs) performance on complex tasks but faces two key limitations: a lack of reliability when solely relying on LLM-generated reasoning chains and interference from natural language reasoning steps with the models’ inference process, also known as the inference logic of LLMs. To address these issues, we propose CoT-RAG, a novel reasoning framework with three key designs: (i) Knowledge Graph-driven CoT Generation, featuring knowledge graphs to modulate reasoning chain generation of LLMs, thereby enhancing reasoning credibility; (ii) Learnable Knowledge Case-aware RAG, which incorporates retrieval-augmented generation (RAG) into knowledge graphs to retrieve relevant sub-cases and sub-descriptions, providing LLMs with learnable information; (iii) Pseudo-Program Prompting Execution, which promotes greater logical rigor by guiding LLMs to execute reasoning tasks as pseudo-programs. Evaluations on nine public datasets spanning three reasoning tasks reveal significant accuracy gains—ranging from 4.0% to 44.3%–over state-of-the-art methods. Furthermore, tests on four domain-specific datasets demonstrate exceptional accuracy and efficient execution, underscoring its practical applicability and scalability. Our code and data are available at https://github.com/hustlfy123/CoT-RAG.

## introduction large language models (llms) have garnered significant attention in both academia and industry due to their exceptional performance in natural language processing (nlp) @xcite , such as machine translation @xcite ), text summarization @xcite and sentiment analysis @xcite . nonetheless, they exhibit notable limitations in complex tasks requiring arithmetic, * corresponding author commonsense, and symbolic reasoning @xcite . to overcome these challenges, chain-of-thought (cot) reasoning has been introduced, wherein llms explicitly generate intermediate reasoning steps prior to reaching a conclusion @xcite . several cot variants have emerged, including @xcite , @xcite , @xcite , and ps @xcite . although cot improves performance on multistep reasoning tasks for llms, existing methods still face two main challenges: ## methodology overview. to address the aforementioned concerns, we design cot-rag, a novel reasoning framework with three stages (figure 2 ). first, experts construct and input a coarse-grained decision tree that represents the reasoning logic of problems in a specific domain, where experts only need to build it once, then the llm transforms it into a knowledge graph for a deeper understanding of its internal logic in the knowledge graph-driven cot generation phase ( §3.1). next, during the learnable knowledge case-aware rag phase ( §3.2), users input query descriptions related to this domain and the llm extracts sub-descriptions to update the knowledge graph for accurately generating results in the next stage. third, the llm uses pseudo-program prompting execution ( §3.3) to process the updated knowledge graph and produces the ultimate result. specific algorithm demonstration and time complexity analysis are given in appendix a, and related notations are shown in appendix b, which displays significant differences between our decision trees and knowledge graphs w.r.t. their conventional notions. ## experimental results ## conclusion to address the issues of low reliability in reasoning chains generated solely by llms and poorer reasoning performance from natural language prompts compared with code prompts, we propose cot-rag, a novel reasoning framework with three key designs: (i) knowledge graph-driven cot generation, which introduces knowledge graphs to regulate llms' reasoning chain generation, thereby enhancing reasoning reliability; (ii) learnable knowledge case-aware rag, intergrating rag into knowledge graphs to retrieve relevant subcases and sub-descriptions for learnable information; (iii) pseudo-program prompting execution, which inspires llms to execute logically reasoning tasks in pseudo-programs. evaluations across nine public datasets demonstrate that cot-rag outperforms existing methods, particularly in four domain-specific datasets, validating its powerful cross-domain scalability.