Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoning (2025)

Abstract: While chain-of-thoughts (CoT) prompting has revolutionized how LLMs perform reasoning tasks, its current methods and variations (e.g, Self-consistency, ReACT, Reflexion, Tree-of-Thoughts (ToT), Cumulative Reasoning (CR) etc.,) suffer from limitations like limited context grounding, hallucination/inconsistent output generation, and iterative sluggishness. To overcome these challenges, we introduce a novel mono/dual-step zero-shot prompting framework built upon two unique strategies Chain of Evidences (CoE) and Evidence to Generate (E2G) . Instead of unverified reasoning claims, our innovative approaches leverage the power of “evidence for decision making” by first focusing exclusively on the thought sequences explicitly mentioned in the context which then serve as extracted evidence, guiding the LLM’s output generation process with greater precision and efficiency. This simple yet potent approach unlocks the full potential of chain-of-thoughts prompting, facilitating faster, more reliable, and contextually aware reasoning in LLMs. Our framework consistently achieves remarkable results across various knowledge-intensive reasoning and generation tasks, surpassing baseline approaches with state-of-the-art LLMs. For instance, (i) on the LogiQA benchmark using GPT-4, CoE achieves a new state-of-the-art accuracy of 53.8%, surpassing CoT by 18%, ToT by 11%, and CR by 9%; (ii) CoE with PaLM-2 outperforms the variable-shot performance of Gemini Ultra by 0.9 F1 points, achieving an F1 score of 83.3 on DROP. We release our prompts and outputs on these benchmarks as a new instruction tuning dataset for future research at Hugging Face .

## introduction retrieval-augmented or context-based generation serves as a mean for leveraging relevant information, empowering large language models (llms) to reduce the factual errors in their generation @xcite . however, despite the expansion in model and data size, llms struggle in contextual reasoning. this challenge is further ampli-1 https://huggingface.co/datasets/kagnlp/chain-of-evidences/ top: overwhelming long-text complexity leads models' failure even when it generates partially/fully correct reasoning (in green). bottom: ungrounded internal reasoning fails to grasp context, confusing "dod" (ungroundeded private org in red) vs walmart (in green). to bolster llm's reasoning capabilities, the chainof-thought (cot) prompting paradigm has emerged as a potent tool @xcite . subsequent methods, including self-consistency (sc; @xcite ), react @xcite , reflexion @xcite , tree of thoughts (tot; @xcite ), and cumulative reasoning (cr; @xcite ), generalize cot with various multi-objective, ensemble-based, or tool-augmented, and trial & error approaches but do not address the complexities of context-grounded or retrieval augmented generations (rag). we highlight two of their pivotal bottlenecks: (i) cot focuses solely on expanding steps without verifying hypotheses; (ii) excessively long retrieved text can lead to incorrect conclusions even with valid cot reasonings (example in figure 1 ). multi-step reasoning prompting has emerged as a promising alternative to traditional chain-of-thought (cot) approaches by decomposing complex problems cot and generic view of its (iterative) variants, (right) the e2g pipeline: in e-step our "generate ans with evidence and explanation" instruction extracts the rationales, coupled with the ans, grounded in the original context, then in g step we use the same instruction to derive the final answer solely from the "evidence and explanation" or along with the original context. our framework consists of two unique and real-time prompting strategies particularly tailored for long context reasoning. first, single-step chain-of-evidences (coe): to address the problem of ungrounded reasoning hypotheses, our designed prompt asks for specific thought sequences that are explicitly mentioned in the context. we call this series of intermediate reasoning steps, with directly extracted rationales from the given context, 'evidence' (as in human decision making). our key distinction from existing cot approaches is that instead of mere "thinking step-by-step" @xcite our prompt instruction asks for "step-bystep reasoning with explicit evidence and explanation". second, dual-step evidence to generate (e2g): to facilitate llms' answering the query properly even with retrieval augmented long-text contexts, we split the task into steps. in the first step (e), we adopt prompts similar to coe and generate both the answer & evidence . then in next step (g), we pass only the evidence as context for a second round of coe to llm. g step answer is predicted as the final answer. in contrast to complex long original context in e step, the evidence is a concise short text that directly answer the input query, g step is very fast, and simpler for the model to generate answer. in experiments with different llms, we show that our prompts consistently outperform existing approaches in a diverse set of eight context-driven tasks, including natural qa, complex multi-hop, long-form qa, fact checking, dialog generation, and reading comprehension tasks. since, even with such techniques, it is non-trivial to comprehend why and how this works and how to setup the prompt to function correctly, costeffectively, and robustly. to this end, we perform case studies, analyze different alternatives and reveal the strengths and weaknesses of our approach. we opensource our prompts and outputs on these benchmarks as a new instruction tuning dataset for future research. ## our prompting framework in this section, we develop our prompting framework for context-grounding and retrieval augmented longtext reasoning. we design two unique (mono/dual-step) prompts that does not require any exemplars and removes the hurdles of choosing multi-objective instructions. below we first present the prompt instruction for defining the objective for the target task (a.k.a system prompt), next the single-step prompting technique chain of evidences (coe) and finally dual-step evidence to generate (e2g) that uses coe twice. ## experimental setup we evaluate our prompting framework across eight context-intensive language tasks, requiring reasoning over given contexts, including those with distracting documents and retrieval augmentation for generation. ## main results arithmetic/logical context reasoning we evaluate our approach on the mrc tasks logiqa and drop, known for heavy arithmetic and logical reasoning complexities. logiqa tasks involve choosing among four options inferred from a small context, while drop tasks require answering questions with complex arithmetic computations from the context. 2 although reasoning in both tasks is largely independent, llms still need to align their reasoning with the context. our method, presented in for instance, in besides, in compare to the best performances of coe-long in these two tasks, f1 performances of coe-short are (logiqa 53.8 vs 51.8) and (83.3 vs 82.7) which validates our intuition that coe-long excels more when the task is based on arithmetic and logical reasoning. in addition, replacing the coe-long with coe-short, we observe a performance drop of around 2% & 0.6% in logiqa amd drop respectively-which validates our intuition that coe-long reasoning is both more context-driven and modular combining both the coe-short and cot. in simple math tasks (e.g., gsm8k), our method performs as good as cot as they are often context-free. multi-hop qa with distracting contexts we tackle more complex qa challenges, evaluating on the dis-q: how many field goals of at least 40 yards did neil rackers make ans: ## conclusion in this paper, we address the limitations of existing prompting frameworks for context-aware and retrieval augmented reasoning. we highlight the challenge of ungrounded reasoning rationales leading to potential hallucinations in llms. our novel framework introduces two new prompting methods to identify evidences in the context and generate answers based on that evidence. across various tasks, our approach empowers llms to deliver robust, and accurate. future work involves llm instruction fine-tuning using our prompted outputs.