PRIC o T : Principle Retrieval and Injection from Inference Successes and Failures for C o T Improvement (2025)

Abstract: In-Context Learning (ICL) approaches, such as Zero-Shot and Few-Shot prompting, allow Large Language Models (LLMs) to tackle reasoning tasks without additional fine-tuning. However, Zero-Shot prompting often struggles with more complex tasks, whereas Few-Shot prompting demands considerable manual effort and domain expertise to design effective prompts. Although existing work has attempted to alleviate these issues by extracting reasoning rules from carefully crafted, task-specific representative examples, creating or obtaining such examples can be impractical in real-world scenarios. In this paper, we propose a novel approach that enhances the inference accuracy by injecting reasoning principles extracted from QA data, without relying on representative Few-Shot exemplars. This offers a lightweight yet adaptive way to boost accuracy on complex reasoning tasks, while avoiding manual effort and the high exploration costs typical of prior methods. Experiments on benchmarks show that, using GPT-4o, our method outperforms similarity-based Few-Shot and Zero-Shot prompting methods on challenging benchmarks such as GPQA-diamond, achieving an absolute accuracy improvement of up to 2% in scenarios where carefully crafted Few-Shot examples are unavailable.

## introduction in-context learning (icl) enables large language models (llms) to improve their performance on a variety of tasks by simply providing relevant task instructions and examples without retraining or fine-tuning the model's internal parameters @xcite @xcite . two representative forms of icl are zero-shot prompting, which encourages reasoning based solely on general instructions (e.g., "let's think step by step") such as zero-shot chain of thought (cot) @xcite , and few-shot prompting @xcite , which provides a handful of task-specific examples, sometimes accompanied by their corresponding cot @xcite to guide the model toward more accurate answers. despite the effectiveness of these approaches, several challenges remain. (1) zero-shot prompting often struggles with complex tasks because it heavily relies on the model's inherent knowledge @xcite . (2) few-shot prompting typically requires careful design and selection of examples by an expert familiar with the task @xcite . whenever the task changes, practitioners must redesign prompts and examples, incurring effort. consequently, there is a growing demand for methods that can automatically construct prompts @xcite . existing methods can be broadly categorized based on the availability of representative few-shot examples-each setting introducing its own set of challenges. when few-shot examples are entirely unavailable, methods based purely on zero-shot prompting must be relied upon. a notable example is plan and solve @xcite , which enhances basic zero-shot chain-of-thought prompting by first generating a high-level plan before reasoning through the task. while such approaches avoid the need for exemplars, their performance is limited on complex tasks due to the lack of task-specific guidance. in settings where raw task data is available but representative few-shot examples are not identified, several methods-such as ape @xcite and dspy @xcite -have been proposed to automatically construct prompts using llms. these methods iteratively optimize prompts based on evaluation feedback. while effective, they often involve repeated trial-and-error few shots initial thoughts principles for avoiding mistakes: …to avoid similar mistakes, always ensure to verify calculations and check if the final answer aligns with the provided options. principles for achieving correct answers: …for similar problems, always determine the rates of each contributor, combine them, and use the reciprocal to find the total time. ## related work in this section, we first describe the methods and challenges involved in zero-shot and few-shot prompting. next, we discuss methods aimed at the automatic design of input prompts that include few-shot examples. we summarize the challenges of each method and describe their relationship with the proposed approach. ## proposed method in this study, we propose a novel approach consisting of two phases. first, in the training phase, qa examples from the training data are processed using plan and solve inference with an llm. the resulting reasoning processes are analyzed to extract common reasoning principles from both successful and failed cases. second, in the testing phase, these extracted principles are applied to test data to improve the reasoning process. figure 2 illustrates the system model of our approach, and figure 1 llm training phase testing phase a: a: q: q: p: p: step1: initial inferences step2: evaluate initial thoughts step3: extraction of reasoning principles llm questions in labeled data initial thoughts wrong thoughts correct thoughts principles to solve the questions principles to avoid the mistakes principles db step5: initial inferences step6: retrieval of principles step4: constructing a database of reasoning principles step7: re-inference p: the principle to solve the question the principle to avoid the mistake llm a question in unlabeled data the initial answer q: a: a: q: the initial thought llm a: q: the initial thought a: q: p: a: output shows sample input and output prompts for the baseline methods (zero-shot prompting, few-shot prompting) and for our method (ours-pfs). ## experiments and results to validate the effectiveness of our method, we conducted experiments in which the reasoning principles were extracted from each llm's inference process on training data, and then applied to correct the reasoning on test data. in these experiments, we primarily focus on two llms, gpt-4o and gpt-4o-mini @xcite , which are representative of the available models and widely used in practice. to measure text similarity, each text is converted into an embedding representation via the sentence transformer (all-minilm-l6-v2) (reimers and gurevych, 2020), and the cosine similarity is calculated. we evaluate the baseline methods (zero-shot, few-shot, and plan-solve) and our proposed approach (ours-pf, ours-ps, ours-pfs) under these models. ## conclusion in this study, we proposed a novel in-context learning approach that applies reasoning principles extracted from llm inferences on training data without relying on task-specific few-shot examples to improve reasoning on test data. experimental results demonstrated that our method achieved an accuracy improvement of up to 2% on complex reasoning tasks such as gpqa compared to zero-shot, plan and solve, and few-shot methods. moreover, the effectiveness of our approach on both gpt-4o and gpt-4o-mini suggests its potential applicability to a wider range of llms. in future work, we plan to further investigate the generality of our method by applying it to diverse tasks and models, including the reasoning trajectories and strategies of ai agents involved in complex decision-making processes, and to refine our approach for even more robust correction of reasoning processes. an important future direction is to explore whether the required amount of training data can be further reduced for tasks with a limited variety of reasoning principles. in addition, verifying the potential for domain transfer remains an open challenge, and we aim to evaluate how well the extracted principles generalize across different domains.