Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot (2025)

Abstract: In-Context Learning (ICL) is an essential emergent ability of Large Language Models (LLMs), and recent studies introduce CoT to exemplars of ICL to enhance the reasoning capability, especially in mathematics tasks. However, given the continuous advancement of model capabilities, it remains unclear whether CoT exemplars still benefit recent, stronger models in such tasks. Through systematic experiments, we find that for recent strong models such as the Qwen2.5 series, adding traditional CoT exemplars does not improve reasoning performance compared to Zero-Shot CoT. Instead, their primary function is to align the output format with human expectations. We further investigate the effectiveness of enhanced CoT exemplars, constructed using answers from advanced models such as Qwen2.5-Max and DeepSeek-R1. Experimental results indicate that these enhanced exemplars still fail to improve the model’s reasoning performance. Further analysis reveals that models tend to ignore the exemplars and focus primarily on the instructions, leading to no observable gain in reasoning ability. Overall, our findings highlight the limitations of the current ICL+CoT framework in mathematical reasoning, calling for a re-examination of the ICL paradigm and the definition of exemplars.

## introduction as large language models (llms) continues to scale, llms exhibit emergent in-context learning (icl) capabilities @xcite , enabling them to perform target tasks by conditioning on a few exemplars without any additional parameter updates. furthermore, the use of chain-of-thought (cot) exemplars @xcite in icl guides models to reason step-by-step. this approach is commonly referred to as few-shot cot. kojima figure 1 : accuracy under different prompting settings on gsm8k (top) and math (bottom). we observe that the zero-shot setting consistently achieves strong performance, suggesting that the model may not attend to the cot exemplars. see section 5 for the full experimental results. @xcite further showed that simply appending the instruction "let's think step by step" can trigger multi-step reasoning even without exemplars, giving rise to the zero-shot cot paradigm, an overview of them is shown in figure 2 . existing research primarily focuses on how the quality, order, and number of exemplars influence icl performance, proposing various strategies for exemplar construction and selection to enhance model performance across different task settings @xcite @xcite . in addition, several studies have investigated the underlying mechanisms and influencing factors of icl from either theoretical or empirical perspectives @xcite @xcite @xcite . however, most of these strategies and experimental conclusions are based on earlier, weaker models. as foundation models become increasingly powerful, it is necessary to revisit a central question: in mathematical reasoning tasks, can cot exemplars still improve the reasoning performance of recent strong models? in this paper, we aim to investigate the actual role of cot exemplars in mathematical reasoning tasks. we conduct systematic experiments on two representative math reasoning datasets, gsm8k @xcite and math @xcite , using several recent open-source llms. we first identify a common evaluation bias in open-source evaluation frameworks @xcite in gsm8k, which significantly underestimates the performance of zero-shot cot, as discussed in section 4. after correcting for this issue, we compare few-shot cot with zero-shot cot prompting. our results show that recent strong models already exhibit strong reasoning capabilities under the zero-shot cot setting, and the primary role of few-shot cot exemplars is to align the output format with human expectations. subsequent analysis confirms that adding traditional cot exemplars does not improve reasoning performance (see section 5.1). inspired by recent advances in reasoning models with more sophisticated capabilities @xcite , we then examine the effectiveness of enhanced cot demonstrations constructed using answers generated by advanced models such as qwen2.5-max and deepseek-r1. experimental results indicate that, regardless of enhancement, models tend to ignore the content of exemplars in mathematical reasoning tasks and fail to acquire advanced capabilities such as self-reflection (see section 5.3). as a result in figure 1 , cot exemplars do not lead to improved reasoning performance in recent models. to summarize, our main empirical findings in mathematical reasoning tasks are as follows: cot prompting icl enables llms to perform tasks without fine-tuning @xcite , but it often falls short in complex reasoning scenarios. to address this, cot prompting (wei et al., 2022) introduces intermediate reasoning steps to guide model outputs. building on cot, researchers have proposed various extensions to enhance reasoning capabilities. for instance, tree-of-thought (yao et al., 2023) generalizes cot to tree-structured reasoning, while graph-of-thought (besta et al., 2024) further expands it to graph-based structures. the least-to-most framework (zhou et al., 2023) decomposes complex problems into simpler subproblems and solves them sequentially. ## experimental setup models to thoroughly validate our conclusions, we evaluate a variety of open-source language models, including the qwen2.5 series (ranging from 0.5b to 72b parameters) @xcite , the llama3 series (1b to 70b) @xcite , the gemma2 series (2b and 9b) @xcite , and ministral-8b @xcite . in addition, to examine the effectiveness of cot prompting on earlier and weaker models, we include llama2-7b @xcite and qwen-7b @xcite for comparative analysis. all models used in our experiments are instructiontuned variants. more details can be found in appendix a.1. datasets we focus on mathematical reasoning tasks and conduct experiments on two datasets of varying difficulty: gsm8k @xcite and math @xcite . to ensure accuracy, we perform inference and evaluation on the full test sets of both datasets and report the complete results. more details can be found in appendix a.2. environment and hyperparameters we utilize the open-source inference framework open-compass @xcite and vllm @xcite as the backend to run all experiments. notably, all experiments incorporate a cot instruction in the prompt: "please reason step by step, and put your final answer within \boxed{}." for reproducibility, all experiments are conducted using a fixed random seed of 42. notably, since greedy decoding is deterministic, the fixed seed does not influence the inference results under a fixed hard- ministral-8b llama-3.1-8b gemma-2-9b qwen-2.5-7b qwen-2.5-14b qwen-2.5-32b qwen-2.5-72b llama-3.3-70b 30 40 50 60 70 80 accuracy (%) settings complex topk fast-votek random dpp mmr zero-shot ministral-8b llama-3.1-8b gemma-2-9b qwen-2.5-7b qwen-2.5-14b qwen-2.5-32b qwen-2.5-72b llama-3.3-70b 70 75 80 85 90 95 accuracy (%) 8shot complex topk fast-votek random dpp mmr explora zero-shot ware setup. hence, we do not report the mean or standard deviation of the results. more details can be found in appendix a.3. ## exemplars help mitigate evaluation bias evaluation bias in gsm8k existing evaluation frameworks for gsm8k (e.g., opencompass @xcite , open-instruct @xcite ) typically extract the last number from model outputs as the predicted answer. however, in zero-shot cot prompting, answers are often enclosed in "\boxed{}" expressions. this mismatch leads to misjudgments during evaluation, as illustrated in figure 21 . to address this, we modify the evaluation script to extract the number inside \boxed{}, reducing artificially low accuracy caused by output-format misalignment. we consider this a form of evaluation bias that affects fair assessment, either due to oversight or simplification. ## cot exemplars can't improve reasoning ability of strong models the preceding sections have shown that the primary contribution of exemplars lies in aligning the output format rather than enhancing reasoning ability. however, since we previously used a fixed set of 8 exemplars, an open question remains: can exemplars improve the reasoning ability of recent llms if we consider different impact factors such as retrieval method, model's intrinsic ability and the quality of exemplars? ## discussion and conclusion in this paper, we investigate the role of cot exemplars in mathematical reasoning tasks with recent strong models within the icl paradigm. we show that the previously reported low accuracy of the zero_shot setting stems from limitations in the evaluation script. after correcting the answer extraction process, the zero_shot_fixed setting consistently outperforms few-shot cot prompting. our findings reveal that: (1) the primary function of exemplars is to align output format; (2) while exemplars benefit weaker models, they fail to enhance the reasoning ability of stronger models. we further explore enhanced cot exemplar settings and observe moderate improvements over traditional examples. however, ablation studies show that even with noisy or irrelevant exemplars, model accuracy remains stable, indicating that: (3) strong models rely more on prompt templates than on exemplar content. finally, attention visualizations support this conclusion by demonstrating weak attention to exemplar tokens. overall, our study highlights the limitations of current icl+cot paradigm in mathematical reasoning and calls for a reevaluation of the role of cot exemplars. we hope our work offers new insights and empirical grounding for future research. are existing evaluation frameworks reliable? as discussed in section 4, opencompass @xcite evaluates gsm8k performance by extracting only the final digit from model outputs. this evaluation imposes strict constraints on output format, potentially overlooking genuine reasoning ability. while such an evaluation may be suitable for measuring output format consistency, it can misrepresent a model's reasoning capabilities. hence, if the research goal is to evaluate reasoning rather than formatting, care must be taken to avoid evaluation-induced bias. we advocate that future studies place particular emphasis on the potential bias introduced by evaluation frameworks and carefully design experiments to ensure faithful assessment of model behavior.