The Impact of Reasoning Step Length on Large Language Models (2024)

Abstract: Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the rationale reasoning steps within CoT demonstrations, while keeping all other factors constant. We have the following key findings. First, the results indicate that lengthening the reasoning steps in prompts, even without adding new information into the prompt, considerably enhances LLMs’ reasoning abilities across multiple datasets. Alternatively, shortening the reasoning steps, even while preserving the key information, significantly diminishes the reasoning abilities of models. This finding highlights the importance of the number of steps in CoT prompts and provides practical guidance to make better use of LLMs’ potential in complex problem-solving scenarios. Second, we also investigated the relationship between the performance of CoT and the rationales used in demonstrations. Surprisingly, the result shows that even incorrect rationales can yield favorable outcomes if they maintain the requisite length of inference. Third, we observed that the advantages of increasing reasoning steps are task-dependent: simpler tasks require fewer steps, whereas complex tasks gain significantly from longer inference sequences.

## introduction today, the advent of large language models (llms) and their advanced prompting strategies *equal contribution i nput accuracy 1 accuracy 2 accuracy 3 . . . . . . . . . reasoning step n . . . . . . . . . . . . k more reasoning steps has marked a significant progression, especially in classical nlp tasks @xcite @xcite @xcite . a key innovation among these is the chain of thought (cot) prompting technique @xcite @xcite , known for its efficacy in multi-step problem solving. this technique, reflecting human sequential reasoning, has shown remarkable effectiveness in various challenges, including crossdomain, length-generalization, and cross-lingual tasks. the cot approach, with its logical, step-bystep methodology, offers crucial interpretability in complex problem-solving scenarios. interestingly, wang et al. found that even incorrect but coherent rationales can improve reasoning performance, highlighting the value of logical continuity @xcite . building on this, fu et al. introduced complexity-based prompting, significantly improving accuracy and setting new benchmarks @xcite . this research further explores the relationship between the length of reasoning steps and the accuracy of conclusions, deepening our understanding of effective problem-solving in nlp. despite its promising results, the research community has yet to reach a consensus on the precise mechanics of how and why cot and its variations function effectively. this knowledge gap means that enhancing cot performance is still a field of exploration, largely reliant on trial-and-error approaches. there still lack established systematic methodologies for improving cot's effectiveness, leaving researchers to rely on conjecture and experimentation. this situation underscores a significant opportunity in the field: to develop a deeper, more structured understanding of cot's inner workings. such advancement would not only demystify the current process, but also pave the way for more reliable and efficient applications of this technique in various complex nlp tasks. in this study, we aim to investigate the hypothesis that the reasoning steps are the most crucial element in the effectiveness of cot prompts. this hypothesis stems from the observation that reasoning steps are a common element in both zero-shot and few-shot cot approaches. we conduct experiments to investigate this by maintaining strict control over variables. particularly, when incorporating new reasoning steps, we ensured that no additional knowledge was introduced. for the zeroshot experiments, we tweaked the initial prompt from "let's think step by step" to "let's think step by step, you must think more steps". then for fewshot setting, we design experiments that expand the rationale reasoning steps within cot demonstrations, while keeping all other factors constant. our first set of experiments evaluated the improvement in zero-shot and few-shot performance using auto-cot @xcite with our strategic intervention. subsequently, we assessed the accuracy of different methods across varying step lengths. we then extended our investigation to compare the effectiveness of our strategy on different llms such as gpt-3.5 and gpt-4. our findings revealed a significant correlation between the length of the reasoning chain and the capabilities of llms, within certain limits. intriguingly, when we introduced misleading information into the reasoning chain, the performance still showed improvement. this highlighted a pivotal insight: the key factor appears to be the length of the thinking chain rather than its accuracy. we have the following key findings, which we hope can help the community better improve cot performance. to validate this, we altered the initial prompt from "let's think step by step" to "let's think step by step, you must think more steps." this modification led to a noticeable enhancement in the reasoning abilities of the llms, particularly evident in datasets involving mathematical problems. ## related works in this section, we summarize two lines of literature that are most relevant to ours. ## analyzing methods in this section, we propose an analysis to examine the relationship between the reasoning steps and the chain-of-thought (cot) prompting performance. our central hypothesis is that the reasoning steps are the most critical component of cot prompts, enabling language models to apply more logical reasoning when generating responses. to test this, we design experiments that expand and compress the rationale reasoning steps within cot demonstrations, while keeping all other factors constant. specifically, we systematically vary only the number of reasoning steps, without introducing new reasoning content or removing existing reasoning content. we evaluate both zero-shot and few-shot cot prompts in the following sections. the overall experimental procedure is illustrated in figure 2 . through this controlled analysis, we aim to shed light on how cot influences the llm's ability to produce logically sound responses. ## experimental results we conduct experiments to answer the following research questions: ro1: what is the relationship of rational reasoning steps in demonstrations with cot performance? (section 4.2) ro2: can we confirm that the reasoning steps are the only factor that affects llm performance? (section 4.3) ro3: will compressing reasoning steps in few-shot demonstrations hurt llm performance? (section 4.4), ro4: can we observe the scaling phenomenon, i.e., the required reasoning steps be related to the size of the llms? (section 4.5) and ro5: what is the influence of questions in rationale towards the llm reasoning ability? (section 4.6) ## conclusions and future work in this work, we make a critical contribution to understanding and optimizing cot in llms, especially in the realm of complex reasoning tasks. our extensive research on the cot technique in natural language processing, particularly with large language models like gpt-3, gpt-3.5, and gpt-4, has led to key insights. we found a notable correlation between the length of the reasoning chain and the performance of these models. interestingly, longer reasoning chains improve model performance, even when they contain misleading information. this suggests that the chain's length is more crucial than its factual accuracy for effective problem-solving. these findings provide valuable guidance for refining cot strategies, highlighting the significance of reasoning length in complex nlp tasks. our next step is to analyze the long and short reasoning steps of llm inference via explaindeter-mineour objective is to ascertain whether longer inferential steps correlate with broader neuronal engagement. to illustrate this, we intend to use visualization techniques to analyze activation patterns between long and short reasoning steps.