RESPROMPT : Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models (2024)

Abstract: Chain-of-thought (CoT) has impressively unlocked the reasoning potential of large language models (LLMs). Yet, it falls short when tackling problems that require multiple reasoning steps. This limitation arises from the complex nature of multi-step reasoning processes: later stages often depend not only on the immediately preceding step, but also on the results from several steps earlier. Such complexities indicate the reasoning process is naturally a graph. The almost linear structure of CoT, however, struggles to capture this complex reasoning graph. To address this challenge, we propose Residual Connection Prompting (ResPrompt), a new prompting strategy that advances multi-step reasoning in LLMs. The core of our idea is to reconstruct the reasoning graph within prompts. We achieve this by integrating necessary connections–links present in reasoning graph but missing in the linear CoT flow–into the prompts. Termed “residual connections”, these links can transform linear CoT into the complex reasoning graphs that multi-step problems entail. On benchmarks across math, sequential, and commonsense domains, ResPrompt demonstrates clear improvements in multi-step reasoning compared with CoT. Through extensive ablation studies and analyses, we pinpoint how to effectively build residual connections and also identify situations where it might be unnecessary.

## introduction recent advancements in scaling up large language models (llms) @xcite @xcite @xcite @xcite @xcite have not only significantly improved their performance but have also enabled entirely new "emergent ability" @xcite . one mile- @xcite . however, standard cot approach has proven to be less effective in addressing questions that involve multiple reasoning steps @xcite @xcite . in figure 1 , we demonstrate that both llama-65b @xcite and llama2-70b @xcite experience a notable decline in performance as the number of reasoning steps increases on the mathematical benchmarks gsm8k @xcite and aqua-rat @xcite . why is this the case? we hypothesize that in many multi-step reasoning processes, later stages rely not only on the immediately preceding step but also on results from several steps prior as prerequisites. this complex interdependence leads to the reasoning process in these multi-step questions essentially forming a graph structure, which we refer to as "reasoning graph". we show an example involving multi-step reasoning from gsm8k bench-question: tobias is buying a new pair of shoes that costs @xmath05 allowance a month. he also mows lawns and shovels driveways. he charges @xmath17 to shovel. after buying the shoes, he has $15 in change. if he mows 4 lawns, how many driveways did he shovel? ## related work we discuss three categories of related work: "in-context learning", "prompting-based reasoning", and "multi-step reasoning". due to space limitation, we provide a concise overview here and direct readers to appendix b for a comprehensive review. in-context learning. our work focuses on more structured prompting strategy, which is closely related to in-context learning @xcite . it refers to llms' capacity to adapt from a few exemplars without model parameter changes. as models grow and train on more data, they exhibit significantly amplified performance accorss many tasks @xcite @xcite , or even obtain new capabilities such as reasoning over complex questions. this phenomenon is recently termed "emergent ability" @xcite . prompting-based reasoning. llms, when ## conclusion we propose resprompt, a new prompting strategy to enhance multi-step reasoning in llms. our core idea is to reconstruct the complex reasoning graphs inherent in multi-step questions. to achieve this, we introduce "residual connection", which adds missing links to transform the linear cot prompts into graph-like structures. experiments demonstrate that resprompt significantly advances multi-step reasoning on llama family.