D yna T hink: Fast or Slow? A Dynamic Decision-Making Framework for Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated emergent capabilities across diverse reasoning tasks via popular Chains-of-Thought (COT) prompting. However, such a simple and fast COT approach often encounters limitations in dealing with complicated problems, while a thorough method, which considers multiple reasoning pathways and verifies each step carefully, results in slower inference. This paper addresses the challenge of enabling LLMs to autonomously select between fast and slow inference methods, thereby optimizing both efficiency and effectiveness. We introduce a dynamic decision-making framework that categorizes tasks into two distinct pathways: ‘Fast,’ designated for tasks where the LLM quickly identifies a high-confidence solution, and ‘Slow,’ allocated for tasks that the LLM perceives as complex and for which it has low confidence in immediate solutions as well as requiring more reasoning paths to verify. Experiments on five popular reasoning benchmarks demonstrated the superiority of the DynaThink over baselines. For example, when we compared it to strong COT with self-consistency baseline on the complicated MATH dataset, DynaThink achieved more than 3% increase in accuracy with lower cost. The code will be made available upon publication.

## introduction llms @xcite @xcite have emerged as prominent foundation models for diverse applications due to their outstanding ability to understand and generate humanlike text. one notable attribute of llms is their capability of cot reasoning @xcite , where they can perform multi-step reasoning using only a few demonstrations. however, the performance of such simple and fast cot prompting is not satisfactory for complex reasoning problems, which often need more careful thought and analysis @xcite . to improve this, @xcite introduced a self-consistency strategy as a replacement for the previous straight-forward decoding method used in cot prompting. this self-consistency strategy involves trying out multiple reasoning paths and selecting the most consistent answers, significantly improving llm performance in different reasoning tasks. recent advanced works @xcite @xcite @xcite expanded on this strategy by considering different reasoning paths and self-evaluation to make holistic decisions. however, it's important to note that these methods require more resources and therefore slow down the decision-making process, especially for high-cost llms such as @xcite . this highlights the need for a method that lets llms adjust resource use based on the specific needs and complexities of tasks, which is crucial for developing and using llms effectively in real-world situations, allowing these models to provide the best value in various applications while managing resources wisely. to address the challenge, we introduce a dynamic decision-making framework called "dyna-think." this framework helps llms make smart choices between quick and thorough ways of solving problems. it optimizes a balance between being efficient and effective in various reasoning tasks. in this framework, we categorize tasks into two groups: "fast" for tasks where llms can quickly find confident answers, and "slow" for more complex tasks that require thorough exploration of different reasoning paths to confirm answers. establishing rules to distinguish between fast and slow reasoning is challenging. we adopt two criteria including consistency verification and reasoning complexity verification. the principle of consistency verification is reflective of the human heuristic, wherein the concurrence of multiple distinct thought processes on a singular answer augments the confidence of its correctness @xcite . different from the straightforward majority voting used in @xcite , which sometimes leads figure 1 : the workflow starts by incorporating the widely used cot prompt, like 'let's think step by step' and then initially querying llms four times for each question (which can be fewer than four to begin with). in the first step, we choose question 1 and question 2 because they have one answer with more than half the votes. however, question 3 has a tie, with two answers getting equal votes, so we consider it a slow-thinking question. next, we look at questions 1 and 2 to see how many steps are needed for each answer. for question 2, the answer with the most votes takes four steps, while the other answer needs three steps, so question 2 is also categorized as a slow-thinking question. regarding question 1, we classify it as a fast-thinking question because the answer with the most votes also requires the fewest steps, allowing us to output this answer directly. however, for slow-thinking questions, we require additional iterations to make a selection. to problems like tie-breakers, we only validate answers when they secure more than half of the total votes, indicating they are high-confidence solutions with the best votes. on the other hand, our reasoning complexity verification criteria are based on reasoning processes with a minimum number of reasoning steps. llms can make mistakes during the reasoning process @xcite , while each reasoning step of llms can be thought of as a decision made under uncertainty. the accumulation of these decisions can lead to a compounding of uncertainty, reducing the overall confidence in the final decision. therefore, fewer reasoning steps often yield more reliable and confident outcomes due to reduced error propagation.to empirically corroborate the two criteria, we have conducted experiments on multiple reasoning tasks and found a strong correlation between the efficacy of llms in problem-solving and both the length of their reasoning paths and the voting of the outputs. questions that meet these criteria are sorted using simple and fast decision making processes. the rest of the questions are subject to more exploration of different reasoning paths and comprehensive self-evaluations @xcite @xcite . ## dynathink dynathink, as shown in figure 1 foot_0 , starts by setting the initial query times for llm, which in this case is 2. the llm is then queried with the problem set p , yielding the response set q. first, we employ consistency verification to analyze the diversity of answers each question receives. questions whose answers get over half the votes are placed in q 1 , serving as potential fast-thinking questions. the rest go into q 2 for more review in the following rounds with added resources. next, we utilize reasoning complexity verification to examine the step counts needed for each response, picking the one with the fewest steps throughout all generations. we then see if this selected answer from q 1 indeed has the least steps. if it does, it goes into q 3 as a fast-thinking question of this round. if not, it returns to q 2 for more rounds of checking. lastly, we see if we've picked any fast-thinking questions this round. if q 3 has questions, we proceed to another iteration; if it's empty, we conclude, leaving the questions in q 2 as the slow-thinking ones. for the fast-thinking questions, we can easily arrive at an answer (more than half vote + least number of reasoning steps). for the slow questions, we repeat the same cot + self-consistency procedure with more queries to the llms so as to identify the answer with more than half vote and the least number of reasoning steps. ## experiments setup we evaluate the efficiency and effectiveness of dynathink on six diverse reasoning datasets, i.e., strategyqa @xcite , gsm8k @xcite , math @xcite , aqua-rat @xcite , svamp @xcite and mathqa @xcite ). 2 we use three popular black-box llms like gpt-3.5-turbo, gpt-4, gemini as well as one popular open-source llm, i.e., mixtral-8x7b @xcite as the backbones. we employ the self-consistency strategy (wang et al., 2022) within the cot reasoning as our main baseline following @xcite @xcite . although it's simple, it's still a strong method according to current research. additionally, for all the open-source models, we set the temperature to 0.7. ## related work the introduction of cot prompting by @xcite has been a seminal development in highlighting the multi-step reasoning capabilities of llms. this approach has paved the way for various refinements and enhancements @xcite @xcite @xcite , to enhance the effectiveness of llms in addressing complex problems. for further improvement of the result of cot, some methods like @xcite and @xcite concentrate on the optimization of reasoning structures and pathways.while others focused on careful planning and verification of the reasoning process.for example, @xcite introduced the prm800k dataset, which is notable for its inclusion of step-wise correctness labels obtained through crowdsourcing.and @xcite extracted insights from multiple reasoning steps and conducted sequential verifications to rectify inconsistencies. while these works have demonstrated significant advancements in handling complex problems, they often require significantly increased resource. balancing resource expenditure and efficiency is essential. this research aims to streamline budget allocation and resource utilization while maintaining high efficacy and efficiency in llm reasoning tasks. the goal is to achieve a harmonious balance between resource deployment and optimal performance, allowing for an efficient and resourceful approach to llm reasoning. ## conclusion in this paper, we introduced a dynamic decisionmaking framework called dynathink, which enables llms to make smarter choices between fast and slow problem-solving methods, striking a balance between efficiency and effectiveness. across various reasoning tasks, dynathink outperforms existing self-consistency methods, enhancing accuracy while optimizing computational resources.