D ial C o T Meets PPO : Decomposing and Exploring Reasoning Paths in Smaller Language Models (2023)

Abstract: Chain-of-Thought (CoT) prompting has successfully enhanced the reasoning capabilities of Large Language Models (LLMs) with at least 100 billion parameters. However, it is ineffective, or even detrimental, to the performance on reasoning tasks in Smaller Language Models (SLMs) with less than 10 billion parameters. In this paper, we propose Dialogue-guided Chain-of-Thought (DialCoT) to improve the reasoning capabilities of SLMs, with the aim of generating intermediate reasoning steps in a dialogue format to guide the model to the final answer. Furthermore, we optimize the model to choose the optimal reasoning path through the Proximal Policy Optimization (PPO) algorithm, further enhancing its reasoning capabilities. Compared to previous methods, our advantages lie in: 1) We transform the process of solving complex reasoning problems into decomposing problems and solving a series of simpler sub-questions, significantly reducing task difficulty and making it more suitable for SLMs. 2) We optimize the model to choose the optimal reasoning path through the PPO algorithm. Comprehensive experiments on four arithmetic reasoning datasets show that our method can achieve significant performance gains over state-of-the-art competitors.

## introduction with the advent of chain-of-thought (cot) prompting @xcite , which encourages large language models (llms) to generate a series of intermediate steps to help get the final answer, reasoning capabilities of llms has seen @xcite . cot prompting guides the model to output all intermediate reasoning steps at once to obtain the final answer, while dialcot leads the model to gradually generate intermediate reasoning steps in a dialogic format to ultimately arrive at the answer. significant improvement. however, preliminary results of @xcite have demonstrated that cot prompting only shows significant performance gains on llms (≥100b), such as lamda-137b @xcite , @xcite and palm-540b @xcite . but it is ineffective, or even detrimental, to the performance on reasoning tasks in smaller language models (≤ 10b). this phenomenon is explained by @xcite , who attribute it to abilities such as semantic comprehension and symbolic mapping, which only manifest at larger scales. the massive computational requirements and inference costs of llms make them unfeasible for widespread deployment. there is a pressing community interest in figuring out how to further enhance the reasoning capabilities within smaller language models (slms). recent works @xcite @xcite have attempted to enhance the performance of slms on reasoning tasks by fine-tuning them with training data generated from llms that contain intermediate reasoning steps. however, the results have been less than optimal. to further boost the reasoning capabilities of slms, we propose dialogue-guided chain-of-thought (dialcot), which aims to progressively generate intermediate reasoning steps in the dialogue format, instead of generating all intermediate reasoning steps at once (as shown in figure 1 ). specifically, we assign to the model two roles: decomposer and solver. the decomposer is tasked with breaking down the original question into a series of sub-questions. the solver sequentially addresses each sub-question presented by the decomposer, thereby obtaining the answer to the original question. they utilize different instructions while sharing the same model parameters. we propose three different forms of dialcot: 1) dialcot-a (all at once), in which the decomposer generates all sub-questions at once and the solver simultaneously provides all answers. 2) dialcot-m (mixed), where the decomposer generates all sub-questions at once but the solver sequentially delivers the answers of the sub-questions generated by the decomposer. 3) dialcot-s @xcite , where both the decomposer and solver operate sequentially to generate sub-questions and their corresponding answers. we provide a detailed comparison of the performance of the three different forms of dialcot in section 4.4. furthermore, building upon dialcot-s, we design dialcot-s-ppo, which leverages the proximal policy optimization algorithm to select the optimal reasoning path, thereby further enhancing its performance in reasoning tasks. compared to previous methods @xcite , our approach has two main advantages: to validate the effectiveness of our approach, we fine-tune flan-t5 @xcite using 7000 training examples from the gsm8k @xcite dataset that include intermediate questions and answers. the results surpass the latest method, specialft @xcite , by 6.2%. notably, the amount of training data we use is only 1/20 of that used by specialft. in addition, to verify the model's generalization capability on outof-distribution tasks, we also test on the multi-arith @xcite , asdiv @xcite and svamp @xcite datasets. our method achieves state-of-the-art performance compared to other baselines. ## dialogue-guided chain-of-thought we propose dialogue-guided chain-of-thought (dialcot), which aims to decompose complex questions into sub-questions in a dialogue format and gradually guide the model to obtain the final answer. specifically, we introduce two roles for the model, namely the decomposer and the solver, who engage in a dialogue-based interaction. the decomposer is responsible for breaking down the original question into a series of simpler subquestions, while the solver sequentially answers these sub-questions. we design distinct instructions for the decomposer and solver, followed by performing instruction tuning @xcite on slms. we first introduce three different forms of dialcot. subsequently, we describe how we incorporate the proximal policy optimization (ppo) algorithm into dialcot to enable the model to select the optimal reasoning path and further enhance its reasoning capabilities. ## conclusion in this paper, we explored strategies to boost the reasoning capabilities of slms and proposed dial-cot, which aims to generate intermediate reasoning steps in a dialogue format leading to the final answer. specifically, we designed two roles for the model, namely decomposer and solver. the decomposer is responsible for breaking down questions into multiple sub-questions, while the solver is tasked to address the sub-questions. they engage in dialogue to arrive at the final answer. we introduced three different dialogue formats: dialcot-a (all at once), dialcot-m (mixed) and dialcot-s @xcite . furthermore, we incorporated the ppo algorithm into dialcot-s to enable the model to choose the optimal reasoning path among multiple options, thereby further improving its performance on reasoning tasks. we conducted extensive experiments on four arithmetic reasoning datasets and the experimental results demonstrate the effectiveness of our method. future work primarily involves extending our method to other types of reasoning tasks, such as commonsense reasoning and symbolic reasoning. in addition, we will explore other decomposition methods or other reinforcement learning methods to optimize the reasoning paths of slms.