SAAS : Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models (2024)

Abstract: This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.

## introduction the advent of large language models (llms) has marked a significant breakthrough in various domains. however, despite their remarkable performance across these domains, a notable challenge persists in the realm of mathematical reasoning @xcite @xcite @xcite @xcite . the ability of llms to comprehend, interpret, and manipulate mathematical concepts is not yet on par with their linguistic capabilities. the significance of mathematical reasoning in llms involves more than just crunching numbers. it also encompasses the ability to engage in logical * equal contribution † corresponding author thinking, problem-solving, and complex decisionmaking, which are essential for understanding and generating human-like responses in the different situations @xcite @xcite . in other words, mathematical reasoning in llms is essential for a comprehensive understanding and manipulation of language in numerous scientific and practical applications. however, the current ability of llms in mathematical reasoning hinder their potential in the fields where numerical and logical comprehension are paramount such as coding. thus, it's critical challenge to enhance the ability of llms in mathematical reasoning. in this study, we explore a learning approach for enhancing both mathematical reasoning ability and problem-solving ability in llms, focusing on learning with both the chain-of-thought (cot) @xcite and the program-of-thought (pot) @xcite . the cot rationale (figure 1-(a) ) consists of a series of intermediate reasoning steps. although it enhances the reasoning ability of llms, it leads to arithmetic calculation errors when dealing with large numbers @xcite , resulting a low problem-solving ability. to address this issue, @xcite proposed the pot rationale (figure 1-(b) ), which expresses the reasoning steps as code and delegate computation steps to an code interpreter. it requires the reasoning steps to be expressed accurately as code. therefore, we hypothesize that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. in other words, the initial learning with cot is essential for solving challenging mathematical problems, since it improves the mathematical reasoning ability @xcite @xcite . our research is motivated by an analysis of existing models @xcite . tora @xcite tried to learn reasoning ability as well as pot by adding reasoning step into the pot rationale. similarly, mammoth @xcite tried to learn both cot and pot by using both cot rationale and pot rationale as training data simultaneously. however, we conjecture that they do not fully utilize the advantages of learning with both cot and pot. this is because they did not consider the sequence of cot learning and pot learning, resulting a less effective learning. in this work, we introduce a sequential learning approach, named saas (solving ability amplification strategy), to effectively utilize the strengths of cot learning and pot learning. this approach transitions from cot learning to pot learning, focusing on enhancing problem-solving ability in pot learning based on logical skills established in cot learning. this pedagogical strategy ensures that the competencies developed during cot learning positively influence the pot learning phase, leading to an overall improvement in solving challenging mathematical problems. we validate the rationality and effectiveness of our saas via extensive experiments on the reputable benchmarks @xcite @xcite @xcite @xcite . most importantly, saas achieved state-of-the-art with remarkable performance. through this, in this paper, we present a novel and effective perspective (i.e., our hypothesis) within the field of mathematics. ## saas: solving ability amplification strategy in this paper, we hypothesize that learning about the problem-solving ability is more effective after logical skills are well established. to explore this, we propose the sequential learning approach, named saas (solving ability amplification strategy), which transitions from cot learning to pot learning as shown in figure 1 . our saas is motivated by the pedagogical strategy of human that first learns logical skills and then develops problem-solving abilities by solving numerous problems @xcite . in the following subsections, we describe cot learning and pot learning in details. ## experiments in this section, we conduct extensive experiments to answer the following key research questions (rqs): ## conclusion in this paper, we demonstrated the following two important points in the sense of solving challenging mathematical problems: (1) prioritizing the learning of mathematical reasoning ability via chain-of-thought (cot) learning is helpful for the amplification of problem-solving ability during program-of-thought (pot) learning; (2) for effective sequential learning, it is necessary to employ a cognitive retention strategy that incorporates some data samples from the initial phase into the subsequent phase. in light of this, we proposed a novel sequential learning approach, named saas (solving ability amplification strategy), which progresses from cot learning to pot learning with cognitive retention strategy. through extensive experiments with the reputable benchmarks, we demonstrated that saas consistently and significantly outperforms all competitor, marking a significant advancement in the field of mathematical reasoning in llms.