MARIO : MA th Reasoning with code Interpreter Output - A Reproducible Pipeline (2024)

Abstract: Large language models (LLMs) have significantly improved in understanding natural language but still lack in mathematical reasoning, a hurdle on the path to true artificial general intelligence. The training of large language models, based on next-token prediction, struggles to capture the precise nature of mathematical reasoning, presenting both practical and theoretical challenges. In this paper, we address this challenge by enriching the data landscape and introducing a reasonable data format, enhanced the text analysis of the LLM with a capability to utilize a Python code interpreter. This dataset is derived from GSM8K and MATH and has been further refined through a combination of GPT annotations, human review, and self-training processes. Additionally, we propose a tentative, easily replicable protocol for the fine-tuning of math-specific LLMs, which has led to a significant improvement in the performance of a 7B-parameter LLM on the GSM8K and MATH datasets. A solution generator and a value estimator are fine-tuned simultaneously in a multi-task fashion, while an outlier-free value model-based inference method is proposed to further boost the performance. We are committed to advancing the field of mathematical reasoning in LLMs and, to that end, we will make the source code and checkpoints publicly available.

## introduction the chain-of-thought (cot) prompting technique @xcite has been empirically shown to enhance the complex reasoning capabilities of large language models (llms) by generating a sequence of intermediate reasoning steps. proprietary llms, including @xcite and claude-2 @xcite , are designed to produce cot responses by default, leading to improved reasoning performance, as evidenced by a 50.36% accuracy rate on the math dataset @xcite with gpt-4 @xcite . moreover, when llms are augmented with a plugin capable of executing code snippets, their accuracy in arithmetic computations-which are typically challenging for llms-is further enhanced, e.g., gpt-4-code achieved a 69.69% accuracy rate on the math dataset @xcite . this underscores the efficacy of integrating text analysis with code execution in datasets designed for math reasoning tasks. recently, @xcite demonstrated that, despite extensive fine-tuning for arithmetic operation simulation task, llms are still unable to achieve perfect accuracy. consequently, integrating code for precise numerical computation has become an inevitable trend. the program-of-thought (pot) framework @xcite and program-aided language (pal) models @xcite represent seminal efforts in generating code-centric datasets. building on this trend, tora @xcite advances the field by employing a proprietary annotation methodology along with gpt-4, setting a new benchmark for state-of-the-art performance in solving mathematical problems via python code. however, it is noteworthy that the solutions contained within these datasets predominantly consist of code snippets, with minimal accompanying textual analysis. an obvious shortcoming of a code-centric solution is that it can overlook common sense in math word problems, e.g.,, as shown in figure 1 , the pot solutions disregard the fact that the quantity of the food taken cannot be negative. mathcoder @xcite represents an initiative that emulates the response patterns of gpt-4, integrating a plugin proficient in both code generation and natural language reasoning. this approach harnesses the capabilities of gpt-figure 1 : a comparison between code-centric solutions and our text-code solution. in the pot solution provided by gsm-hard @xcite , the code-centric approach yields a negative quantity of cups. the pot solution from tora omits the minus sign in its concluding sentence, failing to address the impractical issue. our solution, incorporating both text analysis and a code snippet, recognizes that a negative quantity of cups is illogical. furthermore, we introduce a tentative and easily replicable protocol for the fine-tuning of mathspecific llms. to ensure the reproducibility of our experiments, we begin with the extensively studied llm, llama-2 @xcite , and utilize its math-oriented continual pre-training variant, llemma @xcite . subsequently, we apply supervised fine-tuning to our annotated dataset to establish a baseline model. moreover, we implement a toolkit for evaluating math answers, which allows for the comparison between the ground truth and the llm predictions. unlike text generation tasks such as summa-rization, mathematical reasoning usually yields a unique answer, which simplifies the verification of its correctness. however, assessing the reasoning process that leads to the final answer remains challenging. to address this issue, lightman et al. (2023) introduced a model supervised by processes and a corresponding dataset with manually labeled solution procedures. yet, from a replication standpoint, creating these annotations is both labor-intensive and costly. as a compromise, we recommend training a straightforward outcome-supervised model as described by @xcite , to serve as an auxiliary tool for comparing and selecting the best among various solutions. our outcome value model (ovm) is fine-tuned using the efficient lora training (hu et al., 2021) in a multi-task setting, which enables the model to conserve computational resources while maintaining its generative capabilities. in summary, our main contributions are in three-fold. 1. we create a math reasoning format that integrates both text analyses and code snippets, leveraging logical reasoning and precise computation. ## dataset in this section, we detail our methodology for building the corpus, aiming for seamless integration of text analysis and code snippets. the textual content should articulate the problem-solving process, while the code snippets should perform precise computations. as shown in figure 2 , our data pipeline mainly includes three steps. ## fine-tuning to enhance a large language model's mathematical reasoning capabilities, we propose utilizing the foundational model llemma @xcite . there are two primary reasons for choosing llemma. first, llemma represents a continuation of the pre-training process initiated by llama-2 @xcite , extending its proficiency into both mathematical and coding domains, which aligns seamlessly with our requirements. second, it has been demonstrated that neither llama-2 nor llemma exhibit excessive overfitting on the gsm8k or math dataset, as confirmed by @xcite . ## conclusion this paper introduces a reproducible pipeline that covers both the construction of a math-specific dataset and the fine-tuning of a large language model (llm). our approach demonstrates that integrating text analyses with code snippets enhances the model's capabilities for common sense reasoning and precise computation in mathematical reasoning tasks. moreover, our fine-tuning method enhances model performance by incorporating a verifier model that requires only a negligible number of additional parameters. to the best of our knowledge, our approach sets a new state-of-the-art benchmark for llms with a size around 7 billion parameters on the math datasets, and it exhibits notable generalization ability on challenging outof-domain math datasets.