Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data (2024)

Abstract: While large language models (LLMs) have shown excellent capabilities in language understanding, text generation and many other tasks, they still struggle in complex multi-step reasoning problems such as mathematical reasoning. In this paper, through a newly proposed arithmetical puzzle problem, we show that the model can perform well on multi-step reasoning tasks via fine tuning on high-quality synthetic data. Experiments with the open-llama-3B model on three different test datasets show that not only the model can reach a zero-shot pass@1 at 0.44 on the in-domain dataset, it also demonstrates certain generalization capabilities on the out-of-domain datasets. Specifically, this paper has designed two out-of-domain datasets in the form of extending the numerical range and the composing components of the arithmetical puzzle problem separately. The fine-tuned model have shown encouraging performance on these two far more difficult tasks with the zero-shot pass@1 at 0.33 and 0.35 correspondingly.

## introduction large language models (llms), as zero-shot and multi-task learners, have shown extraordinary capabilities across a variety of natural language tasks @xcite @xcite @xcite @xcite @xcite . however, even the most advanced llms face challenges when it comes to tackling complex multi-step reasoning problems, such as mathematical and scientific reasoning @xcite @xcite @xcite @xcite . this comes from three main reasons: firstly, mathematical reasoning often requires quantitative multiple steps of deduction, since a single logical error is enough to derail a much larger solution @xcite . secondly, the lack of high-quality data limits llms' ability to generalize and excel in mathematical reasoning tasks. lastly, llms encounter difficulty in extrapolation, as they struggle to apply reasoning skills when solving unseen mathematical problems. many prior research has explored along these challenges. gpt-4 @xcite , llama @xcite , gemini @xcite , minerva @xcite , llemma @xcite , @xcite , wizardmath @xcite , mam-moth @xcite , tora @xcite and deepseek @xcite @xcite have emerged as dominant models in popular mathematical reasoning benchmarks such as gsm8k @xcite , math @xcite , cmath @xcite and agieval @xcite . moreover, process supervision and verifiers @xcite @xcite @xcite at the step level have also obtained widespread attention. however, mathematical extrapolation, particularly in terms of abstract forms, is often overlooked. in this paper, we address the aforementioned challenges by introducing a novel and challenging arithmetical puzzle problem and making an initial attempt to solve them. specifically, we propose a puzzle that needs multi-step calculations to generate a correct solution. meanwhile, a data synthesis pipeline is developed to automatically generate a vast amount of high-quality data for supervised fine-tuning (sft). and a series of llms based on open-llama-3b @xcite ) are finetuned on this synthetic dataset. furthermore, to demonstrate the reasoning abilities in extrapolation, example of the synthetic data -prompt-34, 18, 31, 41, 19, 55: -110 -response-31-34=-3, 19+41=60, 60/-3=-20, -20/18=-2, -2*55=-110 our major contributions can be concluded as: (1) we propose a novel arithmetical puzzle problem with corresponding data synthesis pipeline and out-of-domain benchmarks, to verify the multi-step reasoning and extrapolation capabilities of llms fine-tuned on synthetic data. (2) experiments indicate that increasing the amount of high-quality synthetic data leads to performance enhancements across in-domain and out-of-domain datasets. (3) a comprehensive case study has been performed. ## conclusion large language models (llms) are intrinsically zero-shot and multi-task learners. however, mathematical reasoning still poses challenges for llms, we propose that the reasons can be mainly categorized into three folds: (1) requirement of multistep derivation; (2) lack of high quality data for fine-tuning; (3) difficulty in extrapolation. in this paper, we design an arithmetical puzzle and make an early attempt to solve these challenges. we develop a 24-point puzzle-like problem which asks for multi-step calculations to arrive at the correct answer. a corresponding data synthesis pipeline is proposed to generate an arbitrary amount of highquality data, on which a series of llms are finedtuned. in order to verify the extrapolation capability of our models, we have designed two outof-domain benchmarks and show that our model achieves competitive performance. furthermore, a data scaling experiment is conducted and it is concluded that by increasing the amount of training data, both the training loss and in/out-of-domain performance of the fine-tuned model improve accordingly.