Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning (2023)

Abstract: Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language process tasks, but designing prompts that generalize well to diverse problem types can be challenging CITATION, especially in the context of math word problem solving. Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques. To address these issues, we investigate two approaches to leverage the training data in few-shot prompting scenario: dynamic program prompting and program distillation .Our approach is largely inspired by CITATION where they proposed to replace the CoT with the programs as the intermediate reasoning step. Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving.Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the program-annotated training data.Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning.Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of fine-tuned smaller models in MWP solving.

## introduction designing effective prompts is crucial for the success of few-shot prompting with large language models (llms) in tasks requiring complex reasoning skills @xcite @xcite for the task of arithmetic word problem, it poses a significant challenge to design a small number of chain-of-thought (cot) prompts @xcite to solve a wide range of problems. fortunately, a modest amount of training data is usually available though no chain-of-thought (cot) annotation exists. @xcite adopts a retrieval-based approach to select similar samples as prompts. while such a method does not work well for numerical reasoning compared to cot prompting @xcite , recent work @xcite also tried to distill the cot knowledge from large language models to smaller language models. the distilled cot annotations allow us to further fine-tune the small language models. however, there is no guarantee that the generated cot prompts for the training data are correct, and it is challenging to perform an automatic evaluation to verify the cot correctness. as an alternative, recent work @xcite @xcite ) has adopted programs as intermediate reasoning chains in tasks such as math word problems (mwps), allowing for automatic verification of answers through program execution. inspired by these approaches, we can perform prompting with code generation models, such as codex @xcite , to annotate the training data with programs that can be executed. figure 1 shows the process of automatic program annotation using large language models. as we can see in this example, though the final answer "8 dollar" by cot is correctly generated, the intermediate reasoning path is wrong because of incorrect calculation for "5 × 3". instead, the program sampling is relatively more rigorous in that we can execute to obtain the numeric answer rather than cot in natural language. apparently, we can keep sampling the program with different temperatures until the answer executed from the program matches the correct one. once we obtain the annotated program, we can use the "annotated" training data with the pseudo-gold program to further improve the performance on the test set. in this work, we primarily study two approaches for making use of the "annotated" programs: dynamic program prompting and program distillation @xcite . dynamic program prompting employs the top-k similar training samples (with annotated pseudo-gold programs) as fewshot prompts. we use publicly available and stateof-the-art sentence encoders such as openai embeddings @xcite foot_1 and sentence-t5 @xcite for computing the cosine similarity. on the other hand, we follow magister et al. (2022) to fine-tune smaller language models on our pseudo-gold training data. overall, our experiments on three standard math word problem datasets demonstrate the effectiveness of leveraging the training program in our few-shot prompting. we observe significant improvements for all datasets, especially for the mathqa @xcite dataset, where diverse subjects (e.g., physics, probability, etc.) were involved in the problems where the fixed prompt accompanied by limited examples is insufficient to encapsulate the entire dataset #train #program #valid #test gsm8k 07,473 6,363 (85.1%) -1,319 svamp 03,138 3,071 (97.9%) -1,000 mathqa † 16,191 7,676 (47.4%) 2,411 1,605 ## approach training data annotation following the approach in program-aided language model (pal) @xcite , we can sample the program for each math word problem as an annotation. specifically, we use the math prompts from pal as seed prompts to perform few-shot prompting with large language models (i.e., codex @xcite ). we follow the exact same format from pal @xcite without any changes, figure 2 shows an example program from the gsm8k training set. we can verify the answer's correctness by comparing the result from the program execution with the ground-truth value. for each math word problem x in training set d, we first perform greedy decoding with temperature @xmath0. if the predicted answer ŷ from the executed program p matches the ground-truth answer y, we add this tuple (x, p , y) into a new training data set d prog . if the predicted answer is incorrect, we increase the temperature and continue sampling programs until we find one with the correct answer. in practice, we may not always obtain the correct answer and have a limited budget for codex api usage. thus, we sample at most k times for each instance. if we cannot find a program with the correct answer within k samples, we discard the instance x. as a result, the size of the resulting training set d prog is expected to be smaller than the original training set (refer to ## experiments dataset and experiment setting similar to @xcite , we mainly conduct experiments on gsm8k @xcite , svamp @xcite , and mathqa @xcite datasets. the maximum number of sampling k for each training instance is set to 5 foot_2 , and the temperature t is 0.5 following previous practice @xcite . we discard the training instance if we cannot find a proper program. the number of prompts m is set to 8 following previous work in math word problem solving @xcite @xcite . in fine-tuning experiments, we use the 6b codegen language model. the learning rate for fine-tuning experiments is 2e-5. we fine-tune the codegen model with a batch size of 48 and experiment with 40 epochs on all datasets. the fine-tuning experiments are conducted with 8 a100 gpus. we did not perform a hyper-parameter search for fine-tuning. all parameters are set to the above default values. ## related work our work is mostly related to recent literature that incorporates the training data to improve the language model performance on downstream tasks. @xcite shows that we can benefit from additional cot data for both large and small language models. @xcite samples cot reasoning paths for the training data and uses them to diversify the prompts on the gsm8k dataset. alternatively, we can use the sampled cot to further fine-tune the language models @xcite @xcite . in practice, we cannot guarantee the correctness of the sampled cot, especially for the task of math word problem solving, which requires rigorous reasoning paths. recent approaches @xcite attempt to reduce the negative effect by matching the answer with generated cot or assigning different weights for the samples. simultaneously with this study, @xcite proposes to use step-based reward to improve the performance specifically on gsm8k. in order to do so, the authors need to annotate a portion the data to train the underlying reward model. however, these methods cannot completely avoid the underlying limitation as it is challenging the evaluate the step-by-step natural language cot @xcite . our approach is inspired by program generation via few-shot prompting @xcite , we perform prompting on the training data and easily verify the answer correctness by executing the program, which allows us to obtain more reliable pseudogold programs. ## conclusion and future work motivated by program-based prompting @xcite , we are able to obtain the pseudo-gold program as the intermediate reasoning step for training data. we then present two approaches to make use of such data with program annotations in both of the few-shot prompting and fine-tuning scenarios. in few-shot prompting with llms, we sample similar exemplars as prompts for experiments. in the fine-tuning approach, we directly fine-tune a pre-trained language model on program-annotated data. our experiments demonstrate both few-shot prompting and fine-tuning can significantly benefit from the training data annotated with programs, especially for complex problems in the mathqa dataset. for future research, our goal is to design a structured model that leverages the potential of data with program annotations, particularly in light of the substantial underperformance of smaller language models. interestingly, even with their limitations, structured models @xcite have exhibited the capacity to outshine large language model prompting on mathqa. additionally, the recent emergence of instructionfollowing models @xcite , exemplified by alpaca @xcite , has prompted our interest in equipping large language models with mathematical reasoning capacities @xcite while maintaining the integrity of their underlying language understanding capabilities.