P a D : Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning (2024)

Abstract: While large language models (LLMs) excel in various natural language processing tasks, their huge size and the inaccessibility of parameters present challenges for practical deployment. Previous studies try to distill task-specific ability from LLMs to smaller models, using data synthesis and chain-of-thought (CoT) fine-tuning. However, synthetic CoT data often contains faulty reasoning, which deteriorates the quality of distillation, especially in reasoning capabilities. In this work, we propose Program-aided Distillation (PaD), which introduces reasoning programs to suppress the errors in distilled data, and thus achieves better distillation quality for reasoning tasks. In PaD, we utilize the reasoning program to substitute the CoT, allowing automated error checking of synthetic data. Further, through error injecting and further training, the small distilling model could iteratively self-refine the reasoning. Moreover, we conduct a step-wise beam search by step-by-step verifying to acquire more exact reasoning chains. We evaluate PaD on arithmetic reasoning, symbolic reasoning, and general ability.Experimental results demonstrate that smaller models using PaD can not only outperform certain LLMs (e.g., LLaMA-1 13B) but also achieve strong improvement over baselines with a significantly smaller scale of parameters and data. The source code is publicly available athttps://github.com/Xuekai-Zhu/pad.

## introduction in recent years, llms have revolutionized the landscape of natural language processing, showcasing remarkable capabilities across various tasks @xcite @xcite . these models require vast quantities of data and extensive pre-training. however, scaling up the size of the question: ken created a care package to send to his brother, who was away at boarding school. ken placed a box on a scale, and then he poured into the box enough jelly beans to bring the weight to 2 pounds. then, he added enough brownies to cause the weight to triple. next, he added another 2 pounds of jelly beans. and finally, he added enough gummy worms to double the weight once again. what was the final weight of the box of goodies, in pounds? ## methodology we propose a novel approach called program-aided distillation, which induces program-aided reasoning data from llms and fine-tuning small models. during the decoding process, pad can iteratively refine reasoning programs and employ step-wise beam search to generate more reliable reasoning steps. as shown in figure 3 , pad includes the following main parts: (1) synthesizing data from llms, (2) fine-tuning small models, (3) self-refining the error reasoning programs, and (4) reasoning with step-by-step verification. ## discussion why pad is better than fine-tune cot ? as illustrated in figures 6 and 10 , pad effectively narrows the prediction space, focusing on specific central points. the primary distinction between pad and baseline models is the format of their reasoning methodologies. reasoning programs in python exhibit clearer and simpler syntax compared to natural language in cot. this implies that pad primarily adheres to python syntax, instead of extensive sampling across the entire language representation space. this, in essence, reduces the complexity of the task. as depicted in figure 5 , pad consistently achieves lower training and evaluation losses. more discussion is presented in appendix c. ## conclusion in this paper, we propose pad, which distills llms through synthesizing program-aid reasoning data. in the fine-tuning and decoding stage, we conduct self-refinement and step-by-step verification to further enhance the pad. experiments demonstrate pad assign small model reasoning ability even surpassing certain llms. compared to prior related works, pad achieves comprehensive superiority while utilizing less data and smaller models. further analysis reveals that pad possesses higher training efficiency and is suited for smaller models.