AS - ES Learning: Towards efficient C o T learning in small models (2024)

Abstract: Chain-of-Thought (CoT) serves as a critical emerging ability in LLMs, especially when it comes to logical reasoning. Attempts have been made to induce such ability in small models as well by distilling from the data with CoT generated by Large Language Models (LLMs). However, existing methods often simply generate and incorporate more data from LLMs and fail to note the importance of efficiently utilizing existing CoT data. We here propose a new training paradigm AS-ES (Abstractive Segments - Extractive Segments) learning, which exploits the inherent information in CoT for iterative generation. Experiments show that our methods surpass the direct seq2seq training on CoT-extensive tasks like MWP and PET summarization, without data augmentation or altering the model itself. Furthermore, we explore the reason behind the inefficiency of small models in learning CoT and provide an explanation of why AS-ES learning works, giving insights into the underlying mechanism of CoT.

## introduction cot is one of the most important emerging abilities that distinguishes llms from prior models with smaller scales @xcite . the explicit introduction of cot enables llms to tackle complex problems that necessitate critical thinking and intricate logical reasoning, thus enhancing the overall performance of llms. @xcite @xcite . intuitively, numerous works extended the cot capacity to smaller-scale models @xcite @xcite @xcite @xcite . yet, these endeavors predominantly concentrate on generating an increased quantity of high-quality cot data from llms and resort * equal contribution † corresponding author to a direct seq2seq training approach, where the query forms the input and the cot-enriched answer serves as the target. such methods often overlook the limited capacity of small models in learning the complex reasoning in cot. taking this into account, some works tried to decompose cot into more fine-grained reasoning steps, and used llms to generate rationales for each individual step to augment the original cot @xcite @xcite . @xcite further improved the direct seq2seq paradigm using the augmented cot to train separate models for iterative generation. however, the use of augmented cot is just another way of generating more cot data, which is costly and still fails to fully exploit the inherent information in existing datasets. furthermore, although proven to achieve better performance, using separate small models for cot learning raises another question. similar to using different modules for information retrieval and reasoning in multi-hop qa @xcite @xcite , these methods specialize small models in single-task operations, in contrast with llms using a singular framework. such implementation underlines an assumption that small models, due to their constrained computational capacity compared to llms, may require separate, specialized models to perform distinct phases of cot processing @xcite . in response to the above issues, we introduce a novel paradigm different from earlier approaches for distilling cot to small models. we classify the statements inside the cot-format results of llms into two categories: extractive segments (es) that remind the model of the context and set the stage for subsequent conclusions, and abstractive segments (as) that infer additional insights not explicitly stated in the context. with the deconstruction of cot into as and es components, we curate a dataset tailored for an iterative learning process, a. ## related work deduction in nlp tasks deduction is a logical process of reasoning or inferring specific information from given premises or data. in nlp, deduction is used in various tasks, such as summarization, question answering and information extraction @xcite @xcite @xcite . early research employs end-to-end models to directly learn reasoning strategies from labeled and structured data @xcite , which requires the effort of human annotation. @xcite separates the inference process into a traditional generation part and an extra validation part. they construct a symbolic reasoning module to validate the generated facts using a minimal world model. however, the minimal world model must be hand-engineered. in our work, we employ a fully automatic strategy to disentangle the deduction process, achieving both efficiency and effectiveness. cot in small models cot is a significant ability to improve the performance of complex reasoning, which is considered as an emergent ability of llms @xcite . numerous work aims to transfer this ability to small models @xcite @xcite @xcite . one prevailing method is distilling from llms, such as utilizing llms to generate rationales or multi-step solutions as training data for smaller models @xcite . @xcite al. (2023) also allows llms to generate multiple rationales whose consistency is enforced by kl-divergence. ma et al. (2023) employ a two-stage distillation strategy, in which llms not only generate rationale but also provide an answer inference process according to the rationale generated by the student model. however, all these methods require llms as a teacher model. in our work, we require no additional data generated by llms, which substantially reduces the cost of time and computational resources. ## method this section elucidates the methodology of our study, addressing three pivotal questions: (1) how to segment the complete cot into as and es? (2) how to construct a dataset using the segmented as and es? (3) how does the as-es learning work for training and generation? ## 10688 weng earns $12 an hour for babysitting. yesterday, she just did 50 minutes of babysitting. how much did she earn? let's think step by step. ## conclusion in this paper, we introduce a data-efficient cot distillation strategy for small models. by segmenting cot data into extractive part and abstractive part respectively, we improve the model performance through an iterative generation approach without incorporating additional data. the generalization of as-es learning to different model sizes and tasks shows its effectiveness. we further explore whether two models are necessary for this interactive generation approach, and answer the derivative question that the limitation of small models in cot learning stems from the training paradigm instead 8 in figure 4a and figure 4b , grey, pink and yellow denote direct approach, interleaving segmentation and entropy segmentation respectively. figure 4c and figure 4d show the curve of the best training loss among time. all curves for asm are displayed in red with esm in bleu in figure 4c . in figure 4d , curves for asm, esm, usm are displayed in green, blue and pink, respectively.