Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023)

Abstract: Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to few-shot prompted LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80% of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100% of the dataset.

## introduction despite the impressive few-shot ability offered by large language models (llms) @xcite @xcite @xcite , these models are challenging to deploy in real world applications due to their sheer figure 1 : while large language models (llms) offer strong zero/few-shot performance, they are challenging to serve in practice. traditional ways of training small task-specific models, on the other hand, requires large amount of training data. we propose distilling step-by-step, a new paradigm that extracts rationales from llms as informative task knowledge into training small models, which reduces both the deployed model size as well as the data required for training. to circumvent these deployment challenges of large models, practitioners often choose to deploy smaller specialized models instead. these smaller models are trained using one of two common paradigms: finetuning or distillation. finetuning updates a pretrained smaller model (e.g. bert @xcite or t5 @xcite ) using downstream human annotated data @xcite . distillation trains the same smaller models with labels generated by a larger llm @xcite @xcite . unfortunately, these paradigms reduce model size at a cost: to achieve comparable performance to llms, finetuning requires expensive human labels, and dis-tillation requires large amounts of unlabeled data which can be hard to obtain @xcite . in this work, we introduce distilling step-bystep, a new simple mechanism for training smaller models with less training data. our mechanism reduces the amount of training data required for both finetuning and distillation of llms into smaller model sizes. core to our mechanism is changing our perspective from viewing llms as a source of noisy labels to viewing them as agents that can reason: llms can produce natural language rationales justifying their predicted labels @xcite . for example, when asked "jesse's room is 11 feet long and 15 feet wide. if she already has 16 square feet of carpet. how much more carpet does she need to cover the whole floor?", an llm can be prompted by chain-of-thought (cot) technique @xcite to provide intermediate rationales "@xmath0. jesse's room has 11 × 15 square feet." that better connects the input to the final answer "(11 × 15) -16". these rationales can contain relevant task knowledge, such as "@xmath1, that may originally require many data for small task-specific models to learn. we thus utilize these extracted rationales as additional, richer information to train small models through a multi-task training setup, with both label prediction and rationale prediction tasks @xcite . distilling step-by-step allows us to learn taskspecific smaller models that outperform llms using over 500× less model parameters, and it does so with far fewer training examples compared to traditional finetuning or distillation (figure 1 ). our results show three promising empirical conclusions across 4 nlp benchmarks. first, compared to both finetuning and distillation, our resulting models achieve better performance with over 50% less training examples on average across datasets (and up to over 85% reduction). second, our models outperform llms with much smaller model sizes (up to 2000× smaller), drastically reducing the computation cost required for model deployment. third, we simultaneously reduce the model size as well as the amount of data required to outperform llms. we surpass the performance of 540b parameter llms using a 770m t5 model; this smaller model only uses 80% of a labeled dataset that would otherwise be required if using an exist-ing finetuning method. when only unlabeled data is present, our small models still perform on par or better than llms. we outperform 540b palm's performance with only a 11b t5 model. we further show that when a smaller model performs worse than an llm, distilling step-by-step can more efficiently leverage additional unlabeled data to match the llm performance compared to the standard distillation approach. ## related work our work distills task-specific knowledge of llms into smaller specialist models by leveraging the emergent reasoning capabilities of today's llms. we draw on recent knowledge distillation research and other methods that learn from both humangenerated rationales and llm-generated rationales. ## distilling step-by-step we propose a new paradigm, distilling step-bystep, that leverages the ability of llms to reason about their predictions to train smaller models in a data-efficient way. our overall framework is illustrated in figure 2 . our paradigm has two simple steps: first, given an llm and an unlabeled dataset, we prompt the llm to generate output labels along with rationales to justify the labels. rationales are natural language explanations that provide support for the model's predicted label (see figure 2 ). second, we leverage these rationales in addition to the task labels to train smaller downstream models. intuitively, rationales provide richer, more detailed information about why an input is mapped to a specific output label, and often contain relevant task knowledge that may be hard to infer solely from the original inputs. ## experiments we empirically validate the effectiveness of distilling step-by-step. first, we show that when compared to standard finetuning and task distillation approaches, distilling step-by-step achieves better performance with much fewer number of training examples, substantially improving the data efficiency to learn small task-specific models (sec. 4.1). second, we show that distilling step-by-step surpasses the performance of llms with much smaller model size, drastically lowering the deployment cost compared to llms (sec. 4.2). third, we investigate the minimum resources required, w.r.t. both number of training examples and model size, for distilling step-by-step to outperform llms. we show that distilling step-by-step outperforms llms by using less data and smaller model, simultaneously improving both data-and deployability-efficiency (sec. 4.3). finally, we perform ablation studies to understand the influence of different components and design choices in the distilling step-by-step framework (sec. 4.4). ## discussion we propose distilling step-by-step to extract rationales from llms as informative supervision in training small task-specific models. we show that distilling step-by-step reduces the training dataset required to curate task-specific smaller models; it also reduces the model size required to achieve, and even surpass, the original llm's performance. distilling step-by-step proposes a resource-efficient training-to-deployment paradigm compared to existing methods. further studies demonstrate the generalizability and the design choices made in distilling step-by-step. finally, we discuss the limitations, future directions and ethics statement of our work below.