Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step (2023)

Abstract: Chain-of-thought prompting (e.g., “Let’s think step-by-ste”) primes large language models to verbalize rationalization for their predictions. While chain-of-thought can lead to dramatic performance gains, benefits appear to emerge only for sufficiently large models (beyond 50B parameters). We show that orders-of-magnitude smaller models (125M—1.3B parameters) can still benefit from chain-of-thought prompting. To achieve this, we introduce Symbolic Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. Experiments across several commonsense benchmarks show that: 1) SCoTD enhances the performance of the student model in both supervised and few-shot settings, and especially for challenge sets; 2) sampling many reasoning chains per instance from the teacher is paramount; and 3) after distillation, student chain-of-thoughts are judged by humans as comparable to the teacher, despite orders of magnitude fewer parameters. We test several hypotheses regarding what properties of chain-of-thought samples are important, e.g., diversity vs. teacher likelihood vs. open-endedness. We release our corpus of chain-of-thought samples and code.

## introduction empirical scaling laws suggest that the accuracy of large language models (llms) on benchmark tasks can be improved by increasing model size and pre-training data volume @xcite . beyond these training-time improvements, however, an inference-time strategy dubbed "chain-ofthought" (cot) prompting, 1 i.e., eliciting verbalizations of predictive processes via key-phrases like "let's think step-by-step" @xcite , can *work done during an internship at ai2. 1 sometimes called "self-rationalization" or "prompting with explanations." we will use these terms interchangeably in this paper. ## symbolic chain-of-thought distillation our primary goal is to improve the accuracy of a (relatively small) student language model s on a target classification foot_0 task d @xmath0, y i )}. 3 we assume access to 1) (an unlabeled) training set d @xmath1; and 2) a large teacher language model t (e.g., gpt-3 @xcite ), capable of generating chain-of-thoughts in a few-shot fashion. our first step is to curate a set of labeled chainof-thoughts to serve as few-shot prompts for t . for each target task, we sample a small number (e.g., 10) of examples x i from d train , provide a gold classification label y i , and manually author a chain-of-thought z i for each to form the prompt set @xmath2, y i , z i )} 4 . then, for each x i in d train , we sample n chainof-thoughts zi along with the resulting prediction ỹi from the teacher model, i.e., the result of this sampling is a corpus @xmath3, {(ỹ k i , zk i )} n @xmath4, which contain teacherpredicted chain-of-thoughts/labels. depending on the experimental setting (details in § 3), we sometimes filter the entries of c, e.g., in the fully supervised case where d train instances have associated labels, we discard samples for which the sample the teacher model predicted an incorrect label. next, we train the student model using the standard language modeling loss, i.e., we maximize after fine-tuning the student model on the corpus sampled from the teacher, to evaluate the model on a test instance (x test , y test ) from the target task, we decode both a chain-of-thought ztest and a predicted label ỹtest from the student and evaluate ỹtest versus the true label y test . we consider two strategies for decoding. (1) predict the most likely chain-of-thought and the label ztest , ỹtest = argmax z,y s(z, y|x test ). this can be approximated by greedy decoding or beam search. (2) there may be different valid chainof-thoughts for a given question and as a result, large language models distribute probability mass for a certain label across many diverse chain-of-thoughts @xcite . thus, it is beneficial to marginalize out the reasoning paths to find the most consistent answer: ỹtest = argmax y e z∼s(z|xtest) s(y|z, x test ). this can be approximated by sampling multiple reasoning paths and take a majority vote among the predicted answers, dubbed "self-consistency" @xcite . we experiment with both approaches and conduct a discussion in §3.2. ## experiments we evaluate primarily on 3 target tasks: 1) com-monsenseqa (csqa) @xcite , a 5way multi-choice dataset; 2) openbookqa @xcite , and 3) quarel @xcite . while any model capable of few-shot chain-of-thought could be substituted, we use the thought prompts from prior work @xcite when available. we sample from gpt-3 with a temperature of @xmath5.0. for each training example, we sample @xmath6. opt is fine-tuned with a batch size of 32 and a learning rate of 2 × 10 -5 . we use huggingface transformers @xcite , pytorch @xcite , and accelerate foot_4 for the implementation. main experiments can be reproduced on one gpu with 48gb of memory. ## what factors are important for distillation? an important factor underlying the performance gains highlighted in §3 was the number of chain-ofthoughts we sampled from the teacher model perinstance (more @xmath7; figure 2 ). here we ask: is data volume the key contributing factor to the performance improvement? or, are specific aspects of chain-of-thought samples key for the performance improvements? we design several filters to identify potentially important examples/cots among the correct rationales. we apply designed filters (to be introduced) to c ′ , the corpus sampled from the teacher (with wrong cots dropped), that operationalize different hypotheses about what factors are important to distill. we control for dataset size when filtering, i.e., ## related work chain-of-thought prompting. as an extension of few-shot prompting @xcite , chainof-thought has proven more generally applicable than algorithmic/structured reasoning for which intermediate step generation was initially studied, e.g., by roy and roth (2015); ling et al. (2017); chiang and chen (2019); nye et al. (2021). recent studies seek to improve and analyze cots from different perspectives: wang et al. (2022b) improves the original cots through marginalizing over diverse reasoning paths while wang et al. (2022a) marginalize over diverse prompts; zelikman et al. (2022); huang et al. (2022) improves cot through a bootstrap manner of training on self-generated cots; li et al. (2022b) introduce voting classifiers to filter sampled cots before final prediction; golovneva et al. (2022) introduce some automatic metrics for automatic assessment of chain-of-thoughts. this study instead focuses on enabling cot for smaller models via distillation. learning with explanations. hase and bansal (2022) discuss how explanations can serve as inputs @xcite , targets @xcite @xcite @xcite @xcite , and priors @xcite for machine learning models. chain-of-thought extends earlier efforts which treat explanations as intermediate structures, generated at inference time @xcite . most related to our work is @xcite , who do also learn with gpt-3 generated explanations; we show multiple samples improve significantly over their single-sample method, and also use chain-of-thought prompting at inference time vs. predicting explanations+labels via independent multitasking. knowledge distillation. recent work, inspired by knowledge distillation @xcite , has considered symbolic knowledge distillation, @xcite , i.e., instead of distilling from soft representations like logits, large language model serve as training data generators @xcite @xcite @xcite @xcite ; this paper continues this line of work. ## conclusion we demonstrate the effectiveness of symbolic chain-of-thought distillation (scotd): a method that enables smaller language models to effectively use chain-of-thought-style reasoning. we demonstrate the method's effectiveness across several downstream tasks, different student model sizes, different levels of supervision, and in difficult settings (challenge sets, unseen tasks). our ablations shed light on what factors are particularly important to distill in these chain-of-thoughts. our concrete recommendations are: 1) sampling multiple and diverse cots for each input instance, and 2) performing self-consistency when the teacher cots are noisy. several promising av-