SCOTT : Self-Consistent Chain-of-Thought Distillation (2023)

Abstract: Large language models (LMs) beyond a certain scale, demonstrate the emergent capability of generating free-text rationales for their predictions via chain-of-thought (CoT) prompting. While CoT can yield dramatically improved performance, such gains are only observed for sufficiently large LMs. Even more concerning, there is little guarantee that the generated rationales are consistent with LM’s predictions or faithfully justify the decisions. In this work, we propose SCOTT, a faithful knowledge distillation method to learn a small, self-consistent CoT model from a teacher model that is orders of magnitude larger. To form better supervision, we elicit rationales supporting the gold answers from a large LM (teacher) by contrastive decoding, which encourages the teacher to generate tokens that become more plausible only when the answer is considered. To ensure faithful distillation, we use the teacher-generated rationales to learn a student LM with a counterfactual reasoning objective, which prevents the student from ignoring the rationales to make inconsistent predictions. Experiments show that while yielding comparable performance, our method leads to a more faithful model than baselines. Further analysis shows that such a model respects the rationales more when making decisions; thus, we can improve its performance more by refining its rationales.

## introduction large language models (lms) elicit strong reasoning capabilities through chain-of-thought (cot) prompting @xcite , which asks lms to generate free-text rationale for explaining their multi-step reasoning. however, cot prompting does not guarantee that the rationale is consistent with the prediction, rendering the rationale can a bengal cat survive eating only pancakes? the answer is no. why? is material from an aloe plant sometimes enclosed in petroleum-derived products? the answer is yes. why? 👎 error 2 (37%): do not justify the answer. figure 1 : vacuous rationales generated by a prompted lm (gpt-3) for strategyqa. in both types of error cases, lm fails to give rationales consistent with the answers due to hallucination. existing works @xcite propose learning to reason from large lms mainly for computation efficiency or task performance. they prompt a large lm (the teacher) to generate rationales for a downstream dataset, which is then used to train a small lm (the student). however, these works neglect the following two issues which could undermine the faithfulness of the rationales. first, lms are prone to hallucination, meaning they often generate text that is not grounded by the input @xcite . therefore, the teacher may not generate on-topic rationales, which fully support the answer. in our pioneer study (figure 1 ) over 100 random rationales generated by gpt-3, we found 42% of them not providing new information that is not stated in the task input and 37% of them not justifying the answer foot_0 . this inconsistency between the rationale and answer would then be inherited by the student. second, the student may treat rationale generation and answer prediction as two independent processes. this is due to the spurious correlations between the question and answer, which is exploited as a reasoning shortcut by the student @xcite . the two issues together would lead to an unfaithful student which learns to generate vacuous rationales and may make predictions inconsistent with the rationales. to address these issues, we propose to enhance the vanilla kd process from two ends respectively. to elicit more on-topic rationales from the teacher, we propose to leverage contrastive decoding which aims to ground each rationale to the answer ( § 3.1). this technique encourages the teacher to generate tokens that are more plausible only when the answer is considered instead of the ones that are fairly plausible even without the answer during decoding. to train a faithful student, we ask the student to conduct counterfactual reasoning, i.e., predicting accordingly when the rationales are leading to different answers ( § 3.2). we obtain the training data by asking the teacher to generate a rationale for a sampled incorrect answer. the reasoning shortcut between the question and the gold answer is thus removed since now the student needs to give a different answer for the same question, according to the rationales provided during training. we conduct experiments on several open-domain question answering tasks that require knowledgeintensive reasoning. experiments show that: (1) contrastive decoding can lead to a more consistent teacher which generates rationales that are more supportive of the gold answers. (2) trained on the more consistent rationale-answer pairs, the student learns to better associate the answer prediction with the rationale generation. (3) with counterfactual reasoning as an auxiliary training objective, the student learns not to take the reasoning shortcut and instead respect the rationale more. (4) despite being more faithful, our model performs comparably to the baselines. (5) ablation study shows that although performing better, larger student models are more prone to being inconsistent. our method robustly remedies the inconsistency regardless of the size of the student model. (6) with a more faithful student, we can better improve its performance by correcting its rationale, demonstrating the utility of our method in model refinement. ## chain-of-thought distillation our goal is to 1) elicit consistent rationales, i.e., those well justifying the gold answers, from a large lm as supervision, and then 2) train a selfconsistent student model to reason faithfully, i.e., answer accordingly to its generated rationale. we consider the task of language-based reasoning where the required knowledge is not provided in the task input. specifically, we focus on open-domain question answering (qa) which is the most general setting adopted by prior works: given a question q, a qa system is asked to predict the gold answer a * . for interpretability, we also require the model to provide a free-text rationale r, which justifies its prediction. below we describe the overview of a vanilla kd framework as illustrated in figure 2 . we then discuss the limitations and propose our method in § 3. ## distilling a self-consistent student there are two vital issues with the vanilla kd process described in the previous section. firstly, neural lms are known to suffer from the issue of hallucination, meaning they often generate text that is not grounded by the input @xcite . this would lead to the generated rationale not supporting the given answer. the inconsistency between the rationale and the answer 🤖 q: would someone with back pain enjoy picking strawberries? a: the answer is no. q: would someone with back pain enjoy picking strawberries? a: the answer @xmath0. ## conclusion this work presents a faithful kd framework for learning a small, self-consistent cot model from a large teacher model. to ensure the student reason faithfully, we propose (1) contrastive decoding for obtaining a consistent teacher and (2) counterfactual reasoning for teaching a faithful student. experiments show that these two techniques jointly lead to a more faithful student compared to the baselines, while preserving much performance accuracy. our further analysis shows that changing the rationales has a larger impact on the student's behavior and thus we can have more success in debugging the model by refining its rationales.