Paper: KNIFE: Distilling Reasoning Knowledge From Free-Text Rationales (ICLR-2024)

Reviewer comment (Evaluation):
Their best model still falls far short of the Large Language Models (LLM) by a significant margin, and the student model doesn't seem to improve model explanation and reasoning.

Author response:
We conducted the Wilcoxon rank-sum test between KNIFE and the best baseline for each OBQA/StrategyQA setting, and for both human-written and model-annotated FTR sources. KNIFE achieves statistically significant improvements (p<0.05) over the best baseline in all settings, except in the T5-Large→T5-Large on StrategyQA.

We also believe that the comparison to large-scale language models (LLMs) is unfair, as they are often approximately 100 times larger than the LMs we experimented with. Our focus is on how to improve the performance of smaller LMs. To our knowledge, previous works have even failed to improve small-scale LMs by incorporating FTRs. We believe KNIFE represents an important contribution to the field of learning from FTRs.

Reviewer comment (Experiments):
What is the upper-bound performance achievable by the small model when given the task input and free-text rationale together?

Author response:
The upper-bound performance is achieved by providing the task inputs and FTRs to models during both training and inference, which is reflected in the teacher model’s performance when using FTRs during inference as reported in Table 12. Compared to FT (I→O), the accuracy improvement is approximately 16% and 7% on OpenbookQA and StrategyQA, respectively, which is quite substantial. It's important to note that this performance should not be compared to results where the FTRs are not accessible during inference.

Reviewer comment (Evaluation):
Why does incorporating the free-text rationale during the teacher model's fine-tuning enhance the student model's performance? It's possible that the improvement shown in Figure 3(a) is due to the extra knowledge contained in the free-text rationale, which could be beneficial for the downstream tasks. This improvement, while not substantial, might not generalize well to scenarios with less overlap in knowledge or when using larger free-text rationales for teacher model fine-tuning. This raises questions about the broader applicability of this approach.

Author response:
Regarding the improvement of the student model's performance through the incorporation of FTRs during the teacher's training: Our intuition is that the teacher's finetuning aggregates reasoning knowledge from FTRs across all instances, subsequently embedding this knowledge in the teacher’s hidden states. KNIFE leverages these enriched hidden states as soft labels, enabling the distillation of reasoning knowledge from the teacher to the student via the collection of all soft labels. During the student model’s inference, its output is not explicitly conditioned on specific FTRs, but rather, it is implicitly influenced by the distilled reasoning knowledge embedded within the student's parameters.

We appreciate your insights regarding the influence of FTRs on KNIFE's performance. Indeed, since KNIFE distills knowledge from FTRs into the student language model, the student's performance is expected to depend on the quantity and quality of reasoning knowledge contained in the FTRs. You are correct in noting that if the FTRs are uninformative (in terms of providing task-level reasoning knowledge), KNIFE's ability to enhance model performance is limited. We have provided additional analysis and case studies in Section 4.6.

Reviewer comment (Evaluation):
The efficacy of the proposed method, KNIFE, seems marginal as the improvements are not statistically significant based on some results in Table 1. For instance, in the StrategyQA dataset, KNIFE’s performance is comparable to FT (I→RO), suggesting that simple fine-tuning of the language model with FTR and using the answer as the target yields results similar to KNIFE. Additionally, the experimental results suggest that distilling from T5-Large to T5-Base is less effective than from T5-Base to T5-base, which is weird.

Author response:
**(1) Performance Improvement**

We conducted the Wilcoxon rank-sum test between KNIFE and the best baseline for each OBQA/StrategyQA setting, and for both human-written and model-annotated FTR sources. KNIFE achieves statistically significant improvements (p<0.05) over the best baseline in all settings, except in the T5-Large→T5-Large on StrategyQA. By contrast, previous works, to our knowledge, have even failed to improve small-scale LMs by incorporating FTRs. We believe KNIFE represents an important contribution to the field of learning from FTRs.

**(2) Comparison between T5-Base→T5-Base and T5-Large→T5-Base**

We agree with your observation. It is indeed true that distilling from T5-Large to T5-Base (T5-Large→T5-Base) is less effective than distilling from T5-Base to T5-Base (T5-Base→T5-Base) in some settings, and this is not unusual. Our paper states an important implementation detail in Section 4.2: when the student and teacher models both use the same backbone (i.e., either T5-Base or T5-Large), the student’s parameters are initialized with the teacher’s. Additionally, T5-Base and T5-Large have different representation spaces in their hidden states, and since we perform distillation via these hidden states (as mentioned in footnote 4), there is an added complexity. We stated that we jointly train two linear projection layers to transform the hidden states for T5-Large→T5-Base distillation, while T5-Base→T5-Base distillation does not require this. These factors contribute to the increased difficulty in transferring reasoning knowledge from T5-Large to T5-Base compared to T5-Base to T5-Base. Although a T5-Large teacher model is expected to be stronger, the complexity of the distillation process also impacts performance. Due to the page limit, we will include a more comprehensive discussion on this topic in our next version.

Reviewer comment (Novelty):
Lately, there have been significant works [1,2,3] into distilling the reasoning ability of large LMs (e.g., Llama-2 or GPT-3.5-turbo) into smaller LMs. Given this context, how does the proposed method compare to existing approaches? In Appendix A.1., the authors posit that their method is advantageous when large-scale LMs lack reasoning abilities, a premise that seems out of sync with current trends in large language models.

Author response:
**(6) Relevant Works about CoT Distillation**

To clarify, our method, KNIFE, is not predicated on the absence of reasoning abilities in large-scale LMs. Instead, one advantage actually lies in its independence from LLMs. This allows KNIFE to work effectively without relying on the computational resources required for large LMs. Regarding the comparison with the approaches mentioned in your review and our Appendix A.1, it is important to note that these methods leverage large LMs to enhance smaller models, whereas KNIFE utilizes FTRs. This difference in approach makes fair comparison challenging.

In fact, we posit that relying on large LMs, as these methods do, is a stronger assumption compared to access to FTRs, as people can employ LLMs for generating FTRs. We have experimented with generating FTRs using a medium-sized LM, GPT-neox, and found KNIFE effective in this context. Due to computational constraints, exploring this further with larger LMs like ChatGPT and LLaMA-70B remains a goal for future research.

Most of the referenced works can be seen as utilizing LMs for FTR generation and training another model in I→RO. Our findings indicate that the I→RO is not effective with the FTRs we have experimented with. This observation leads us to believe that the **generation of high-quality FTRs** by LLMs is crucial in those approaches. As our work focuses on the **utilization of FTRs**, positioning it also as complementary to those approaches.

We will add more discussion on the points above in our next version.

[1] Narang et al. “WT5?! Training Text-to-Text Models to Explain their Predictions.” 2020. 

[2] Sun et al. “Investigating the Benefits of Free-Form Rationales.” 2022.

[3] Wiegreffe et al. “Measuring Association Between Labels and Free-Text Rationales.” 2021.

Reviewer comment (Novelty):
What is the advantage of this work compared to recent CoT distillation works mentioned in the second point of weaknesses?

Author response:
**(6) Relevant Works about CoT Distillation**

To clarify, our method, KNIFE, is not predicated on the absence of reasoning abilities in large-scale LMs. Instead, one advantage actually lies in its independence from LLMs. This allows KNIFE to work effectively without relying on the computational resources required for large LMs. Regarding the comparison with the approaches mentioned in your review and our Appendix A.1, it is important to note that these methods leverage large LMs to enhance smaller models, whereas KNIFE utilizes FTRs. This difference in approach makes fair comparison challenging.

In fact, we posit that relying on large LMs, as these methods do, is a stronger assumption compared to access to FTRs, as people can employ LLMs for generating FTRs. We have experimented with generating FTRs using a medium-sized LM, GPT-neox, and found KNIFE effective in this context. Due to computational constraints, exploring this further with larger LMs like ChatGPT and LLaMA-70B remains a goal for future research.

Most of the referenced works can be seen as utilizing LMs for FTR generation and training another model in I→RO. Our findings indicate that the I→RO is not effective with the FTRs we have experimented with. This observation leads us to believe that the **generation of high-quality FTRs** by LLMs is crucial in those approaches. As our work focuses on the **utilization of FTRs**, positioning it also as complementary to those approaches.

We will add more discussion on the points above in our next version.

[1] Narang et al. “WT5?! Training Text-to-Text Models to Explain their Predictions.” 2020. 

[2] Sun et al. “Investigating the Benefits of Free-Form Rationales.” 2022.

[3] Wiegreffe et al. “Measuring Association Between Labels and Free-Text Rationales.” 2021.

Reviewer comment (Experiments):
Why do the authors not include results with In only and out only in Table 1? I think this baseline is important to show the significance of using both objectives in KNIFE.

Author response:
**(5) Individual Contributions of KNIFE (In) and KNIFE (Out)**

To clarify, both KNIFE (In) and KNIFE (Out) are novel contributions of this work and not main baselines used for comparison. We refer readers to Table 2 (instead of Tables 8 or 9) for a comparison among KNIFE (In), KNIFE (Out), and KNIFE (In+Out). We observe that KNIFE (In+Out) generally achieves the highest performance, with a few exceptions. This indicates that valuable FTR knowledge can be distilled through both encoder and decoder hidden states, thus we recommend using both by default. Additionally, KNIFE (In) and KNIFE (Out) outperform all baselines in almost all cases. We included only KNIFE (In+Out) in the main table for clarity of presentation, aiming to convey the message that “KNIFE can outperform all baselines” without distracting readers. Then, a detailed comparison among the three KNIFE variants is provided in the following part of our paper. We will improve our presentation to better convey our messages in our next version.

Table 8 mainly presents results for KNIFE without the bottleneck design, which is not used for comparing the three KNIFE variants as the bottleneck is a critical design feature of KNIFE. Table 9 focuses on results for KNIFE with task loss, which is also not used in the comparison since incorporating task loss degrades KNIFE’s performance.