Paper: Knowledge Distillation Based on Transformed Teacher Matching (ICLR-2024)

Reviewer comment (Novelty):
The paper could benefit from addressing the lack of novelty by acknowledging that techniques such as R´enyi or f divergence, temperature scaling, and logits normalization have already been widely used in knowledge distillation. For example, Information Theoretic Representation Distillation (BMVC) employed R´enyi divergence for standard distillation, and AlphaNet (ICML2021) utilized the f divergence to distill different sub-networks. Moreover, this method is likely already considered in the distiller's search work (KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs, NeurIPS-2023).

Author response:
One of the major contributions of our work is showing that dropping the temperature on the student side leads to a better version of KD, dubbed as TTM in the paper. In our work, R´enyi entropy is not something deliberately involved by us to build our own method. Instead, it naturally emerges when we remove the temperature on the student side, which explains why dropping the temperature on the student side is beneficial. Therefore, although similar concepts appear in both our work and the works you listed, the context is totally different. 

To further alleviate your concern, we next show that how our work is different from the works you mentioned one by one as follows: (1) Information Theoretic Representation Distillation is a feature-based distillation method, while our proposed method is logits-based, with only minimal modifications to the original KD. (2) AlphaNet utilizes $\alpha$-divergence in distillation instead of the conventional KL divergence. However, we stick to the KL divergence used in the original KD. (3) We acknowledge that KD-Zero does search for distillers in a pretty wide searching space. However, according to Table 1 in the paper of KD-Zero, we firmly believe that the adaptive weighting term $U$ proposed in our work is not considered in the searching space of KD-Zero.

Actually, based on our literature review, the most similar work to us is Self-Distillation as Instance-Specific Label Smoothing (NIPS 2020), and we have already carefully addressed this work in Subsection 2.3.

Based on the above discussion, we have full confidence in the novelty of our work.

Reviewer comment (Experiments):
To strengthen the paper's findings, it is important to validate the proposed method on downstream tasks such as object detection and segmentation. Including evaluation results on these tasks will demonstrate the practical effectiveness and applicability of the proposed method. Additionally, providing more examples and visualizations will enhance the readers' understanding of how the method works and its impact on the learning process.

Author response:
The major contribution of this paper is to show that it is better off to drop the temperature T from the student side. This is well supported by both theoretical analysis and empirical results. This is a new understanding about KD and, together with the statistical perspective of KD, offers a new explanation of why KD helps. On top of this contribution, we further introduce a sample-adaptive coefficient to KD, yielding the method called WTTM. While TTM can be regarded as a correction of KD, WTTM is new and achieves the state-of-the-art performance. In addition, WTTM has essentially the same computation complexity as KD does. It can be applied wherever KD is applicable.  In the literature, KD has already been widely used on downstream tasks such as object detection and segmentation. So, there’s no reason to believe our proposed methods are not applicable to them. Of course, how much gain TTM and WTTM would offer in comparison with KD and other existing distillers on those tasks needs to be found out in future work.

As for examples and visualizations, please refer to Figure 1 in our paper, where we visualize the regularization effect of our proposed objectives compared to vanilla KD, and Figure 2, where we visualize the behavior of the average $D(p^t_T||q)$ for TTM and WTTM throughout the learning process, which helps explain why WTTM performs better than TTM.

Reviewer comment (Novelty):
Furthermore, it is essential to incorporate a thorough discussion of relevant KD-related studies, including Self-Regulated Feature Learning via Teacher-free Feature Distillation (ECCV2022), NORM: Knowledge Distillation via N-to-One Representation Matching (ICLR2023), Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer (NIPS2022), DisWOT: Student Architecture Search for Distillation Without Training (CVPR2023), and Automated Knowledge Distillation via Monte Carlo Tree Search (ICCV2023). This discussion will help position the proposed approach within the existing literature, establish connections, and provide valuable insights for potential comparisons.

Author response:
Please refer to appendix A.7 of our revised paper, where we cite all these papers you mentioned and provide a thorough discussion of relevant KD-related studies.

Reviewer comment (Experiments):
Some references and comparisons are missing:

[1] Knowledge distillation from a stronger teacher.

[2] From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels.

[3] Curriculum Temperature for Knowledge Distillation.

[4] VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale.

Author response:
Please refer to appendix A.7 in our revised paper, where we include the 4 papers you mentioned as references. As for comparison, in the revised version, we include the first 3 papers you mentioned as benchmarks in Table 5, and our method is still achieving outstanding performance. However, there are no overlapping experimental settings between our work and VanillaKD to form a fair comparison, and it's infeasible for us to follow the settings in VanillaKD to generate some new results in such limited time based on our computational resources, so we didn't include VanillaKD into comparison.

Reviewer comment (Experiments):
When temperature=1, is TTM the same as the original KD? In some papers, the temperature on ImageNet is actually 1.0.

Author response:
Yes, TTM (and WTTM) is the same as the original KD when temperature=1, and we know that the commonly used temperature on ImageNet is 1.0 for the original KD. However, with temperature dropped on the student side, the effect of the temperature in distillation is changed, so it's reasonable for TTM (and WTTM) to have a different optimal temperature from the original KD. According to our experiments, the optimal $\gamma$ for our method is around 0.8 on ImageNet (see Table 10 in our revised paper), which is equivalent to temperature=1.25. Consequently, our method is different from the original KD on ImageNet.

Reviewer comment (Experiments):
Could TTM still achieve better performance for larger models (e.g. DeiT-T or DeiT-S)? VanillaKD shows under strong training settings, the original KD also performs well.

Author response:
Both TTM and WTTM have essentially the same computation complexity as KD does. They can be applied wherever KD is applicable. In the literature, KD has already been widely used in improving the performance of larger models such as the transformer models you mentioned. So, there’s no reason why TTM and WTTM cannot.

In the interest of time and also due to our limited computational resources, we tried our best in the last a couple of days and managed to generate one result for DeiT-T on ImageNet without tuning any hyperparameters. The result is shown in the table below.

|Teacher|Student|KD|WTTM|
|:---:|:---:|:---:|:---:|
|RegNetY-16GF (82.9) |DeiT-T (72.2)|72.2|72.7|

Following the training strategy of the original DeiT paper [1], KD is not able to improve the Top-1 accuracy (%) performance of DeiT-T on ImageNet (72.2%), while WTTM can improve the performance to 72.7%. This small example at least shows the potential of WTTM for larger models.

Note that the training strategy in [1] is also quite strong, including AdamW optimizer, cosine LR decay, label smoothing, stochastic depth, repeated augmentation, Rand Augment, mixup, cutmix, random erasing and so on, with carefully tuned hyperparameters for each ingredient. Therefore, we believe that our method is still better than the original KD even in strong training settings.

[1] Touvron et al., "Training data-efficient image transformers & distillation through attention", ICML 2021.

Reviewer comment (Novelty):
It's better to provide a detailed summary and comparison of the latest related works.

Author response:
**Response**: Please refer to appendix A.7 in our revised paper, where we provide a summary of the latest related works. As for comparison, in the revised version, we include more benchmarks in Table 5 including many works published last year and this year, which are DKD [1], DIST [2], KD++ [3], NKD [4], CTKD [5], and KD-Zero [6], and our method is still achieving outstanding performance.

[1] Zhao et al., "Decoupled knowledge distillation", CVPR 2022.  
[2] Huang et al., "Knowledge distillation from a stronger teacher", NIPS 2022.  
[3] Wang et al., "Improving knowledge distillation via regularizing feature norm and direction", Under review of ICLR 2024.  
[4] Yang et al., "From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels", ICCV 2023.  
[5] Li et al., "Curriculum Temperature for Knowledge Distillation", AAAI 2023.  
[6] Li et al., "KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs", NIPS 2023.

Reviewer comment (Experiments):
It's also more convincing to show results on transformer models such as ViT.

Author response:
**Response**: Both TTM and WTTM have essentially the same computation complexity as KD does. They can be applied wherever KD is applicable. In the literature, KD has already been widely used in improving the performance of transformer models. So, there’s no reason why TTM and WTTM cannot.

In the interest of time and also due to our limited computational resources, we tried our best in the last a couple of days and managed to generate one result for one of the smallest transformer models, namely DeiT-tiny, on ImageNet without tuning any hyperparameters. The result is shown in the table below.

|Teacher|Student|KD|WTTM|
|:---:|:---:|:---:|:---:|
|RegNetY-16GF (82.9) |DeiT-tiny (72.2)|72.2|72.7|

Following the training strategy of the original DeiT paper [7], KD is not able to improve the Top-1 accuracy (%) performance of DeiT-tiny on ImageNet (72.2%), while WTTM can improve the performance to 72.7%. This small example at least shows the potential of WTTM for transformer models.

[7] Touvron et al., "Training data-efficient image transformers & distillation through attention", ICML 2021.