Paper: Unlocking the Potential of Knowledge Distillation: The Role of Teacher Calibration (ICLR-2024)

Reviewer comment (Experiments):
It is necessary to assess whether the proposed method works well even if various calibration studies are applied. And also they should be covered in the related works.

Author response:
**Weakness - It is necessary to assess whether the proposed method works well even if various calibration studies are applied. And also they should be covered in the related works.**

Thank you for the suggestion. We provided its results in official comments, namely **Additional Experiments 1: KD results with various calibration methods**. Following your recommendation, we conducted experiments with various calibration methods and applied these methods to the teacher model in our KD setup. The result is in **Additional Experiment 1** in the official comment. We observed that KD performance improves with the application of diverse calibration methods. 

This experiment further supports our claim that applying calibration methods to the teacher enhances KD performance. Additionally, it shows that this improvement is not limited to a specific method but extends across various calibration approaches. We're grateful for your suggestion, as it has enriched our research.

Reviewer comment (Theory):
In the introduction, the authors described that two things are important in the KD process. The first is to choose an effective distillation method, and the second is to choose the best one of teachers for students. But in real-world applications, under the deployment conditions, isn't it more typical to choose a student model that can learn the best teacher model?

Author response:
**Q1: choosing a student model in deployment condition.**

In real-world deployments on edge devices, it's typical to select a student model that satisfies constraints such as memory bounds or inference time. Selecting a student that best learns from a given teacher is a challenging approach because we don’t know which student model is the best student for the teacher until we perform KD. Additionally, determining the most suitable teacher model for a student is also difficult, as indicated by previous studies[1][2], which have demonstrated that a teacher model's higher accuracy does not necessarily translate to higher accuracy for the student. Therefore, our research focused on identifying the criteria for a good teacher under the fixed constraints of edge devices, assuming a set student model.

Reviewer comment (Experiments):
In Table 2, ResNet56 selected as the student model is a larger-sized model than a few teacher models (e.g., vgg19). Is this the right KD experiment?

Author response:
**Q3: Isn’t it ResNet56(student) is larger than some teacher model(e.g., VGG19)?**

It seems there might be a common misconception regarding the size of ResNet56. According to the ResNet original paper by He et al.[3] that introduced the ResNet architecture, ResNet20, ResNet32, ResNet44, ResNet56, and ResNet110 are smaller architectures designed for the CIFAR-10 dataset, while ResNet18, ResNet34, ResNet50, and ResNet101 are larger architectures designed for ImageNet, thus having a larger parameter size. The parameter sizes for each architecture are as follows:

| model | ResNet20 | ResNet32 | ResNet56 | ResNet110 | ResNet18 | ResNet34 | ResNet50 |
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| # params | 0.27M | 0.46M | 0.85M | 1.7M | 11.2M | 21.3M | 23.7M |

In the aspect of parameters, ResNet56 has 0.85M parameters, while the parameter size of VGG19 is 20.1M. The smallest model we used as a teacher model is WRN-28-2, which has 1.5M parameters. Hence, the student model in our experiments is always smaller than the teacher model. Detailed parameter sizes of other teacher models are in Appendix A section of our paper.

Reviewer comment (Evaluation):
After the proposed KD process, is there no need for one more calibration of the student model? I know this paper focuses on increasing the accuracy of the student model, but since it has already been learned from a calibrated teacher model, I think it is also another contribution of this paper if there is no need to conduct the calibration process for student model. Simply put, I would like to know the ECE and ACE of the student model that applied the calibration of the student model in Table 7 after applying the proposed method.

Author response:
**Q4: After the proposed KD process, is there no need for one more calibration of the student model?**

Thank you for suggesting a new insightful experiment. we performed an experiment to apply the calibration method to the student of Table 7, as you mentioned. We used temperature scaling for the calibration method. and for a fair comparison, we split the full-test set into the validation set and the test set with a 2:8 ratio. A validation set is used for calibration, and calibration metrics are reported with the test set. Then, the result is like below. Note that the detailed number differs from Table 7 due to the split test set, but it doesn’t change the tendency. 

|  |  | ShuffleNetV1 |  |  |
| --- | --- | --- | --- | --- |
| metric | ECE(%) | ECE_over(%) | ECE_under(%) | ACE(%) |
| before calibration | **8.6057** | **8.3768** | 0.2289 | **0.1068** |
| after calibration | 11.4309 | 11.3343 | **0.0966** | 0.1164 |


|  |  | WRN-16-2 |  |  |
| --- | --- | --- | --- | --- |
| metric | ECE(%) | ECE_over(%) | ECE_under(%) | ACE(%) |
| before calibration | **3.7519** | **3.0179** | 0.734 | **0.0936** |
| after calibration | 10.5391 | 10.5186 | **0.0205** | 0.0976 |

In this table, we can check calibration errors are better when we did not apply the calibration method. While asserting that no further calibration is needed after KD with a calibrated teacher would require more extensive experiments across a variety of models and calibration methods, our preliminary results suggest that using a calibrated teacher for KD results in a student model with robust calibration error. This finding can serve as a strong foundation for future research.

We hope this comment successfully clarifies your concerns about our paper. We look forward to further discussions and appreciate your consideration.

Reviewer comment (Novelty):
The proposed method is too simple. The temperature scaling has been proposed in the traditional KD. The technique contribution is small.

Author response:
The primary aim of our experiment utilizing temperature scaling was to demonstrate that reducing the teacher's calibration error can enhance KD performance.  Although temperature scaling itself is not our core contribution, our key contribution lies in demonstrating the effectiveness of reducing calibration error in boosting KD performance. The simplicity of our approach is advantageous, showing that even basic methods that reduce teacher calibration error can significantly improve KD. This finding suggests a foundation for future research to incorporate our core idea that lowering teacher calibration error can enhance KD methods even with more sophisticated methods. Because our research is based on a simple method, we think our results have the potential to influence the research field positively.

Furthermore, **Additional Experiment 1** in the official comment extends our result beyond just temperature scaling. By applying our hypothesis to widely-used calibration methods, namely mixup and vector scaling, we show performance enhancement on KD. Which suggests that there exists room for enhancement. Also, considering vector scaling requires additional validation set and mixup also enhances teacher’s accuracy, we can see that our methods enable fair comparison between baseline and ours.

Reviewer comment (Novelty):
Althogh this work achieves SOTA performance, it is based on previous SOTA method MLLD. All experiments are conducted on the image classification task. Object detection or other tasks are not considered.

Author response:
Continuing from the above response, applying calibration methods to enhance the performance of SOTA models demonstrates that reducing calibration error can improve KD performance, even in complexly designed methods. This indicates that calibration errors could be considered in designing future SOTA methodologies. Our paper lays the groundwork for future KD method development, considering the teacher model's calibration error in method design could lead to a more effective KD method. 

Our method of reducing the calibration error for the teacher is very simple to apply to any classification problem. Therefore, we believe that our method would also work for other tasks involving classification as their sub-task. These include segmentation tasks and two-stage detection tasks such as Faster-RCNN, which employ classifiers in their final stages. In our final version, we would like to provide these results.  

Thank you for suggesting an experiment on object detection. As you suggested, we conducted an additional experiment using our method for the object detection task. Since classifying objects is also included in object detection, and due to the simplicity of applying our method, we were able to easily apply our method to object detection by implementing temperature scaling on the teacher's classifier. The experiment results showed consistent performance improvements over the vanilla KD method. The results are as follows.

|  | mAP | AP50 | AP75 | APl | APm | APs |
| --- | --- | --- | --- | --- | --- | --- |
| Teacher: ResNet101 | 42.04 | 62.48 | 45.88 | 54.60 | 45.55 | 25.22 |
| Student: ResNet50 | 37.93 | 58.84 | 41.05 | 49.10 | 41.14 | 22.44 |
||||||||
| KD | 38.35 | 59.41 | 41.71 | 49.48 | 41.80 | 22.73 |
| KD+Ours | **39.04** | **60.74** | **42.21** | **50.38** | **42.35** | **22.88** |

|  | mAP | AP50 | AP75 | APl | APm | APs |
| --- | --- | --- | --- | --- | --- | --- |
| Teacher: ResNet101 | 42.04 | 62.48 | 45.88 | 54.60 | 45.55 | 25.22 |
| Student: ResNet18 | 33.26 | 53.61 | 35.26 | 43.16 | 35.68 | 18.96 |
||||||||
| KD | 33.97 | 54.66 | 36.62 | 44.14 | 36.67 | 18.71 |
| KD+Ours | **34.65** | **55.99** | **36.90** | **45.30** | **37.26** | **20.00** |

|  | mAP | AP50 | AP75 | APl | APm | APs |
| --- | --- | --- | --- | --- | --- | --- |
| Teacher: ResNet50 | 37.93 | 58.84 | 41.05 | 49.10 | 41.14 | 22.44 |
| Student: MobileNet-V2 | 29.47 | 48.87 | 30.90 | 38.86 | 30.77 | 16.33 |
||||||||
| KD | 30.13 | 50.28 | 31.35 | 39.56 | 31.91 | 16.69 |
| KD+Ours | **31.90** | **52.81** | **33.49** | **41.23** | **34.05** | **18.29** |

The experiment was conducted following the commonly used object detection settings in previous knowledge distillation papers(Chen et al.[1], Zhao et al.[2], Jin et al.[3]). We conducted our experiment using the COCO2017 dataset and employed Faster RCNN, testing with Teacher-Student pairs of ResNet110-ResNet50, ResNet110-ResNet18, and ResNet50-MobileNetV2. 

Despite the simplicity of our method, the results show steady enhancements in performance. These results indicate that our main claim is not limited to classification tasks but is also applicable to object detection tasks. This additional experiment strongly supports our claim, so we will include it in the appendix of the final version. Thank you for suggesting this additional experiment. It has significantly strengthened the generality of our claim.

We hope this additional experiment effectively addresses your concerns. Thank you for your insightful comments, and we look forward to further discussion.

\
[1] Chen, P., Liu, S., Zhao, H., & Jia, J. (2021). Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5008-5017).

[2] Zhao, B., Cui, Q., Song, R., Qiu, Y., & Liang, J. (2022). Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition (pp. 11953-11962).

[3] Jin, Y., Wang, J., & Lin, D. (2023). Multi-Level Logit Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 24276-24285).