Paper: Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR-2024)

Reviewer comment (Experiments):
The experimental results show improved performance on the HIL and Criteo datasets. However, the paper does not clarify if this is due to the adopted sampling method. The oversampling of the minority class as a preprocessing step is mentioned, but its direct influence is not explored.

Author response:
**A1:** We employ oversampling when the dataset is imbalanced, which is a widely accepted practice in the research community, especially for classification tasks. To analyze the impact of oversampling on performance, we compare our TGeo-KD with five other baselines across varying imbalance ratios on the HIL and Criteo datasets. The imbalance ratio $\rho$ is defined as the proportion between the sample sizes of the most prevalent class and the least prevalent class, where $\rho=1$ represents a balanced setting. As shown in the tables below (Table 11 in the manuscript), we calculate the average classification accuracy (with standard deviation) over 5 repeated runs. TGeo-KD consistently outperforms all the baselines (e.g., the improved accuracy of 2.24% and 3.15% over the best baseline ADA-KD with $\rho=1$ and without oversampling on HIL dataset, respectively), which underscores the robustness of our TGeo-KD on imbalanced datasets. We have also provided the results in Appendix A.4 of the manuscript.

*Table: Top-1 ACC on HIL.*
|Method|$\rho=1$|$\rho=5$ |w/o oversampling|
|:-:|:-:|:-:|:-:|
|Vanilla KD|87.55±0.56|81.72±0.66|72.07±0.71|
|ANL-KD|87.27±0.23|81.33±0.30|71.82±0.29|
|ADA-KD|90.15±0.34|83.49±0.22|75.23±0.28|
|WLS-KD|90.05±0.28|82.83±0.24|74.77±0.27|
|RW-KD|89.40±0.45|82.22±0.59|73.91±0.58|
|**TGeo-KD**|**92.39±0.49**|**85.65±0.27**|**78.38±0.21**|

*Table: Top-1 ACC on Criteo.*
|Method|$\rho=1$|$\rho=5$| w/o oversampling |
|:-:|:-:|:-:|:-:|
|Vanilla KD|71.08±0.48|67.32±0.56|61.01±0.39|
|ANL-KD|72.71±0.35|68.03±0.38|62.18±0.32|
|ADA-KD|72.15±0.33|67.74±0.42|61.35±0.46|
|WLS-KD|75.30±0.38|71.21±0.55|64.52±0.43|
|RW-KD|75.05±0.44|70.58±0.37|63.90±0.38|
|**TGeo-KD**|**77.80±0.29**|**74.09±0.32**|**67.77±0.30**|

Reviewer comment (Experiments):
In Section 4.3, this paper analyzes the effectiveness of incorporating ST by comparing the learned fusion ratio distributions across four scenarios. With ST, the proposed TGeo-KD reduces the fusion ratio on L_kd, thereby encouraging the student to acquire more knowledge from ground truths when the teacher predictions are incorrect on samples (e.g., outliers). However, more experiments should be conducted to investigate that the exploitation of inter-sample relations on these incorrectly predicted samples does not contribute significantly to decreased fusion ratios.

Author response:
**A2:** To analyze the impact of prediction discrepancy $\mathcal{ST}$ and inter-sample relations $\Delta^{\mathcal{S\bar{T}G}}$ on the fusion ratios for incorrectly predicted samples, we have added the following table (Table 14 in the main manuscript), indicating the mean and standard deviation of the fusion ratios across these samples when learning fusion ratio with and without the consideration of $\mathcal{ST}$ and $\Delta^{\mathcal{S\bar{T}G}}$. When considering both $\mathcal{ST}$ and $\Delta^{\mathcal{S\bar{T}G}}$, our TGeo-KD demonstrates the fusion ratios below 0.3 for incorrectly predicted samples. This suggests that the student acquires more knowledge from the ground truth, leading to decreased fusion ratios on $\mathcal{L}^{KD}$. Furthermore, in comparison to solely exploiting $\Delta^{\mathcal{S\bar{T}G}}$ when learning fusion ratios, $\mathcal{ST}$ contributes more effectively to the decreased fusion ratios across incorrectly predicted samples. We have also provided the results in Appendix A.4 of the manuscript.

|Method|$\mathcal{SG+TG}$|$\mathcal{ST}$|$\Delta^{\mathcal{S\bar{T}G}}$|Final fusion ratio on incorrectly predicted samples|
|:-:|:-:|:-:|:-:|:-:|
|TGeo-KD|✓|-|-|0.58±0.29|
|TGeo-KD|✓|✓|-|0.32±0.15|
|TGeo-KD|✓|-|✓|0.43±0.18|
|TGeo-KD|✓|✓|✓|0.12±0.07|

Reviewer comment (Evaluation):
The fusion ratio is learned through a neural network, which is unclear to explain how the learning process unfolds, such as which samples are assigned larger/smaller fusion ratios.

Author response:
**A3:** Section 4.3 and Section 4.4 provide the analysis regarding the learning process of fusion ratios across normal samples and outliers. Specifically, Figure 4 shows the final fusion ratio distributions among sample-wise based baselines and our TGeo-KD on CIFAR-100. In comparison to the two baselines, our TGeo-KD increases the final fusion ratios for normal samples within the range of 0.4 and 0.6, which signifies the effective guidance of the student by valuable supervision signals from both the ground truth and the teacher. Additionally, as the teacher may make incorrect predictions on outliers, imparting misleading knowledge to the student, our TGeo-KD decreases the final fusion ratios below 0.3 for outliers. This indicates that the student is encouraged to learn more informative knowledge from the ground truth, resulting in decreased fusion ratios on $\mathcal{L}^{KD}$. 

Table 4 illustrates the average fusion ratios during the training. With the incorporation of $\mathcal{ST}$, our TGeo-KD consistently reduces the fusion ratios for incorrectly predicted samples as the training progresses. In cases where the teacher predicts accurately, the fusion ratio is increased in scenarios with a significant discrepancy between the student and teacher. This implies that the student is expected to emulate the teacher more closely, as the teacher possesses greater potential for offering valuable knowledge. On the contrary, when the discrepancy is smaller, the student is encouraged to rely more on the ground truth, resulting in a decline in the fusion ratio.

Reviewer comment (Theory):
Why do the authors choose to model the relation in Euclidean space?

Author response:
**A4:** Our proposed method can model the sample-wise trilateral geometric relations in various spaces, including hyperbolic space and Euclidean space. Considering that Euclidean distance is a commonly used metric to model the relations among samples in knowledge distillation studies [6,7,8], we take Euclidean space as an illustrative example to represent the sample-wise trilateral geometry encompassing the student predictions, teacher predictions, and ground truth.

Reviewer comment (Experiments):
Given that this paper proposes an adaptive learning method for sample-wise fusion ratio, does the proposed method demonstrate consistent performance across different numbers and categories of samples?

Author response:
**A5:** Our method showcases robust adaptability and generalization across a spectrum of datasets, each with its own unique characteristics in terms of sample categories and numbers.
* ***CIFAR-100***: This dataset is a staple in the realm of knowledge distillation, encompassing 60,000 color images categorized into 100 distinct classes.  CIFAR-100 includes a wide variety of objects, animals, and scenes, standing as an excellent benchmark for the development and testing of models and algorithms. In comparison to the best baseline WLS-KD [5],  our method consistently exhibits performance enhancements of 1.37% and 1.26% across homo- and hetero-architectures knowledge distillation scenarios.  
* ***ImageNet***: ImageNet is a large-scale dataset commonly used for image classification, containing millions of labeled images spanning thousands of categories, such as animals and scenes, etc.  In comparison to the best baseline WLS-KD [5], our method consistently exhibits performance enhancements of 1.10% and 0.94% across homo- and hetero-architectures knowledge distillation scenarios.  
* ***HIL***: Primarily used for conceptualizing attack detection in industrial IoT systems, HIL emerges from a real-world transmission network, offering a tabular structure with 128 diverse features. Notably, it covers three event categories—normal operation (label 0), natural fault (label 1), and malicious attack (label 2) —with an inherent data imbalance among these classes (4,405, 18,309, and 55,663, respectively). In comparison to the best baseline ADA-KD [4], our method consistently exhibits performance enhancements of 2.24%. 
* ***Criteo***: This dataset is a benchmark for CTR prediction challenges. With a whopping 45 million user interactions, it features a mix of integer and categorical data attributes. It contains the majority (74.25%) of samples belonging to the "no-click" class (0) and a minority (25.75%) to the "click" class (1). In comparison to the best baseline WLS-KD [5], our method consistently exhibits performance enhancements of 2.50%.

Reviewer comment (Theory):
Is there a specific reason to formulate this work into bilevel optimization? Will it lead to less efficient training? The motivation is not quite clear to me.

Author response:
**A6:** Our ultimate objective is to find the optimal sample-wise ratios $\alpha_{i}=f_{\omega}(\Delta_{i})$ that enable the student network parameterized by  to generalize well on test data. This naturally implies a bilevel optimization problem [1] with  as the outer level variable and  as the inner loop variable. 

In Appendix A.5 of the manuscript, the computational cost comparison of our proposed TGeo-KD with the baselines is provided in the table below. We run experiments on CIFAR-100 and report the mean training time (in second) on a batch of 128 images per iteration. The student network is ResNet-32. The experiment is conducted on one NVIDIA RTX-3090 GPU. As shown in the table below (Table 15 in the manuscript), TGeo-KD increases marginal training costs with significant accuracy improvement compared to other methods.

*Table: Mean training time (in seconds) on a batch for CIFAR-100 per iteration.*
||Vanilla KD [2]|ANL-KD [3]|ADA-KD [4]|WLS-KD [5]|**TGeo-KD (ours)**|
|:-:|:-:|:-:|:-:|:-:|:-:|
|Training time (sec)| 0.095|0.095|0.101| 0.097|**0.097**|
|ACC (%)|74.07|72.08|71.45|75.46|**76.83**|

Ref:
* [1] Franceschi, et al. Bilevel programming for hyperparameter optimization and meta-learning. ICML, 2018.
* [2] Hinton et al. Distilling the knowledge in a neural network. NeurIPS, 2014. 
* [3] Clark et al. BAM! born-again multi-task networks for natural language understanding. ACL, 2019.
* [4] Lukasik et al. Teacher’s pet: understanding and mitigating biases in distillation. TMLR, 2022. 
* [5] Zhou et al. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. ICLR, 2021.
* [6] Liu et al. Knowledge distillation via instance relationship graph. CVPR, 2019.
* [7] Park et al. Relational knowledge distillation. ICCV, 2019.
* [8] Romero et al. Fitnets: Hints for thin deep nets. ICLR, 2015.

Reviewer comment (Experiments):
How do the outliers hurt KD? As a key motivation, the corresponding empirical evidence seems lacking.

Author response:
**A1:** Given the substantial divergence between outliers and normal training data, the teacher network may perform poorly on these outliers, even with high absolute values of confidence margin. As illustrated in our motivation experiment presented in Figure 1 of the manuscript, the subset $D’$ includes samples on which the teacher makes inaccurate predictions. Notably, an increased fusion ratio  degrades the student’s performance across all $g’$ groups $(D’={g’_1, g’_2, g’_3, g’_4, g’_5})$. This observation underscores the risk of the teacher transferring misleading knowledge to the student when relying on the teacher’s prediction as the supervisory signal, ultimately leading to a decline in the student’s performance. Therefore, it is crucial to determine an appropriate sample-wise fusion ratio to effectively guide the student training process, particularly when addressing outliers.

Reviewer comment (Novelty):
In addition, the teacher model indeed may underperform on the outliers. According to previous discussion, directly decreasing the $\alpha$ also can solve this issue. Thus, why to introduce the inter-sample relations? This motivation needs to be further claified.

Author response:
**A2:** We argue that directly decreasing the same alpha value across all samples oversimplifies the varied impact of different outliers. Each outlier can uniquely influence the learning process, and a blanket reduction in alpha fails to address these nuances. We lack clarity on the extent of alpha value reduction needed for different samples without other guidance and signals.

Our approach, which factors in inter-sample relations, aligns more closely with the principles of Knowledge Distillation, which is to understand the dynamics among the teacher, student, and ground truth. By examining these relationships, we can adjust the alpha value more precisely, tailoring the learning process to accommodate the diverse nature of outliers. Adopting this sophisticated, context-aware strategy is crucial for optimizing model performance, ensuring each outlier's impact is individually assessed and addressed, rather than resorting to a generalized, less effective solution.

Reviewer comment (Experiments):
Evaluating this method on real OOD datasets, like DomainNet, will be more convicing.

Author response:
**A3:** To better show the effectiveness of our method on Out-of-Distribution Detection (OOD) dataset, we conduct experiments on DomainNet. DomainNet [1] is a domain adaptation dataset with six domains (including Clipart, Real, Sketch, Infograph, Painting, and Quickdraw) and 0.6 million images distributed across 345 categories. In the experiment setup, we adopt a standard leave-one-domain-out manner, and train our student (teacher: ResNet-50 and student: ResNet-18) on the training set of the source domains and evaluate the trained student on all images of the held-out target domain. All results are reported in terms of average classification accuracy (with standard deviation) over 5 repeated runs. As reported in the table below (Table 8 in the manuscript), our TGeo-KD can outperform all the relevant KD-based baseline methods. We have also added the experimental results in Appendix A.3 of the manuscript. 

*Table: Classification accuracy (\%) on DomainNet.*


||Clipart|Real|Sketch|Infograph|Painting|Quickdraw|Avg|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Vanilla KD|50.6±0.2|55.1±0.3|40.7±0.3|17.9±0.2|42.5±0.4|9.8±0.3|34.3±0.3|
|ANL-KD|52.5±0.4|57.0±0.3|41.9±0.3|19.6±0.2|44.6±0.3|11.3±0.3|37.8±0.3|
|ADA-KD|53.6±0.2|57.9±0.4|42.3±0.3|20.5±0.2|45.1±0.3|12.4±0.4|38.7±0.4|
|WLS-KD|53.1±0.4|57.7±0.3|42.8±0.2|21.1±0.1|45.3±0.2|12.0±0.2|38.5±0.3|
|RW-KD|51.3±0.3|56.2±0.3|41.2±0.2|18.8±0.3|43.7±0.2|10.6±0.3|37.1±0.2|
|**TGeo-KD**|**55.2**±0.3|**59.3**±0.4|**44.6**±0.1|**21.7**±0.3|**46.9**±0.3|**14.1**±0.3|**40.2**±0.3|

Reviewer comment (Theory):
As they state that Eq. 4 is an implicit function of ω as θ* depends on ω, how to perform alternately optimization?

Author response:
**A5:** To conduct the optimization for the bilevel optimization problem in Eq. 4, we speed up the training by using an approximation that has been validated in [5,6,7]. The key idea is to define a proxy function to link $\omega$ to the outer objective as follows: $\tilde{\theta}(\omega) := \theta - \alpha \nabla_{\theta}\mathcal{J}^{\text{inner}}(\theta, \omega)$, where $\alpha$ is the learning rate.

Then the iterative optimization process from step $t$ to step $t+1$ can be illustrated as follows:
* $\theta$ update: $\theta^{t+1} \leftarrow OPT_{\theta}\Big(\theta^t, \nabla_{\theta^t}\mathcal{J}^{\text{inner}}(\theta^t, \omega^t)\Big)$
* Proxy: $\tilde{\theta}^{t+1}(\omega^t) := \theta^t - \alpha \nabla_{\theta^t}\mathcal{J}^{\text{inner}}(\theta^t, \omega^t)$
* $\omega$ update: $\omega^{t+1} \leftarrow OPT_{\omega}\Big(\omega^t, \nabla_{\omega^t}\mathcal{J}^{\text{outer}}(\tilde{\theta}^{t+1}(\omega^t))\Big)$

Ref:
* [1] Peng et al. “Moment matching for multi-source domain adaptation.” ICCV, 2019.
* [2] Liu et al. "Knowledge distillation via instance relationship graph." CVPR, 2019.
* [3] Park et al. "Relational knowledge distillation." ICCV, 2019.
* [4] Romero et al. “Fitnets: Hints for thin deep nets.” ICLR, 2015
* [5] Liu et al. "Darts: Differentiable Architecture Search." ICLR, 2019. 
* [6] Chen et al. "λOpt: Learn to Regularize Recommender Models in Finer Levels." KDD, 2019.
* [7] Ma et al. "Probabilistic Metric Learning with Adaptive Margin for Top-K Recommendation." KDD, 2020.

Reviewer comment (Experiments):
My main concern is related to the impact of the proposed solution. Like most distillation papers conclusions are drawn based on CIFAR-100 experiments using same network architecture. Claims made by the authors could substantiate better if they are shown to hold on bigger datasets like ImageNet.

Author response:
**Q1: Experiment on ImageNet for higher impact.**

**A1:** In Section 4.2, we conduct the experiments on ImageNet to further demonstrate the effectiveness of our approach on larger datasets. As depicted in the table below (Table 2 in the manuscript), TGeo-KD consistently outperforms all competing baselines. Compared with the best baseline WLS-KD [1], TGeo-KD exhibits the performance improvement of 1.10% when the teacher (ResNet-34) and student (ResNet-18) share the same architecture style. In a hetero-architecture KD experiment with ResNet-50 and MobileNetV1 as teacher and student respectively, our method realizes an improvement of 0.94%.

*Table: Top-1 and Top-5 classification accuracy on ImageNet. We re-implemented the methods denoted by \* and used the author-provided or author-verified results for the others.*

*Teacher: ResNet-34 → Student: ResNet-18*

| Method| Top-1 ACC| Top-5 ACC|
|:-:|:-:|:-:|
|Teacher|73.31|91.42|
|Student|69.75|89.07|
|AT|71.03|90.04|
|NST|70.29|89.53|
|FSP|70.58|89.61|
|RKD|70.40|89.78|
|Overhaul|71.03|90.15|
|CRD|71.17|90.13|
|Vanilla KD|70.67|90.04|
|ANL-KD\*|71.83 ±0.22|90.21 ±0.26|
|ADA-KD\*|71.96 ±0.17|90.45 ±0.21|
|WLS-KD| 72.04*|*90.70*|
|RW-KD\*|70.62 ±0.22|89.76 ±0.15|
|**TGeo-KD**|**72.89 ±0.15**|**91.80 ±0.04**|

*Teacher: ResNet-50 → Student: MobileNetV1*

|Method|Top-1 ACC|Top-5 ACC|
|:-:|:-:|:-:|
|Teacher|76.16|92.87|
|Student|68.87|88.76|
|AT|70.18|89.68|
|FT|69.88|89.50|
|AB|68.89|88.71|
|RKD|68.50|88.32|
|Overhaul|71.33|90.33|
|CRD|69.07|88.94|
|Vanilla KD|70.49|89.92|
|ANL-KD\*|70.40±0.15|89.25±0.22|
|ADA-KD\*|71.08±0.24|90.17±0.16|
|WLS-KD|*71.52*|*90.34*|
|RW-KD\*|70.15±0.16|89.40 ±0.19|
|**TGeo-KD**|**72.46±0.14**|**90.95 ±0.17**|

Reviewer comment (Experiments):
Tasks other than classification should be considered (e.g. detection). Experiments on HIL and Criteo datasets are of low impact.

Author response:
**Q2: Other tasks and impact.**

**A2:** While existing knowledge distillation works have demonstrated success in image classification tasks, their applications in real-world industrial systems, including cyber-physical systems and recommender systems, still remain limited. Hence,  to illustrate the broad applicability of our approach across diverse application scenarios, we conduct extensive experiments spanning three tasks: CIFAR-100 and ImageNet for image classification, HIL for attack detection in cyber-physical systems, and Criteo for click-through prediction in recommender systems. 

Moreover, the selection of three tasks considers the representation of diverse domains characterized by different data natures. CIFAR-100 and ImageNet serve as widely recognized benchmarks for image-based knowledge distillation. HIL, as the representative of time series data, captures the dynamics of real-world industrial systems, thus presenting unique challenges different from image datasets [1-3]. Criteo is vital for its reflection of human behavior through its vast collection of feature values and click feedback for display ads, offering a distinct challenge relative to the others [4-6]. 

In addition to the experiments included in the current manuscript, we conduct additional experiments on DomainNet dataset [7], to demonstrate the generalization ability of our method in the Out-of-Distribution Detection (OOD) task. DomainNet [7] is a domain adaptation dataset with six domains (including Clipart, Real, Sketch, Infograph, Painting, and Quickdraw) and 0.6 million images distributed across 345 categories. In the experiment setup, we adopt a standard leave-one-domain-out manner, and train our student (teacher: ResNet-50 and student: ResNet-18) on the training set of the source domains and evaluate the trained student on all images of the held-out target domain. All results are reported in terms of average classification accuracy (with standard deviation) over 5 repeated runs. As reported in the table below (Table 8 in the manuscript), our TGeo-KD can outperform all the relevant KD-based baseline methods. We have also added the experimental results in Appendix A.3 of the manuscript. 

*Table: Classification accuracy (\%) on DomainNet.*

|Method|Clipart|Real|Sketch|Infograph|Painting|Quickdraw|Avg|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Vanilla KD|50.6 ±0.2| 55.1 ±0.3 | 40.7 ±0.3 | 17.9 ±0.2 | 42.5 ±0.4 | 9.8 ±0.3  | 34.3 ±0.3 |
|ANL-KD|52.5 ±0.4| 57.0 ±0.3 | 41.9 ±0.3 | 19.6 ±0.2 | 44.6 ±0.3 | 11.3 ±0.3 | 37.8 ±0.3 |
|ADA-KD|53.6 ±0.2| 57.9 ±0.4 | 42.3 ±0.3 | 20.5 ±0.2 | 45.1 ±0.3 | 12.4 ±0.4 | 38.7 ±0.4 |
|WLS-KD|53.1 ±0.4 | 57.7 ±0.3 | 42.8 ±0.2 | 21.1 ±0.1 | 45.3 ±0.2 | 12.0 ±0.2 | 38.5 ±0.3 |
|RW-KD|51.3 ±0.3| 56.2 ±0.3 | 41.2 ±0.2 | 18.8 ±0.3 | 43.7 ±0.2 | 10.6 ±0.3 | 37.1 ±0.2 |
|TGeo-KD|**55.2** ±0.3 | **59.3** ±0.4 | **44.6** ±0.1 | **21.7** ±0.3 | **46.9** ±0.3 | **14.1** ±0.3 | **40.2** ±0.3 |