Paper: Knowledge Distillation with Perturbed Loss: From a Vanilla Teacher to a Proxy Teacher (ICLR-2024)

Reviewer comment (Experiments):
Only the case of no known ground-truth targets (i.e. unlabeled distillation data) is considered, but this is only mentioned briefly on page 3, and should be made clear much earlier. Either way, it would be relevant to compare the proposed procedure to incorporating a weighted ground-truth loss as is common in many applications, as this also shifts the matching distribution towards the ground-truth distribution.

Author response:
Thank you for your valuable feedback on our paper. To emphasize the setting, we have used a distinctive paragraph in Sec. 2 Preliminaries, initiated with a bold, non-indented sentence to capture the reader's attention immediately. Furthermore, we have provided a more detailed explanation in the footnote on page 3 as to why this setting is reasonable. Under this setting, incorporating a weighted ground-truth loss is beyond our scope because in real-world scenario, the large teacher models (e.g., ChatGPT and GPT4) provide access only to their results and/or APIs rather than their initial training datasets with ground-truth targets. 

We admit that this could be made clear and earlier, we would like to mention it in the introduction part in the next version.

Reviewer comment (Writing):
There is no common consensus on the aim of distillation procedures; whether it is to match the teacher as well as possible (see e.g. [1]) or to get the best-performing student on the ground-truth data (see e.g. [2]). The success of self-distillation is due to not perfectly matching the teacher, and such nuance should be evident from your introduction. Currently, it merely states that matching an imperfect teacher is suboptimal. Also, what is your definition of bias here?

Author response:
Our primary goal is to optimize for the best student performance on downstream tasks. The 'bias' mentioned in our work refers to the distribution discrepancy between the teacher's predictions and the ground truth. We observe (and empirically verify) this distribution gap is positively correlated with the student model downstream task performances. Thus, we design our perturbation methods to reduce this distribution gap and thus to improve the student model's performance on downstream tasks.

Reviewer comment (Experiments):
You argue in the introduction that a grid search for the temperature is computationally expensive, but your proposed method still requires a search over the perturbation coefficients. Also, how does the search-time scale with \(M\), number of sample perturbations, and validation set size?

Author response:
Thanks for pointing this out. Our method indeed requires a search over perturbation coefficients, but it requires significantly less computations than a grid search for the hyper-parameters (e.g., temperature). This is because a grid search over $k$ hyper-parameters with $n$ values involves multiple iterations of model training and consumes $O(T_{train} \times n^k)$ time where $T_{train}$ denotes the time of one-round model training. In comparison, our method searches the perturbation coefficients by computing a deterministic quality score via Eq.11. As Algorithm 1 shows, our search method does not involve model training, and only scales with the number of sample perturbations and validation set size by $O(N_k \times N_v)$ , where $N_k$ is the number of sample perturbations, and $N_v$ is the validation set size. 

As detailed in Appendix A.4, our search process is remarkably efficient; for instance, with $N_k=100$ and $N_v=1000$, the search completes in under two minutes on a standard CPU with 64G memory.

We also provide the python code for this process. Please check the uploaded **.ipynb file in the Supplementary Material** for reference.

Reviewer comment (Experiments):
Figure 4.b.: What is the point of this experiment? Injecting perturbations and choosing such perturbations randomly is very unlikely to cause an improvement in the performance. Your method is indeed outperforming random selection, but it also should?

Author response:
The purpose of Figure 4(b) is to demonstrate it is necessary to use our proposed method to select the perturbation coefficients rather than just random selection. In this set of experiments, we maintained a constant search time of 50 across M values from 1 to 5, this consistency did not yield evidence that M>1 outperforms M=1. However, increasing the search time to 100 led to a difference, with M=5 reaching an accuracy of 91.12 compared to M=1's 90.87. 

We would like to provide more background about the perturbation order M to make it clear: M controls the granularity of the perturbation. A larger M means we can add perturbation to the higher-order terms in the polynomial, thus making finer-grained adjustments to the original loss function. The space of the perturbed loss function with a larger M encompasses that with a lower M. In practice, we use M=5 to ensure a reasonable level of perturbation detail without rendering the search space impractically large.

Reviewer comment (Novelty):
Missing references to theoretical distillation works: Phuong and Lampert "Towards Understanding Knowledge Distillation", Mobahi et al. "Self-Distillation Amplifies Regularization in Hilbert Space", and Borup and Andersen "Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation". Also, comparison to the continuous categorical distribution would be highly relevant; Gordon-Rodriguez et al. "The continuous categorical: a novel simplex-valued exponential family".

Author response:
Thank you for highlighting these references, they will be addressed in the updated version of the manuscript.

Reviewer comment (Writing):
Throughout the paper (especially Sections 2 and 3) you have inconsistent use of transposes of the various vectors. E.g. should \(y_n\) in (2) be transposed?

Author response:
There is no need for a transpose. $y_n$ represents a one-hot vector and $ p_{x_n}$ is the predicted probability distribution vector. The operation $y_n log(\mathbf{p}_{x_n})$ is a dot product, thus $y_n$ does not need to be transposed here.

Reviewer comment (Writing):
"This observation also resonates with our intuition that an accurate, well-calibrated, and certain teacher [...]". How is well-calibrated and certain not contradictions here?

Author response:
In this context, 'certain' refers to the teacher's reliability to accurately predict the correct class rather than assigning high probabilities. We understand the confusion this may have caused and have removed the term 'certain' in the revised draft to avoid such misunderstandings. Thank you for bringing this to our attention.

Reviewer comment (Experiments):
How does your proposed PTLoss compare to using e.g. the likelihood of the continuous categorical [3,4]?

Author response:
The setting of continuous categorical is different from ours. In [3], the study of CC–LS poses the learning objective under the background of regularization, instead of model distillation as we study in this paper. They do not have teacher models, but directly learn from the targets. In [4], though it includes a model distillation experiments on MINST, it follows the standard knowledge distillation setting as in (Hinton et.al, 2015), so it is not directly comparable to our work.

However, we modify our settings to make the expected comparison practical. Specifically, we follow the setting of Table 2 in [4] and present the results as follows. We use M=3 for PTLoss in this set of experiments, and report the average performance of 5 runs. We omit the s/epoch metric for our method because we did not find the description of computing hardware in [4]. 

| OBJECTIVE | ACCURACY | RMSE  | S/EPOCH |
|-----------|----------|-------|---------|
| DIRICHLET | 90.6%    | 0.041 | 0.9     |
| SOFT XE   | 94.9%    | 0.029 | 0.8     |
| CC        | 95.6%    | 0.024 | 2.2     |
| HARD XE   | 93.2%    | ---   | 0.8     |
| PTLoss    | 96.0%    | 0.022 | ---      |

Reviewer comment (Novelty):
How does Theorem 1 differ from Proposition 3 and (8) in [5]? They appear equivalent.

Author response:
Theorem 1 in our paper differs from Proposition 3 and Equation (8) in [5] primarily due to the inclusion of an entropy term, i.e., the third term in the equation of Theorem 1. The original paper does not account for this term as it assumes label entropy to be zero, which holds in their context. However, in the case of distillation, the assumption of zero label entropy does not apply, making the entropy term a component of our theorem.

Reviewer comment (Experiments):
More experiments details should be given instead of putting some statistical results.

Author response:
We appreciate your valuable feedback on our work. Due to the space limitations, we put some experiment details in the appendix. Specifically, **A.3** introduces how we solve the proxy teacher via numerical methods; **A.4** presents the search time of perturbation coefficients; **A.5** introduces the detailed setup of the synthetic Gaussian experiment; **A.7** lists the hyper-parameters.

In addition, we also upload a **.ipynb file** to the **Supplementary Material** to show the code of the perturbation coefficients searching method. If you have specific questions / concerns, please let us know and we are willing to address them.

Reviewer comment (Evaluation):
Figure 3 (a) is shown to verify our assumption that the teacher outputting a distribution closer to the ground truth distribution leads to a better student. Why the OHT (grey) method achieves better accuracy than LS (yellow) with lager L2-distance between pt and p*.

Author response:
Thank you for highlighting the observations from Figure 3(a). The curve presented in this figure indeed indicates a general trend rather than a strict correlation. It's a normal phenomenon in such analyses that exceptions to the expected pattern occur. Furthermore,  the blue dots representing noisy data points reinforce the concept that we are observing a trend, emphasizing the non-deterministic nature of this correlation.

Reviewer comment (Reproducibility):
How to quickly find a certain m in   ?

Author response:
If we understand this question correctly, it is asking how to quickly find the perturbation order M. As the hyperparameter to control the granularity of the perturbation, a larger M means we can add perturbation to the higher-order terms in the polynomial, thus making finer-grained adjustments to the original loss function. The space of the perturbed loss function with a larger M encompasses that with a lower M. In practice, we use M=5 to ensure a reasonable level of perturbation detail without rendering the search space impractically large.

Reviewer comment (Theory):
In Figure 1,   is  unknown, how to make sure that “PTLoss implicitly shift   to    such that   “?

Author response:
This question is not clear. We guess the reviewer is asking “$\mathbf{p^*}$ is unknown, how to make sure that PTLoss implicitly shifts $\mathbf{p}^t$ to $\mathbf{p}^{t_{px}}$ such that 
$ \| \mathbf{p}^* - \mathbf{p}^t \|_2 > \| \mathbf{p}^* - \mathbf{p}^{t_px} \|_2
$”
This is demonstrated in Sec. 4.4. Although the ground truth distribution $\mathbf{p^*}$ is unknown, we can use an unbiased estimator to replace it. We leverage a small-size validation dataset to compute thee sample mean to replace the expectation in terms of $\mathbf{p^*}$. 

For a detailed explanation of this methodology, we refer you to Theorem 1 and Eq. 11 in our paper, where we elaborate on this derivation.

Reviewer comment (Theory):
The derivation from Eq.4 to Eq.5 is poor. If Eq.11, if   is known, why do not use ground-truth directly?

Author response:
The derivation from Eq. 4 to Eq. 5 is straightforward, involving a basic linear transformation. In Eq.5, the derivation from the $=$ to the $\approx$ leverages Eq.4. 

The second question is not clear, we think it pertains to “if $y_n$ is known, why do not use ground-truth directly”. Here $y_n$ is the label of the validation set, which is exclusively used for validation, not training. To be consistent, our approach, as well as the baseline methods, do not use validation data directly in the learning process.

We acknowledge that scaling up the number of classes and the perturbation order is challenging. Nevertheless, we theoretically and empirically show that a moderate-order can already help close the distribution gap between the proxy teacher and the ground truth. In practice, the value of M depends on the application needs. We can balance the desired performance and the manageable size of the search space size for perturbation coefficients.

Reviewer comment (Experiments):
Although the proposed coefficients selection method provides a principal way to determine the perturbation hyperparameters, it remains challenging to scale up the number of classes and the perturbation order.

Author response:
The derivation from Eq. 4 to Eq. 5 is straightforward, involving a basic linear transformation. In Eq.5, the derivation from the $=$ to the $\approx$ leverages Eq.4. 

The second question is not clear, we think it pertains to “if $y_n$ is known, why do not use ground-truth directly”. Here $y_n$ is the label of the validation set, which is exclusively used for validation, not training. To be consistent, our approach, as well as the baseline methods, do not use validation data directly in the learning process.

We acknowledge that scaling up the number of classes and the perturbation order is challenging. Nevertheless, we theoretically and empirically show that a moderate-order can already help close the distribution gap between the proxy teacher and the ground truth. In practice, the value of M depends on the application needs. We can balance the desired performance and the manageable size of the search space size for perturbation coefficients.