Paper: Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR-2024)

Reviewer comment (Writing):
The use of $\mathbb{E}$ for the *empirical* expectation is very unconventional, and I would strongly suggest to change this notation. This also conflicts with mentions of distributions later on. It is unclear when we are considering random variables and empirical observations.

Author response:
Thanks for your suggestion. We will change the notation in the revision.

Reviewer comment (Theory):
*"[...] of the distillation data, which is often reused from the teacher’s training data."* It appears as reusing the teacher's training data is an underlying assumption of the paper, which is a notable assumption, and should be stated explicitly as an assumption.

Author response:
We thank the reviewer for pointing this out. We will explicitly state this assumption in the revision.

Reviewer comment (Theory):
*"In particular, minimization of Eq. (1) would lead to $f(x) = 1(y)$ for any input x, [...]"*. Why does this hold? There are no assumptions on $\ell(f(x), y)$ in (1), and the statement would not hold in general?

Author response:
We agree with the reviewer that we may need mild assumptions on the loss function to let $f(x) = 1(y)$, although most widely used loss functions would satisfy this requirement. We thank the reviewer for pointing this out and will update accordingly in the revision.

Reviewer comment (Writing):
Page 4 above (3). Is this the same softmax as other places? If not, why? If so, fix notation.

Author response:
Here the softmax should be our modified softmax. Thanks for pointing this out and we will revise accordingly.

Reviewer comment (Theory):
*"[...] a continuous distribution with finite variance, namely $\mathrm{Var}[X|z] \leq \nu_z$"*. Note, $\mathrm{Var}[X|z] \leq \nu_z$ is *bounded* variance not only finite variance. What is the true assumption here?

Author response:
We note that we will need bounded variance here. We thank the reviewer for pointing this out and we will revise accordingly.

Reviewer comment (Experiments):
The authors never compares to the conventional and very simple setting of early-stopping the teacher and/or carefully tuning the temperature scaling of the teacher predictions as is commonly done in practice.

Author response:
We note that properly early stopping the teacher training may indeed be able to achieve student performance comparable to SoTeacher. However, as also mentioned in Line 356 in our paper, it is often difficult to locate the right epoch number for early stopping. Slight deviation from the best epoch number may significantly impair the student's performance. A meticulous search for the best epoch number will induce high computation costs as one may need to conduct full student training for each epoch number. In contrast, SoTeacher can achieve the best student performance even when the teacher fully converges, which will be much more efficient and friendly to use in practice.  

For other popular training or post-training techniques including classic regularizations such as l1 and l2, label smoothing, advanced data augmentation, temperature scaling, and uncertainty learning methods, we have provided a detailed comparison in Appendix F.

Reviewer comment (Writing):
You both state that *"This demonstrates that our regularizations are not sensitive to hyperparameter selection."* and argue that *"It is thus possible to further boost the performance by careful hyperparameter tuning"*, which seems contradicting.

Author response:
Here *"This demonstrates that our regularizations are not sensitive to hyperparameter selection."* refers to the analyses of hyperparameter sensitivity as shown in Figure 2, while *"It is thus possible to further boost the performance by careful hyperparameter tuning"* refers to the main result reported in Tables 1 and 2, where we didn't report the best result yielded by the hyperparameter search but instead report the result by a more intuitive hyperparameter selection.

Reviewer comment (Evaluation):
About Figure 2(a): *"[...] the teacher accuracy constantly decreases while the student accuracy increases and converges."* The teacher accuracy does not constantly decrease, but at best appear to have a downward trend. Furthermore, it is a stretch to say that the student accuracy converges due to no increase in 1 of 4 intervals. *"This demonstrates that excessively strong Lipschitz regularization hurts the performance of neural network training [...]"*. This is not supported by the figure where the teacher test accuracy for $\lambda_{LR} = 0.5 \cdot 10^{-6}$ and $\lambda_{LR} = 2 \cdot 10^{-6}$ is equal. Hence, the stronger regularization does not really affect the teacher performance.

Author response:
We agree with the reviewer that this figure may need more careful interpretation. Here we generally look at the trend of the curves as the hyperparameter value varies. We will add more grid search points in the revision.

Reviewer comment (Evaluation):
Throughout the paper, it is argued that a reduced teacher performance can yield improvements in the student generalization. However, from Figure 2(c) at the performance of both the teacher and the student increase when using the cosine, cycle or piecewise schedules compared to the standard procedure. In fact, only the case of linear schedule (used elsewhere in the paper) show an increase in student performance combined with a decrease in teacher performance. Thus, it is unclear what the effect the experiments actual show? Is the improvement merely an artifact of other parts of the training scheme?

Author response:
We would like to clarify that we didn't argue reducing teacher performance can improve student performance in our paper. Instead, our argument is that strong teacher performance does not necessarily improve student performance. Therefore, we may need to tailor the teacher training such that the teacher's soft predictions better approximate the true label distribution. Such a goal does not necessarily imply an inferior teacher.

Reviewer comment (Evaluation):
Figure 1: Why does a student distilled from a teacher checkpoint with 55-60% accuracy yield a test accuracy of about 71%, thereby exceeding the teacher significantly?

Author response:
We note that it is possible that the student performance can exceed the teacher performance significantly in Figure 1. This is because in the distillation experiments here, the true labels are always used for training the student on top of the soft labels given by the teacher, and the student here is sufficiently trained, unlike the teacher that is early stopped. The strong regularization effect brought by an early-stopped teacher may further boost student performance.

Reviewer comment (Experiments):
The author should either make the above more explicit in the text, i.e., explicitly restrict the scope of their claims to multi-classification and images, or provide evidence that their technique gives substantial improvements on binary classification tasks in NLP datasets (but even in vision datasets).

Author response:
We note that our theory can be readily applied to binary classification case as binary classification can be viewed as a special case of multi-class classification. This is covered by our theory as we didn't set any assumptions on the dimension of the output prediction.

We note that for NLP datasets, our theory needs to be slightly modified. Instead of assuming a "mixture of features" data model for the vision tasks, we may need a "bag of words" data model for the NLP tasks. Other theorems in our theory can be readily adapted as long as the NLP task can be formulated as a multi-class classification problem.

Reviewer comment (Experiments):
One of the main reasons why knowledge distillation is such a popular technique is because the teacher can generate pseudo-labels for new, unlabeled examples, increasing the size of the student's dataset (known as semi-supervised distillation, or distillation with unlabeled examples, see e.g. [2, 3, 4]). It seems that the current approach typically decreases the test-accuracy of the teacher model — a fact which typically impairs the student's performance in the semi-supervised setting [3, 4].

Author response:
We note that our proposed method may not necessarily hurt the distillation performance on unlabeled data. The distillation on unlabeled data often works the best if the student learns from the soft predictions of the teacher. In this case, a teacher that can approximate the true label distribution of the unlabeled dataset may be optimal, which may not necessarily coincide with a teacher with the best test accuracy. We thank the reviewer for bringing this and the relevant papers [1, 2, 3, 4] and will include this discussion in the revision.

Reviewer comment (Theory):
The dependence on $\delta$ in Lemma 3.5 seems to be very strong. Can Matrix Bernstein be used instead of Chebyshev to get a $\log(1/\delta)$ dependence?

Author response:
We thank the reviewer for the great suggestion. The dependence can indeed be improved using advanced concentration inequalities. We will modify the bound accordingly in the revision.

Reviewer comment (Theory):
The proposed theory does not explain why (higher test accuracy) teachers may not lead to better students.

Author response:
Our theory suggests that the "teacher overfitting" phenomenon, namely that a converged teacher leads to worse student performance, may result from unsatisfactory feature extractors that fail to match regularity constraints, especially the transformation-robust condition. We briefly mentioned this as a motivation for our method in Section 4. Per the reviewer's question, we realized that we may need to explain this in a standalone section. We will update accordingly in the revision.

Reviewer comment (Theory):
The method seems to be at odds with the theory. If the assumptions of Lipschitz continuity and transformation-robustness of the feature extractor tend to be satisfied in practice, then the theoretical results hold. So, why do we need the additional regularization terms that are introduced by SoTeacher?

Author response:
We note that the regularity conditions of the feature extractors are not always satisfied in practice. As mentioned in Section 4, the Lipschitz continuity is probably satisfied in practice. However, the transformation-robustness may not be satisfied in practice, especially when the data is scarce. Therefore, it is important to explicitly enforce such regularity conditions as proposed by our method.

Reviewer comment (Experiments):
The approach introduces two additional hyperparameters corresponding to the two additional regularization terms that have to be tuned.

Author response:
We note that although our method introduced additional hyperparameters, the end performance is not sensitive to their values, as mentioned in Section 5.2.

Reviewer comment (Evaluation):
Evaluations on ImageNet, the largest and most real-world dataset considered, show only a marginal improvement.

Author response:
We agree that on the large dataset, our method doesn't achieve significant performance improvement. This can be partly explained by our theory, as with sufficient training data, the regularity conditions, especially the transformation robustness, can already be satisfied.

Reviewer comment (Theory):
Does the presented theory explain why a better teacher may not necessarily lead to a better student?

Author response:
Our theory suggests that the "teacher overfitting" phenomenon, namely that a converged teacher leads to worse student performance, may result from unsatisfactory feature extractors that fail to match regularity constraints, especially the transformation-robust condition. We briefly mentioned this as a motivation for our method in Section 4. Per the reviewer's question, we realized that we may need to explain this in a standalone section. We will update accordingly in the revision.

Reviewer comment (Evaluation):
Why does the method not perform as well on ImageNet?

Author response:
We agree that on the large dataset, our method doesn't achieve significant performance improvement. This can be partly explained by our theory, as with sufficient training data, the regularity conditions, especially the transformation robustness, can already be satisfied.

Reviewer comment (Presentation):
The Lipschitz and Consistency regularizations are one of the most important contributions of this paper, but the readers need to read Appendix to fully understand how these two regularizations are applied. Considering the length of the Appendix is not very long, I would suggest making them clear in the main paper.

Author response:
Thanks for your suggestion. We will move the details of the proposed methods into the main paper.

Reviewer comment (Experiments):
In the experiments, it is unclear how the number of epochs impact the results.

Author response:
In this paper, we follow the widely used settings for training the teacher, namely 240 epochs for CIFAR-100 and 90 epochs for ImageNet and Tiny-ImageNet. We will explore alternative training settings in the revision.