R EDUCING THE T EACHER -S TUDENT G AP VIA A DAPTIVE T EMPERATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowledge distillation aims to obtain a small and effective deep model (student) by learning the output from a larger model (teacher).
Previous studies found a severe degradation problem, that student performance would degrade unexpectedly when distilled from oversized teachers.
It is well known that larger models tend to have sharper outputs (Guo et al., 2017).
Based on this observation, we found that the sharpness gap between the teacher and student output may cause this degradation problem.
To solve this problem, we ﬁrst propose a metric to quantify the sharpness of the model output.
Based on the second-order Taylor expansion of this metric, we propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperature of the teacher and the student, to reduce the sharpness gap G s gap .
We conducted extensive experiments on CIFAR100 and
ImageNet and achieved signiﬁcant improvements.
Speciﬁcally, ATKD trained the best ResNet18 model on ImageNet as we knew (73.0% accuracy).
INTRODUCTION
Deep neural networks have achieved remarkable success in most of computer vision tasks (He et al., 2016; Deng et al., 2009), while the increasing network capacity in the current state-of-the-art models also results in severe computational burdens and high inference time (Tian et al., 2020;
Yim et al., 2017).
One direction to reduce the model size is knowledge distillation (KD), which trains small efﬁcient models (student) by learning from the output of large models (teacher) (Tang et al., 2020).
In 2015, Hinton proposed to soften the model output by raising the temperature of the last softmax layer(Hinton et al., 2015).
Since then, this temperature-based kowledge distillation has drawn the main stream attentions and achieved many successes (Zagoruyko & Komodakis, 2017;
Kim et al., 2020; Furlanello et al., 2018; Fu et al., 2020).
However, recent researches found that knowledge distillation suffers from a mysterious performance degradation problem(Cho & Hariharan, 2019; Mirzadeh et al., 2019).
Speciﬁcally, since the idea of
KD is transferring the knowledge of teacher into student, a nature assumption is larger teachers would train better students.
But recent researches refuted this assumption and found that the student often performs worse with oversized teachers(Cho & Hariharan, 2019).
This paper studies this degradation problem by considering the sharpness (or softness) of the model output.
It is well known that networks with more parameters tend to produce sharper output (Guo et al., 2017; Lee et al., 2018), which would create the gap of sharpness between teacher and student, i.e. the teacher is usually much sharper than the student.
It should be noted that temperature technology in vanilla knowledge distillation does not reduce this gap.
This is because in vanilla knowledge distillation, the teacher and student are softened with the same temperature, which maintains the large gap of sharpness.
When the teacher grows larger, this gap will further increases, making it more difﬁcult for the student to learn from the teacher.
To solve this problem, let us consider a modiﬁed softer teacher model with the same accuracy.
In this way, the sharpness gap between the student and the teacher models would be narrowed, making learning more accessible.
One way to build this softer teacher is to use higher temperatures for the teacher model, but it would require labor-intensive searches on the validation data set.
With this insight, we propose to adaptively change the temperatures of the teacher and the student by their sharpness.
Our contribution is three-fold: We
propose a metric to quantify
the
sharpness
of models , allowing us to precisely control the sharpness of model output.
Formally, we use the realsoftmax function (
Nielsen & Sun, 2016) on
the logits vector as the sharpness metric, which approximates a smoother version
of the max function.
If we denote student logits as z , the teacher logits as v , teacher temperature and student temperature as τ T and τ S respectively ( τ T is equal to τ S in the vanilla knowledge distillation), the sharpness metric and the gap of sharpness are deﬁned as follows:
[Equation]
We propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperatures of the teacher and the student, to reduce the sharpness gap G s gap .
ATKD relieves the burden of searching for the appropriate temperature on the validation set.
Speciﬁcally, by Taylor second-order expansion, our method can be implemented easily by normalizing the logits with the standard deviation.
Our S sharpness metric can be used to explain why two existing methods work.
Speciﬁcally, the Early Stopped method and the Teacher Assistant method both reduce the sharpness gap between the teacher and the student.
We present comprehensive experiments on CIFAR-100 Krizhevsky et al.
and ImageNet datasets (Deng et al., 2009) to evaluate our method.
The experiment results show that: 1)
The proposed ATKD method tremendously mitigates the performance degradation problem and trains the best ResNet18 model on ImageNet as we know (73.01% accuracy).
The proposed ATKD method is easy to optimize, exhibits much lower training/test loss with oversized teachers than KD.
METHODOLOGY 2. B ACKGROUND
Vanillay Knowledge Distillation During training, we minimize the negative log likelihood of the ground truth class to update model parameters.
After the model is properly trained, the probability of the ground truth would be close to 1, while the rest wrong predictions are near zeros.
In 2015, Hinton noticed that these small wrong probabilities are useful to unveil “dark knowledge” (Hinton et al., 2015).
Take a picture of “cat” for example, the model are more likely to output higher probability for class “dog” than class “airplane”.
These wrong probabilities imply the relationship between the two classes and unveil how a model tends to generalize.
This observation inspired to use the output of large models as soft targets to train efﬁcient small models.
However, modern deep networks tend to produce peaky probabilities (Guo et al., 2017; Lee et al., 2018), that the numbers of those wrong classes (near zero values) would be negligible compared to the ground truth (near one).
Thus Hinton proposed to raise the temperature of the last softmax to soften the output probabilities, which can be used as soft targets to train small networks.
We denote logits of the teacher as v , and logits of the student as z , temperature as τ , i and j denote the ith and jth value of logits (i.e. the ith and jth category of K classes). the loss of knowledge distillation is as follows:
[Equation]
The ﬁnal loss for the student is then the weighted sum of the cross entropy loss L cls and the knowledge distillation loss L KD : [Equation]
[Table]
The popular choice of the temperature τ is in { 3 , 4 , 5 } and the weight λ = 0 .
9 (Hinton et al., 2015;
Cho & Hariharan, 2019; Tian et al., 2020).
Performance Degradation Problem While knowledge distillation achieved success in many ﬁelds, a mysterious performance degradation problem was spotted in 2019.
Since the idea of knowledge distillation is transferring teacher knowledge into students, one natural hypothesis is that a larger and more accurate teacher would capture more knowledge and thus train better students.
Unfortunately, previous studies invalidate this hypothesis by showing that the student performance would degenerate unexpectedly with larger teachers.
Table 1 shows our experiment result.
The student is ResNet14.
With larger teachers, the student accuracy degrades, and the KD loss increase.
Cho & Hariharan (2019) hypothesizes that the mismatch of capacity causes this problem.
We will discuss this problem in the rest of the section.
2. A DAPTIVE T EMPERATURE K NOWLEDGE D ISTILLATION
We investigated the degradation problem and proposed to use adaptive temperatures during training to mitigate this problem:
We found that the mismatch sharpness between the teacher and the student may cause the degradation problem.
We propose a metric to quantify the sharpness of a model.
We propose the adaptive temperature knowledge distillation (ATKD) methods.
The sharpness of neural networks Two main theories explain why knowledge distillation is effective.
The ﬁrst theory originated from Hinton’s ﬁrst paper, which argued that teachers are more accurate in capturing category similarities.
This information of category similarity helps students to generalize better on unseen data.
On the other hand, some contend that the soft output of the teacher prevents the student network from overconﬁdence (ie.
label smoothing technique).
However, when the teacher gets larger, these two theories will contradict each other.
This is because that larger models are usually more accurate and peaky at the same time.
On the one hand, a more accurate teacher is beneﬁcial to the student model.
On the other hand, a sharper teacher tends to make the student overconﬁdent.
Therefore, the larger teacher capacity is a double-edged sword for distillation.
This analysis shows that the sharpness of teachers should be investigated carefully, while previous studies have overlooked this critical attribute of teachers.
To this end, we propose to use realsoftmax (Nielsen & Sun, 2016) to quantify the sharpness of the model output, which is deﬁned as the logarithm of the sum of the exponentials of the logits:
[Equation]
Such metric has at least two advantages, therefore reasonably measure the sharpness of the model output: 1)
This metric is a smooth approximation to the maximum function max j ( z ) 2)
It is also differentiable.
Meanwhile, we quantify the sharpness gap between the teacher and the student as the difference of the sharpness metric:
[Equation]
[Table]
If we consider the temperatures, the sharpness gap is:
[Equation]
Table 2 shows sharpness metric value for models of different capacity.
It shows that S sharpness increases with network capacity, and there is a sharpness gap between networks of different sizes.
Adaptive Temperature One idea is using higher temperatures for the teacher to reduce the sharpness gap between teachers and students.
However, searching on the validation data set requires a lot of computational costs.
Instead, we propose adaptive temperature knowledge distillation, which automatically tunes the temperature according to the S sharpness .
By Taylor second expansion:
[Equation]
We follow the assumption from Hinton (Hinton et al., 2015), that the logits have been zero-meaned separately for each training example so that (cid:80) j z j = (cid:80) j z j = 0 .
We found from experiments that the sum of logits is indeed very small numbers close to zero.
The experiment results and further theoretical analysis are provided in the appendix.
Given the above assumption (cid:80) j z j = (cid:80) j z j = 0 , We can get:
[Equation]
The 1 K (cid:80) j v 2 j in the equation is the variance of logits, which can be represented by the square of the standard deviation std 2 : [Equation]
This result shows the relationship between the temperature and sharpness gap is pretty straightforward, that the standard deviation controls the sharpness gap between the teacher and the student.
Therefore, we set the temperature of the teacher and the student as the standard deviation of the logits vector respectively.
These adaptive temperatures would automatically change during training, reducing the sharpness gap between the teacher and the student.
It is worth noting that there is no need to search for the proper temperature on the validation set like the vanilla KD.
Formally, if we denote standard deviation function as Std , we propose Adaptive Temperature
Knowledge Distillation as follows:
[Equation]
2. S HARPNESS G AP A NALYSIS
We investigated the relationship between the sharpness gap and the temperature, and found that the G s gap of ATKD decrease with 1 τ 3 , while the vanilla KD decrease with 1 τ 2 .
Therefore, If the temperature of KD and adaptive temperatures of ATKD are in the same temperature range, the sharpness gap of ATKD G s gap would be smaller than KD in a considerable margin.
If we use K to denote K classes, the sharpness gap would decrease with 1 τ 2 under vanilla KD:
[Equation]
This shows that when the temperature is very high and the linear function (1+ x ) can approximate e x , G s gap would be reduced.
However, with this linear approximation, the KD loss would degenerate into MSE loss, as Hinton points out (Hinton et al., 2015):
[Equation]
However, high temperatures could be harmful to student, because it would encourage the student to learn more from those negative logits of the teacher (Hinton et al., 2015), and as Hinton pointed out, these negative values of logits could be very noisy.
On the other hand, the sharpness gap of ATKD would decrease faster with 1 τ 3 , and enable the
ATKD to achieve much lower sharpness gap than KD.
Because we can use second-order Taylor approximation (1+ x + 12 x 2 ) , which is a better approximation than linear function (1+ x ) in a wide range.
[Equation]
EXPERIMENTS
In this section, we will show comprehensive experiment results to validate the effectiveness of ATKD from several perspectives.
Speciﬁcally, we ﬁrst conducts experiments on two popular CV datasets [Table] [Table] to demonstrate the performance of ATKD.
Then we focused on evaluating whether it could alleviate the performance degradation problem.
Dataset 1) CIFAR-100 (Krizhevsky et al., 2009) is a relatively small data set and is widely used for testing various of deep learning methods.
CIFAR-100 contains 50,000 images in the training set and 10,000 images in the dev set, divided into 100 ﬁne-grained categories.
2) ImageNet (Deng et al., 2009) is a much larger one than CIFAR-100.
ImageNet contains 1.2M images for training and 50K for validation, that distributes in 1000 classes.
CIFAR Experimental settings We run a total of 240 epochs for all methods.
The learning rate is initialized as 0.05, then decay by 0.1 every 30 epochs after 150 epochs.
Temperature is 4 for vanilla
KD, and the weight of ATKD or KD and cross-entropy is 0.9 and 0.1 for all the settings.
ImageNet Experimental settings Here we use ResNet18 as student for all of methods.
Training settings like learning rate or training epochs are the same with Heo et al.
for ImageNet.
The teacher network is well-trained previously and ﬁxed during training.
3. CIFAR-100 AND I MAGE N ET
Baselines.
We selected many SOTA KD methods to evaluate the performances of ATKD.
Knowledge deﬁned from intermediate layers: FitNet (Romero et al., 2015), AT (Zagoruyko & Komodakis, 2017), SP (Tung & Mori, 2019), PKT (Passalis & Tefas, 2018), FT (Kim et al., 2020), FSP (Yim et al., 2017) .
1) Knowledge deﬁned via mutual information: CC (Peng et al., 2019), VID (Ahn et al., 2019),
CRD (Tian et al., 2020).
2) Structured Knowledge: RKD (Park et al., 2019).
3) Knowledge from logits: KD (Hinton et al., 2015), NST (Huang & Wang, 2017), ES (Cho & Hariharan, 2019),
TA (Mirzadeh et al., 2019)
Results in CIFAR-100.
Table 3 shows that ATKD always has an outstanding improvement compared with all other methods.
In some situations (e.g. those where teacher/student is WRN-402/WRN-40-1 or ResNet110/ResNet32), the performances of ATKD are even very close to those of teacher.
Results in ImageNet.
All experiments used ResNet34 as the teacher and ResNet18 as the student.
Table 4 (the Top1 accuracy) shows that ATKD exceeds all of the previous SOTA by a large mar- [Table]
gin.
Before ATKD [Table] the improvement of this task is limited.
Fig.
1 shows the training process of vanilla KD and ATKD.
It is worth noting that ATKD provides comparable performance to KD’s ﬁnal performance after ﬁrst 30th epoch training.
3. P ERFORMANCE D EGRADATION E XPERIMENTS
CFIFAR-100 On CIFAR-100 task, We trained the ResNet14 with multiple teachers on the CIFAR100 dataset, and the result is shown in Table 5. Experimental details follows the above settings
[Table]
Under ATKD, student performance continues to increase as the teacher gets bigger.
The training loss of ATKD is also much lower than the vanilla KD.
In addition, the G s gap (computed with temperatures) metric shows that ATKD signiﬁcantly reduces the sharpness gap.
In the analysis of section 2.3, we concluded that larger temperatures would have smaller sharpness gap.
Therefore the comparison between KD and ATKD need to take temperature values into account.
The temperature of KD is 4, while most of the mean adaptive temperatures used by ATKD are between (3 , 4) .
We provide more details in the appendix. ImageNet Table 6 shows the degradation problem in ImageNet with ResNet18 as the student, and ’-’ denote that this speciﬁc experiment was not conducted in the cited paper.
We compared ATKD with two previous methods that aim to alleviate the degradation problem, Early Stop (Cho & Hariharan, 2019) (ES) and Teacher Assistant (Mirzadeh et al., 2019)
(TA).
Both of these two methods explicitly regularizing the teacher capacity: 1) TA proposed to distill the large teacher to an intermediate teacher and then distill to the student, so that each knowledge distillation step has a better match between student and teacher capacity.
2) ES methods use the early stopped teacher, the teacher capacity would be regularized by fewer training steps.
We can see that ATKD exceeds Early Stop and TA methods with a large margin in all teacher settings.
For example, when distilled by ResNet50 and ResNet152, the performance exceeds other methods by two percent.
Finally, we obtained 73.01% accuracy, which is the best ResNet18 results as we know.
An interesting ﬁnding of the early stopped teacher method is that these regularized teachers have smaller S sharpness , which can reduce the sharpness gap between the teacher and the student.
The results is in Table 7.
Similarly, the Teacher Assistant method also reduces the sharpness gap by using medium-sized teachers.
This ﬁnding indicates that S sharpness metric could provide a universal framework to mitigate the degradation problem.
RELATED WORK
Knowledge Distillation Buciluˇa et al.
ﬁrst proposed compressing a trained cumbersome model into a smaller model by matching the logits between them.
Then Hinton et al.
(2015) advanced this idea and formed a more widely used framework known as knowledge distillation (KD).
Knowledge distillation tries to minimize the KL divergence between the soft output probabilities generated by the logits through softmax.
Xu et al.
(2020) proposed to normalize the feature, the penultimate layer of the network, to perform distillation.
This method is similar to our methods
[Table] [Table] [Figure] except that we perform on the logits layer.
Knowledge distillation is also a kind of soft label training method.
Similar to label smoothing, previous studies have found that knowledge distillation helps to regularize the training of network.
Performance Degradation Problem: Although distillation has shown a great potential in many tasks, a mysterious problem is found that larger teachers often harm the distillation performance, despite its more powerful ability (Cho & Hariharan, 2019; Mirzadeh et al., 2019).
This problem is particularly severe on ImageNet, resulting in poor performance for KD.
It was widely accepted that the mismatch of capacity caused this problem.
Previous research proposed to regularize the teacher capacity to alleviate this problem heuristically.
Cho & Hariharan (2019) proposed to early stop the training of the teacher.
Moreover, Mirzadeh et al.
(2019) proposed to use a medium-size teacher assistant (TA) to perform a sort of sequence distillation.
This TA ﬁrst learns from the teacher, then the student can learn from this TA.
However, the accuracy of the early stopped teacher or TA is also harmed, which is lower than the original teacher.
CONCLUSION
The vanilla knowledge distillation overlooks the sharpness gap between the teacher and the student, which may cause the performance degradation problem.
In this paper, we propose to use a metric to measure the sharpness of neural networks, allowing us to control the sharpness of models by using adaptive temperatures.
Recently some papers have focused on combining label smoothing and knowledge distillation to train neural networks (Shen et al., 2021; M¨uller et al., 2019).
An appealing point of view is that
ATKD is inherently consistent with these methods.
ATKD smooths teachers by adapting temperatures, while these methods let students learn from teachers trained by label smoothing.
The deeper relationship between knowledge distillation and label smoothing remains to be explored
A A PPENDIX A. E XPERIMENTS OF D IFFERENT F IXED T EMPERATURES
We conducted experiments of different ﬁxed temperatures (Table 8) [Table] which shows no observable performance improvement.
This may because that sharpness of model output varies with the input, and one ﬁxed temperature is not suitable for all training data. We distilled ResNet14 by ResNet on CIFAR-100 dataset [Table]
and the rest details followed the setting in section
[Table]
[Table]
We add degradation problem experiments where teacher models vary with width here (Table 9). We use Wide ResNet to train on CIFAR-100 dataset, the rest experiment details follows the setting in Section 3.
[Table]
Entropy was used as conﬁdence score at Pereyra et al. (2017), that is related to the sharpness of model output. We measured the entropy values of different models (Table 10).
Experimental details follow the setting in section
[Table]
[Table]
[Figure]
[Figure]
[Table]
Figure 2 shows the sharpness gap changes during the training here
[Table]
where the sharpness gap of KD stops to decrease after certain epochs.
ResNet20 and ResNet56 are used as students and ResNet and WRN-16-3 are used as teachers.
A. E XPERIMENT ON SVHN
We add another dataset SVHN here Goodfellow et al.
Results shows in tabel 11.
SVHN is similar to MNIST (e
[Table]
g., the images are of small cropped digits)
[Table]
but comes from real house numbers in Google Street View images.
We use ResNet14 as student.
The experiment setting follows the section 3, further exploration on parameter tuning might get better result.
[Table]
A. T HE S UM OF L OGITS
In the paper of Hinton et al.
[Table]
they presume that the (cid:80)
j
z j equals to zero during training without further analysis (In page 3, Eq. 3).
Here, we analyze this assumption theoretically and experimentally.
It is well known that the gradient to logits z j is:
[Equation]
Where p j is the jth class probability of the student and the q j is the probability of the teacher.
We can get the gradient to (cid:80)
z j by adding these gradients:
[Equation]
Therefore, the gradient to the sum of logits is zero.
Besides, we know that the logits is the product of the weight W and the feature vector f in the penultimate layer:
[Equation]
Where W i is the ith column vector of W .
Considering that the initial weights of W is sampled from zero-meaned normal distribution in the popular initialization settings
He et al.
, we assume that the initialized (cid:80) j W ij equals to zero.
Therefore, the (cid:80) z j should be zero all the time given the above assumptions.
We conducted experiments to verify this assumption.
We train these models on CIFAR-100 dataset.
Table 12 shows that all these models logits are close to zero.
[Table]
Logits
[Table]
A. T HE A DAPTIVE T EMPERATURES
We provide the mean value of the temperatures here.
[Table]