Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
Abstract
Knowledge Distillation (KD) aims at transferring the knowledge of a wellperformed neural network (the teacher ) to a weaker one (the student ).
A peculiar phenomenon is that a more accurate model doesn’t necessarily teach better, and temperature adjustment can neither alleviate the mismatched capacity.
To explain this, we decompose the efﬁcacy of KD into three parts: correct guidance , smooth regularization , and class discriminability .
The last term describes the distinctness of wrong class probabilities that the teacher provides in KD.
Complex teachers tend to be over-conﬁdent and traditional temperature scaling limits the efﬁcacy of class discriminability , resulting in less discriminative wrong class probabilities.
Therefore, we propose Asymmetric Temperature Scaling (ATS) , which separately applies a higher/lower temperature to the correct/wrong class.
ATS enlarges the variance of wrong class probabilities in the teacher’s label and makes the students grasp the absolute afﬁnities of wrong classes to the target class as discriminative as possible.
Both theoretical analysis and extensive experimental results demonstrate the effectiveness of ATS.
The demo developed in Mindspore is available at https://gitee.com/lxcnju/ats-mindspore and will be available at https: //gitee.com/mindspore/models/tree/master/research/cv/ats .
Introduction
Although large-scale deep neural networks have achieved overwhelming successes in many realworld applications [22, 11, 60], the vast capacity hinders them from being deployed on portable devices with limited computation and storage resources [3].
Some efﬁcient architectures, e.g.,
MobileNets [14, 37] and ShufﬂeNets [59, 29], have been proposed for lightweight deployment, while their performances are usually constrained.
Fortunately, knowledge distillation (KD) [46, 13] could transfer the knowledge of a more complex and well-performed network (i.e., the teacher ) to them.
The original KD [13] forces the student to mimic the teacher’s behavior via minimizing the KullbackLeibler (KL) divergence between their output probabilities.
Recent studies generalize KD to various types of knowledge [36, 57, 17, 12, 33, 1, 34, 44, 52, 27, 54, 45, 50, 26, 23] or various distillation schemes [61, 2, 58, 20].
An intuitive sense after the proposal of KD [13] is that larger teachers could teach students better because their accuracies are higher.
A recent work [6] ﬁrst points out that the teacher accuracy is a poor predictor of the student’s performance.
That is, more accurate neural networks don’t necessarily teach better.
Until now, this phenomenon is still counter-intuitive [51], surprising [31], and unexplored [24].
Different from some existing empirical studies and theoretical analysis [40, 18, 30, 35, 55, 63, 6, 28, 15], we investigate the miraculous phenomenon in detail and aim to answer the following questions: What’s the real reason that more complex teachers can’t teach
[Figure]
of KD
[Figure]
well?
Is it really impossible to make larger teachers teach better through simple operations, such as temperature scaling?
To answer the ﬁrst question, we focus on analyzing the distinctness of wrong class probabilities that a teacher provides in KD.
We decompose the teacher’s label into three parts (see Sect.
4.1): (I)
Correct Guidance : the correct class’s probability; (II) Smooth Regularization : the average probability of wrong classes; (III) Class Discriminability : the variance of wrong class probabilities (deﬁned as derived variance ).
The commonly utilized temperature scaling could control the efﬁcacy of these three terms (the left of Fig. 1).
More complex teachers are over-conﬁdent and assign a larger score for the correct class or less varied scores for the wrong classes.
If we use a uniform temperature to scale their logits, the class discriminability of the larger teacher is less effective (theoretically analyzed in
Sect. 4.2), i.e., the probabilities of wrong classes are less distinct (the right of Fig. 1).
As to the second question, we focus on enlarging the variance of wrong class probabilities (i.e., derived variance ) that a teacher provides to make the distillation process more discriminative.
To speciﬁcally enhance the distinctness of wrong class probabilities, we separately apply a higher/lower temperature to the correct/wrong class’s logit instead of a uniform temperature (see Sect. 4.3).
We name our method Asymmetric Temperature Scaling (ATS) , and abundant experimental studies verify that utilizing this simple operation could make larger teachers teach well again.
Related Works
KD with Larger Teacher : Although KD has been a general technique for knowledge transfer in various applications [13, 61, 42, 25], could any student learn from any teacher?
[6] ﬁrst studies the
KD’s dependence on student and teacher architectures.
They ﬁnd that larger models do not often make better teachers and propose the early-stopped teacher as a solution.
[31] introduces a multi-step
KD process, employing an intermediate-sized network (the teacher assistant ) to bridge the capacity gap.
[51] formulates KD as a multi-task learning problem with several knowledge transfer losses.
The transfer loss will be utilized only when its gradient direction is consistent with the cross-entropy loss.
[10, 24] deﬁne the knowledge gap as residual , which is utilized to teach the residual student , and then they take the ensemble of the student and residual student for inference.
These works attribute the worse teaching performance to capacity mismatch, i.e., weaker students can’t completely mimic the excellent teachers.
However, they don’t explain this peculiar phenomenon in detail.
Understanding of KD :
Quite a few works focus on understanding the advantages of KD from a principled perspective.
[28] uniﬁes KD and privileged information into generalized distillation .
[35, 18] utilize gradient ﬂow and neural tangent kernel to analyze the convergence property of KD under deep linear networks and inﬁnitely wide networks.
[5] explains KD via quantifying the task-
[Table]
f
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
relevant and task-irrelevant visual concepts.
[7] casts KD as a semiparametric inference problem and proposes corresponding enhancements.
Our work is more related to KD decompositions.
[9] treats the teacher’s correct/wrong outputs differently, respectively explaining them as importance weighting and class similarities.
[40] further decomposes the “dark knowledge” into universal knowledge, domain knowledge, and gradient rescaling.
[30] establishes a bias-variance tradeoff to quantify the divergence of a teacher with the Bayes teacher .
[63] utilizes bias-variance decomposition to analyze
KD and discovers regularization samples that could increase bias and decrease variance.
Our work is also related to Label smoothing (LS).
[55] points out that the regularization effect in KD is similar to
LS.
[32] ﬁnds that training a teacher with LS could degrade its teaching quality, and attributes this to the fact that LS erases relative information between teacher logits.
Recently, [38] further studies this problem and proposes a metric to measure the degree of erased information quantitatively.
Our work also decomposes KD into several effects to study why more complex teachers can’t teach well.
Detailed relatedness to these works is presented in Sect.
Background
We consider a C -class classiﬁcation problem with Y =
[ C ] = { 1 , 2 , ⋯ ,C } .
Given a neural network and a sample pair ( x ,y ) , we could obtain the “logits” as f ( x ) ∈ R C .
We denote the softmax function with temperature τ as SF (⋅ ; τ ) , i.e., p c ( τ ) =
exp ( f c ( x )/ τ )/
Z ( τ ) and Z ( τ ) =
∑ Cj = 1 exp ( f j ( x )/ τ ) , where p ( τ ) is the softened probability vector that a network outputs and c is the index of class.
Later, we may omit the dependence on x and τ if without any ambiguity.
We use f y and p y to denote the correct class ’s logit and probability, while we use g and q to represent the vector of wrong classes ’ logits and probabilities, i.e., g =
[ f c ] c ≠ y and q =
[ p c ] c ≠ y .
The notations could be found in Tab. 1.
The most standard KD [13] contains two stages of training.
The ﬁrst stage trains complex teachers, and then the second stage transfers the knowledge from teachers to a smaller student via minimizing the KL divergence between softened probabilities.
Usually, the loss function during the second stage (i.e., the student’s learning objective) is a combination of cross-entropy loss and distillation loss:
[Equation]
KD Loss , (1) where the upper script “T”/“S” denotes “Teacher”/“Student” respectively.
Commonly, a default temperature of 1 is utilized for the CE loss, and the student could also take a temperature of 1 for the
KD loss, e.g., p Sc ( τ = 1 ) [13, 31, 45, 44].
Suppose we have two teachers, denoted as T large and T small , and the larger teacher performs better on both training and test data.
If we use them to teach the same student S , we could ﬁnd that the student’s performance is worse when mimicking the larger teacher’s outputs.
Adjusting the temperature could neither make the larger teacher teach well.
The details of this phenomenon could be found in [6, 31] and Fig.
Obviously, p T large could differ a lot from p T small , which is the only difference in the loss function when teaching the student.
Hence, we focus on analyzing what probability distributions are tended to be provided by teachers with different capacities .
Proposed Methods
This section ﬁrst decomposes KD into three parts and deﬁnes several quantitative metrics.
Then, we present theoretical analysis to demonstrate why larger networks can’t teach well.
Finally, we propose a more appropriate temperature scaling approach as an alternative
[Table]
4.1 KD Decomposition
We omit the coefﬁcient of λτ 2 in Eq.
[Table]
and deﬁne e ( q T ( τ ))
= 1 C −
∑ Cj = 1 ,j ≠ y p Tj ( τ ) , where q T ( τ ) =
[ p Tc ( τ )] c ≠ y .
Then, we have the following decomposition:
[Equation]
[Table]
(2) (I) Correct Guidance : this term guarantees correctness during teaching.
The decomposition in [9] also contains this term, which is explained as importance weighting.
This term works similarly to the cross-entropy loss, which could be dealt with separately when applying temperature scaling.
(II) Smooth Regularization : some previous works [55, 62, 63] attribute the success of KD to the efﬁcacy of regularization and study its relation to label smoothing (LS).
The combination of this term with correct guidance works similarly to LS.
Notably, e ( q T ( τ )) differs across samples, implying that the strength of smoothing is instance-speciﬁc, which is similar to the analysis in [62].
(III) Class
Discriminability : this term tells the student the afﬁnity of wrong classes to the correct class.
Transferring the knowledge of class similarities to students has been the mainstream guess of the “dark knowledge” in KD [13, 38].
Ideally, a good teacher should be as discriminating as possible in telling students which classes are more related to the correct class.
Illustrations of the decomposition are presented in the left of Fig.
Obviously, an appropriate temperature should simultaneously contain the efﬁcacy of the three terms, e.g., the shaded row in
Fig.
A too high or too low temperature could lead to smaller class discriminability , making the guidance less different among wrong classes, which weakens the distillation performance in practical.
Among these three terms, we advocate that class discriminability is more fundamental in KD and present more discussions in Appendix A (veriﬁed in Fig.
2 and Fig. 3).
To measure these three terms quantitatively, we use the target class probability (i.e., p y ), the average of wrong class probabilities (i.e., e ( q ) = 1 C − 1 ∑ j ≠ y p j ), and the variance of wrong class probabilities (i.e., v ( q ) =
1 C − 1 ∑ j ≠ y ( p j − e ( q )) 2 ) as estimators.
e (⋅) and v (⋅) calculates the mean and variance of the elements in a vector.
In some cases, we use the standard deviation as an estimator for the third term, i.e., σ ( q ) =
v 1 / 2 ( q ) .
Because the latter two terms are calculated after applying softmax to the complete logit vector, we deﬁne them as Derived Average (DA) and Derived Variance (DV) , respectively.
In experiments, we calculate these metrics for all training samples and sometimes report the average or standard deviation across these samples.
4.2 Theoretical Analysis
We analyze the mean and variance of the softened probability vector, i.e., the teacher’s label p T ( τ ) used in KD.
We defer the proofs of Lemma 4.1 and Proposition 4.3, 4.4 to Appendix B.
Lemma 4.1 (Variance of Softened Probabilities) .
Given a logit vector f ∈ R C and the softened probability vector p =
SF ( f ; τ ) ,τ ∈ ( 0 , ∞) , v ( p ) monotonically decreases as τ increases.
As τ increases, p ( τ ) becomes more uniform, i.e., it’s entropy increases.
However, we especially focus on the wrong classes, where the mean and variance are more intuitive to calculate and analyze.
Assumption 4.2.
The target logit is higher than other classes’ logits, i.e., f y ≥ f c , ∀ c ≠ y .
Assumption 4.2 is rational because well-performed teachers could almost achieve a higher accuracy (e.g., >95%) on the training set, and most training samples meet this requirement.
Proposition 4.3.
Under Assumption 4.2, p y monotonically decreases as τ increases, and e ( q ) monotonically increases as τ increases.
As τ → ∞ , e ( q ) → 1 / C .
Proposition 4.3 implies that increasing temperature could lead to a higher derived average (empirically see Fig. 7) and strengthen the smooth regularization term in Eq. 2.
Before we analyze the class discriminability term, we deﬁne ˜
q ( τ ) as the result of applying softmax only to the wrong logits with temperature τ , i.e., ˜ q ( τ ) =
SF ( g ; τ ) .
For the element index c ′ of q , we have
[Equation]
Notably, ˜ q differs from q a lot.
Speciﬁcally, the former satisﬁes ∑ c ′ ˜ q c ′ =
1 , while the summation of the latter is ∑ c ≠ y p c = 1 − p y .
The former does not depend on the correct class’s logit while the latter does.
We name v ( ˜ q ) Inherent Variance (IV) because it only depends on wrong classes’ logits.
Proposition 4.4 (Derived Variance vs. Inherent Variance) .
The derived variance is determined by the square of derived average and the inherent variance via:
[Equation]
With τ increases, e ( q ) increases (Proposition 4.3) while v ( ˜ q ) decreases (Lemma 4.1), and hence, it is not so easy to judge the speciﬁc monotonicity of v ( q ) w.r.t.
τ .
Empirically, we observe that the derived variance ﬁrst increases and then decreases (see Fig. 7), which conforms to the change of the class discriminability as illustrated in Fig.
We could use Proposition 4.4 to clearly analyze why larger teacher networks can’t teach well.
Before this, we present another two properties and a corollary without detailed proof.
Remark 4.5 .
Fixing g and τ , a higher target logit f
y leads to a higher p y , i.e., a smaller derived average e ( q ) .
Remark 4.6 .
Fixing τ , less varied wrong logits g leads to less varied ˜ q , i.e., a smaller inherent variance v ( ˜ q ) .
Corollary 4.7.
Suppose we have two teachers T 1 and T 2 , and their logit vectors for a same sample are f T 1 and f T 2 .
• If f T 1 y ≥ f T 2 y while g T 1 and g T 2 are nearly the same, then p T 1 y ≥ p T 2 y (Remark 4.5) while v ( ˜ q T 1 )
≈ v ( ˜ q T 2 ) .
Hence, v ( q T 1 ) ≤ v ( q T 2 ) .
• If f T 1 y ≈ f T 2 y while v ( g T 1 ) ≤ v ( g T 2 ) , then p T 1 y ≈ p
T 2 y while v ( ˜ q T 1 ) ≤ v ( ˜ q T 2 )
(Remark 4.6).
Hence, v ( q T 1 ) ≤ v ( q T 2 ) .
This corollary explains why a larger teacher can’t teach better.
Because the larger teacher tends to be over-conﬁdent, the target logit f
y may be larger or the variance of wrong logits v ( g ) may be smaller.
These are illustrated in Fig.
1 and empirically veriﬁed in Fig.
Then the derived variance v ( q ) may be smaller, limiting the efﬁcacy of class discriminability in Eq. 2.
Empirical results are in Fig.
Notably, we focus on analyzing the variance of wrong class probabilities instead of all classes.
Maximizing the variance of all classes’ probabilities does not mean maximizing the variance of wrong classes’.
For example, although a very low temperature can maximize the variance of all classes’ probabilities, the generated teacher’s label is one-hot that shows no distinctness between wrong classes.
In other words, the effectiveness of KD should be more related to the distinctness between wrong classes rather than all classes.
However, traditional temperature scaling applies a uniform temperature for all classes, which cannot separately handle the wrong classes.
4.3 Asymmetric Temperature Scaling
We conclude the above analysis: if a larger teacher makes an over-conﬁdent prediction, the wrong class probabilities it provides could be not discriminative enough.
Utilizing a uniform temperature could not enlarge the derived variance as much as possible with the interference of the target class’s logit (see the middle of Fig. 7).
Thanks to the decomposition in Eq.
2, the correct guidance term works similarly to the cross-entropy loss and allows us to deal with it separately.
Hence, we propose a novel temperature scaling approach:
[Equation]
[Figure]
ResNet → SFV
[Figure] [Figure]
where we take τ 1 > τ 2 > 0 .
This approach is named Asymmetric Temperature Scaling (ATS) because we apply different temperatures to the logits of correct and wrong classes.
According to Eq. 4, ATS could bring such beneﬁts when the teacher is over-conﬁdent:
• If the teacher outputs a larger logit f y for the correct class, a relatively larger τ 1 could decrease it to a reasonable magnitude, i.e., decreasing p y and increasing e ( q ) , and ﬁnally increasing the derived variance v ( q ) ; • If the teacher outputs less varied logits g for wrong classes, a relatively smaller temperature τ 2 could make them more diverse,
i
[Figure]
e.
, increasing v ( ˜q ) , ﬁnally increasing the derived variance v ( q ) .
ATS is more ﬂexible in enlarging the derived variance (see the right of Fig. 7), i.e., it could generate more discriminative distillation guidance during teaching.
Take the demo in Fig.
1 as an example, the smaller and larger teacher captures the same relative class afﬁnities, e.g., they both know that the fourth/ﬁfth class (shaded cells) is the most/least relevant to the target class.
However, with a uniform temperature 4 . 0 , the smaller teacher provides probabilities ( 0 . 073 , 0 . 060 ) for these two classes, while over-conﬁdent larger teachers provide ( 0 . 040 , 0 . 033 ) or ( 0 . 074 , 0 . 067 ) .
Clearly, the absolute afﬁnities of the larger teachers are not so discriminative as the smaller teacher’s.
Utilizing ATS, we could respectively apply ( τ 1 = 4 .
67 ,τ 2 = 4 . 0 ) or ( τ 1 = 4 . 0 ,τ 2 = 2 . 0 ) to the over-conﬁdent teacher’s logits, generating the same probability vector as the smaller teacher’s.
ATS utilizes two temperatures and creates the wiggle room to make the distribution over wrong classes more discriminative.
Experiments
We use CIFAR-10/CIFAR-100 [21], TinyImageNet [43], CUB [47], Stanford Dogs [19], and
Google Speech Commands [48] as the datasets.
For teacher networks, we use different versions of ResNet [11], WideResNet [56], ResNeXt [49].
For student networks, we use VGG [39], ShufﬂeNetV1/V2 [59, 29], AlexNet [22], MobileNetV2 [37], and DSCNN [60].
We majorly follow the training settings in [44] 1 .
Except that the Google Speech Commands takes 50 epochs, we train networks on other datasets with 240 epochs.
We use the SGD optimizer with 0.9 momentum.
For VGG, AlexNet, ResNet, WideResNet, and ResNeXt, we set the learning rate as 0.05 (recommended by [44]).
For ShufﬂeNet and MobileNet, we use a smaller learning rate of 0.
(recommended by [44]).
We use the pre-trained models provided in PyTorch for CUB and Stanford
Dogs, and correspondingly, their learning rates are scaled by 0 .
1 × .
During training, we decay the learning rate by 0.1 every 30 epochs after the ﬁrst 150 epochs (recommended by [44]).
For Google
Speech Commands, we decay the learning rate via the cosine annealing.
We set the batch size asfor CIFAR data, 64 for other datasets.
Other dataset, network and training details are in Appendix C.
5.1 Observations
Class discriminability matters a lot in KD and correlates with the KD improvement.
To show the importance of class discriminability in KD, we omit the distinctness of wrong class probabilities during distillation.
Speciﬁcally, we only keep the ﬁrst two terms in Eq. 2, which works similarly to
[Figure]
E S
[Figure]
CIFAR
[Figure] [Figure]
f y
CIFAR-100: σ ( g )
[Figure]
E
[Figure]
[Figure] [Figure]
N
[Figure]
CIFAR
[Figure] [Figure]
f
[Figure]
CIFAR
[Figure] [Figure]
N
[Figure]
N
[Figure] [Figure]
[Figure] [Figure]
[Figure]
the instance-speciﬁc label smoothing (abbreviated as “ILS”).
Fig.
2 shows students’ performances on
CIFAR-100.
Without the third term in Eq. 2, both small teachers (“ST”) and larger teachers (“LT”) teach worse signiﬁcantly.
Then, we investigate the correlations of KD performance improvement (i.e., the test accuracy change ratio with KD w.r.t.
without KD) with smooth regularization and class discriminability under 270 pairs of “(teacher, student, temperature)”.
Details are in Appendix C.4.
Fig.
3 plots the scatters, where dots show pairs whose improvement is higher than 2% .
Clearly, teachers with a larger derived variance tend to guide better.
These observations show the rationality of the proposed KD decomposition and imply that enhancing derived variance is beneﬁcial .
Larger teachers provide a larger target logit or less varied wrong logits.
We ﬁrst compare the logit distributions provided by larger and smaller teachers on the training set.
Fig.
4 plots the histograms (200 bins) of the target logit (i.e., f y ) and the standard deviation of wrong logits (i.e., σ ( g ) ).
The left and right, respectively, show the results on CIFAR-100 and CIFAR-10.
Clearly, the ﬁrst column shows that ResNet110 tends to generate a larger target logit (i.e., E x
[ f y ]
≈ 15 . 0 )
than
ResNet14 (i.e., E x
[ f y ] ≈ 10 . 0 ).
On CIFAR-10, the smallest f y given by WRN28-8 is larger than
WRN28-1.
Furthermore, WRN28-8 gives smaller variance (the fourth column), i.e., smaller inherent variance .
These reveal speciﬁc manifestations of complex networks’ over-conﬁdence .
Larger and smaller teachers have similar inherent variance while different derived variance under traditional temperature scaling.
We use τ = 1 . 0 to soften the complete logits and only the wrong class logits , respectively, and then show the mean and standard deviation of derived variance and inherent variance across training samples in Fig.
Although the larger models’ inherent variance is smaller, the difference between RNX29-4-4d and RNX29-64-4d is only up to 1 .
3 × .
However, the derived variance differs a lot, where the smaller teacher’s variance is approximately 9 .
7 × as the larger teacher’s.
These observations imply that traditional temperature scaling could seriously decrease the derived variance of larger teachers though they have appreciable inherent variance .
Complex teachers know approximately the same as smaller teachers on relative class afﬁnities.
Only if the relative magnitudes of the wrong class probabilities are correct, it is valid to enlarge the discrimination between them.
Otherwise, if a teacher himself misunderstands the knowledge’s principles, it will be counterproductive to reinforce this knowledge to students.
Given two teachers
T 1 and T 2 , we ﬁrst calculate the Spearman correlation between p T 1 and p T 2 .
Second, the set overlap
[Figure] [Figure]
V GG
ResNet
KDTSKDATS 0 .
S hu ﬄ e N e t V
KDTSKDATS
R E S 14 R E S 20 R E S 32 R E S 44 R E S 56 R E S 110 0 .0 .
M o b il e N e t V
KDTSKDATS 0 .
WideResNet
KDTSKDATS 0 .
KDTSKDATS
W R N 28 - 1 W R N 28 - 2 W R N 28 - 3 W R N 28 - 4 W R N 28 - 6 W R N 28 - 8 0 .0 .
KDTSKDATS 0 .
ResNeXt KDTSKDATS 0 .
KDTSKDATS
R N X 29 - 4 - 4 d R N X 29 - 8 - 4 d R N X 29 - 16 - 4 d R N X 29 - 24 - 4 d R N X 29 - 32 - 4 d R N X 29 - 64 - 4 d 0 .0 .
KDTSKDATS
Teacher
[Table]
between the top-5 predictions is calculated
[Table]
i.e., ∣C T 1 ∩ C T 2 ∣/∣C T 1 ∪ C T 2 ∣ .
C denotes the set of toppredicted classes.
Third, we calculate Kendall’s τ between p T 1 and p T 2 , which directly shows the rank correlation of two teachers.
These metrics only depend on classes’ relative magnitudes.
The results are in Fig.
Excitingly, these metrics among teachers with different capacities do not vary a lot, and the Spearman correlations are almost all larger than 0 . 85 .
According to the interpretation of Kendall’s τ [8, 53], if the smaller teacher predicts that class i is more related to the target class than that of class j , then the larger teacher has a probability of ( 0 .
75 + 1 )/ 2 = 0 .
875 to give the same relative afﬁnity.
As a comparison, we also calculate the distillation distance deﬁned in [15], which utilizes L1 distance and depends on the absolute magnitudes of probabilities.
Using this metric, the distance increases quickly as the capacity gap increases.
These observations demonstrate that teachers know approximately the same about relative class afﬁnities while their absolute values differ signiﬁcantly under traditional temperature scaling .
The proposed ATS could enlarge the derived variance of larger teachers.
The above analysis veriﬁes that the over-conﬁdent teachers experience lower derived variance under traditional temperature scaling although they grasp the relative class afﬁnities well.
For an intuitive visualization, we plot the probabilities for a randomly sampled instance whose correct class is y = 1 from CIFAR-100.
The top two rows in Fig.
8 show the results of RNX29-4-4d and RNX29-64-4d under traditional temperature scaling (TS), where the latter really experiences a smaller derived variance .
Using ATS could enhance the derived variance as shown at the bottom row (the bars are more jagged).
Then, we study the change of derived average (DA) and derived variance (DV) as temperature increases.
Given a τ , we obtain the DA and DV for all training samples via softmax and then calculate the average.
For ATS, we use τ 1 = 1 .
25 τ and τ 2 = 0 .
75 τ .
The curves are shown in Fig.
According to
Proposition 4.3, e
[Table]
Notably, this term changes nearly the same between teachers with various capacities.
However, the derived variance v ( q ) differs a lot.
Empirically, v ( q ) ﬁrst increases and then decreases, and the maximal of the larger teacher’s DV is smaller, which veriﬁes the Corollary 4.7.
Because derived variance corresponds to the efﬁcacy of class discriminability , this shows why larger teachers can’t teach well.
Using the proposed ATS could enhance the derived variance , which equivalently improves the efﬁcacy of class discriminability in KD.
We conclude that traditional temperature scaling leads to distillation labels with less discriminative information among wrong class probabilities; our proposed
ATS could enhance the discrimination among wrong classes and beneﬁt the distillation process .
5.2 Performances
ATS makes larger teachers teach well again.
Previous studies ﬁnd that more accurate teachers can’t necessarily teach well [6, 31].
As shown in Fig.
9, although we tune temperatures in { 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 12 . 0 , 16 . 0 } , larger teachers still teach worse under traditional temperature scaling (the solid curves).
However, using ATS (the dashed curves) could make larger teachers teach well or better again.
The details are in Appendix C.4.
ATS surpasses previous methods with advanced techniques.
We compare with SOTA methods and list the results on CIFAR-100, TinyImageNet, CUB, and Dogs in Tab. 2 and Tab. 3.
The sota compared methods include ESKD [6], TAKD [31], SCKD [51], and ResKD [10, 24].
NoKD trains students without the teacher’s supervision.
ST-KD trains students under the guidance of a smaller teacher.
KD trains students under the guidance of the larger teacher.
More details of compared methods are in Appendix C.3.
The last column of these tables shows the average performance of corresponding rows.
The larger teacher could slightly improve the students’ performances via traditional KD, i.e., 68 .
24% → 69 . 21% and 57 .
94 → 58 . 89% .
However, using a smaller teacher (the row of “ST-KD”) could obtain about 2% improvement on average, i.e., 68 .
24% → 70 .
32% and 57 .
94% → 59 .
This again veriﬁes that larger teachers really teach worse on various datasets.
Taking advantage of the early-stopped teacher (ESKD), the teacher assistant (TAKD), and the student customized teacher (SCKD) only improves the larger teacher slightly, e.g., 69 .
21% → 69 .
86% and 58 .
89% → 59 .
These results do not even surpass the small teacher.
ResKD improves the students’ performances a lot via introducing the residual student and taking the two students’ ensemble, which surpasses the ensemble performances of two separately trained students under different initializations (the “Ens” row).
For a fair comparison, we also test the performances of our methods via repeating the “KD+ATS” two times and making predictions via the ensemble.
The results are almost 1% higher than ResKD.
Results on CIFAR-10 and speech data are in Appendix D.
Ablation Studies One might argue that the validity of ATS is due to the tuning from a larger hyperparameter space.
We vary τ 1 from 1.0 to 6.0, τ 2 from 1.0 to 5.0, and record the performances in
Fig.
Obviously, setting τ 1 > τ 2 could be better, and especially, we recommend the setting of τ 2 ∈ [ τ 1 − 2 ,τ 1 − 1 ] .
Although we introduce one more hyper-parameter, ATS is simple to implement and surpasses SOTA methods that take advanced techniques.
We achieve the goal of only utilizing simple operations to make larger teachers teach well again.
Then, we study the trade off of the KD τ 2 = 1 .
τ 2 = 2 .
τ 2 = 3 .
τ 2 = 4 .
τ 2 = 5 .τ
1 = 1 .τ 1 = 2 .τ
= 3 .τ 1 = 4 .τ
1 = 5 .τ
= 6 .69.
CIFAR-100:WRN28-8 → VGG     τ 2 = 1 .
τ 2 = 2 .
τ 2 = 3 .
τ 2 = 4 .
τ 2 = 5 .63.53 64.44 63.97 63.14 63.39 65.92 64.58 64.68 64.32 64.
CIFAR-100:RNX29-64-4d → MV    τ 2 = 1 .
τ 2 = 2 .
τ 2 = 3 .
τ 2 = 4 .
τ 2 = 5 .45.54 45.20 46.39 46.18 45.59 49.20 47.
TinyImageNet:WRN50-2 → SFV
1 ,τ 2 in Eq. 5).
λ
=0.
λ
=0.
λ
=0.
λ
=0.
λ
=0.
T e s t
A cc u r a c y
WRN28-8 → MV
KD TS: τ =2.0 KD ATS:
τ 1 =3.0, τ 2 =2.
loss and CE loss under different λ .
The results of “WRN28-8 → MV2” on CIFAR-100 (Fig. 11) veriﬁes that ATS could improve the performances under various λ .
We also compare ATS with other types of KD under various scenes, and the results are in Appendix D.3.
Conclusion
We study the miraculous phenomenon in KD that a more accurate model doesn’t necessarily teach better.
The proposed KD decomposition attributes the success of a better teacher to three factors, including correct guidance , smooth regularization , and class discriminability .
Through theoretical analysis, over-conﬁdent teachers could not release their potential abilities of the class discriminability under traditional temperature scaling.
As a simple yet effective solution, we propose Asymmetric
Temperature Scaling (ATS) to enhance the derived variance of larger teachers, making their distillation labels more discriminative when teaching students.
Extensive experimental results verify the superiorities of our proposed methods.
Broader Impact
We focus on the variance of wrong class probabilities to analyze why larger teachers cannot teach well and hope that our research could bring a new perspective to the KD ﬁeld.
Our work has no potential negative societal impacts.
Acknowledgements
This work is partially supported by National Natural Science Foundation of China (Grant No.
61921006, 41901270), and Natural Science Foundation of Jiangsu Province (Grant No. BK20190296).
Thanks to Huawei Noah’s Ark Lab NetMIND Research Team.
We gratefully acknowledge the support of Mindspore used for this reasearch.
Professor De-Chuan Zhan is the corresponding author.
A A More Discussions About KD Decomposition (Eq. 2)
We decompose KD into three terms as shown in Eq.
2, i.e., the correct guidance , the smooth regularization , and the class discriminability .
We take a reverse thinking that how can we construct these three terms if without KD?
First, to ensure the learning process of the student is correct, the target class’s label should be the highest.
This is simple because we have one-hot labels of training samples.
That is, the cross-entropy loss could guarantee the student learns correctly.
Second, how can we introduce smooth regularization ?
This is also simple because we could utilize label smoothing (LS) as regularizations.
However, using cross-entropy and LS is still not as effective as KD.
Hence, the class discriminability also matters a lot.
This term is not easy to construct because we do not have the prior class similarities or the instance-level class similarities.
Fortunately, a teacher may contain the information of class similarities in KD.
We could ﬁnd that the class discriminability may matter more through the above analysis.
Hence, we focus on its efﬁcacy when analyzing why larger models can’t teach well.
Furthermore, the correct guidance is closely related to the second term, i.e., smooth regularization , as e ( q ) = 1 C − 1 ( 1 − p y ) .
That is, the strength of correct guidance is negatively correlated with the strength of smooth regularization .
As shown in Fig.
7, after applying ATS, the smooth regularization curves in these three subﬁgures are nearly the same, which indicates that the correct guidance term is slightly affected when utilizing ATS.
B Detailed Proofs of Lemma 4.1, Proposition 4.3 and Proposition 4.
We ﬁrst present some basic results about softmax operations.
The softmax operation is p c = exp ( f c / τ ) Z , where Z =
∑ Cj = 1 exp ( f j / τ ) .
We then present some basic derivatives:
[Equation]
Lemma B.1 (Variance of Softened Probabilities (Lemma 4.1 in the Body)) .
Given a logit vector f ∈ R C and the softened probability vector via softmax function, i.e., p = SF ( f ; τ ) ,τ ∈ ( 0 , ∞) , v ( p ) monotonically decreases as τ increases.
Proof.
Obviously, e ( p ) = 1 C .
Then, we have:
[Equation]
We take the derivative of v ( p ) w.r.t.
τ , and obtain:
[Equation]
Substituting f c = τ log p c + τ log Z , we get:
[Equation]
Then, we deﬁne ˆ p c = p 2 c /∑
Cc = 1 p 2 c ,
and we could derive the following equation:
[Equation]
We then calculate the KL divergence of ˆ p and p :
[Equation]
Due the non-negativity of KL divergence, we have:
[Equation]
Hence, we have:
[Equation]
where the ﬁrst inequality is according to Eq.
12 and the last inequality is according to Jensen’s inequality, This proves that ∂v ( p ) ∂τ ≤ 0 .
Proposition B.2 ((Proposition 4.3 in the Body)) .
Under the Assumption 4.2, p y monotonically decreases as τ increases, and e ( q ) monotonically increases as τ increases.
As τ → ∞ , e ( q ) → 1 / C .
Proof.
We take the derivative of p y w.r.t.
τ , and obtain:
[Equation]
which shows the monotonicity of p y w.r.t.
τ .
Then the later is obvious because e ( q ) = 1 C − 1 ∑ c
≠ y p c = 1 C − 1 ( 1 − p y ) .
Finally, according to the properties of limitation, lim τ →∞ p y =
lim
τ →∞
[Equation]
= 1 / C .
Proposition B.3 (Derived Variance vs. Inherent Variance (Proposition 4.4 in the Body)) .
The derived variance is determined by the square of derived average and the inherent variance via the following equation:
[Equation]
Proof.
With the property in Eq.
3, we have: ˜ q c ′ =
exp ( g c ′ / τ ) ∑ j exp ( g j / τ ) =
p c ∑ j ≠ y p
j = p c 1 − p y .
c ′ is the corresponding index of p c in q after removing the correct class probability p y , i.e., c ′ =
c when c < y , and c ′ =
c − 1 when c > y .
[Equation]
C Details of Datasets, Networks and Training C.1 Dataset Details
The datasets used in our experiments are CIFAR-10/CIFAR-100 (C10/C100) [21], TinyImageNet (TIN) [43], CUB [47], Stanford Dogs (Dogs) [19], and Google Speech Commands (GSC) [48].
CIFAR-10/CIFAR-100 are image classiﬁcation datasets, and each contains 50K training and 10K test samples of size 32 × 32 .
TinyImageNet contains 200 classes, 100K training samples, and 10K
[Table]
[Table]
test samples of size 64 × 64 .
CUB and Stanford Dogs are ﬁne-grained recognition datasets, which correspondingly contain 200 and 120 classes.
Google Speech Commands is a benchmark for keyword spotting [60, 41], which usually needs efﬁcient model deployment.
It aims to identify whether a 1s-long speech recording is a word, silence, or unknown.
There are total 35 classes (35 words) in this task.
We extract 40 MFCC features for each 30ms window frame with a stride of 10ms.
We also follow the settings in Google Speech Commands (GSC) [48]: performing random time-shift of Y ∈ [− 100 , 100 ] milliseconds and adding 0.1 volume background noise with a probability of 0.8.
The details are listed in Tab. 4.
C denotes the number of classes.
“Num.
Tr” and “Num.
Te” denote the number of training and test samples.
C.2 Network Details
For teacher networks, we use different versions of ResNet [11], WideResNet [56], ResNeXt [49].
For student networks, we use VGG [39], ShufﬂeNetV1/V2 [59, 29], AlexNet [22], MobileNetV2 [37], and DSCNN [60].
• ResNet (RES) :
We utilize ResNet for CIFAR-10/100 and Google Speech Commands as teachers.
For CIFAR-10/100, we use the original ResNet versions with “6n+2” layers in [11].
These ResNets only use the “basic block” for CIFAR data.
Speciﬁcally, we vary the number of layers in { 14 , 20 , 32 , 44 , 56 , 110 } .
For Google Speech Commands, we utilize the ResNet version proposed in [41].
We modify the number of layer as 15 and the channel as 128. • WideResNet (WRN) :
We utilize WideResNet for CIFAR-10/100 and TinyImageNet as teachers.
For CIFAR-10/100, we use the WideResNet proposed in [56].
We keep the depth as 28 and vary the widen factors in { 1 , 2 , 3 , 4 , 6 , 8 } .
For TinyImageNet, we use the WideResNet50-2 provided in PyTorch 2 .
• ResNeXt (RNX) :
We utilize ResNeXt for CIFAR-10/100, CUB and Stanford Dogs as teachers.
For CIFAR10-/100, we utilize the original ResNeXt versions for CIFAR data as proposed in [49].
We take 29 layers and set the base width as 4, varying the number of groups in { 4 , 8 , 16 , 24 , 32 , 64 } .
For CUB and Stanford Dogs, we use the ResNeXt101-32-8d provided in PyTorch.
• VGG8 : We utilize VGG8 [39] for CIFAR-10/100 as the student.
Although it contains more parameters, the plain architecture makes it weaker.
• ShufﬂeNetV1/2 (SFV1/2) :
We utilize ShufﬂeNet for CIFAR-10/100, TinyImageNet, CUB, and Dogs as students.
For CIFAR-10/100, we use ShufﬂeNetV1 [59] provided in RepDistiller repository 3 .
For other datasets, we use ShufﬂeNetV2 [29] provided in PyTorch.
• MobileNetV2 (MV2) :
We utilize MobileNet for CIFAR-10/100, TinyImageNet, CUB, and Dogs as students.
For CIFAR-10/100, we use MobileNetV2 [37] in RepDistiller repository.
For other datasets, we use ShufﬂeNetV2 [37] provided in PyTorch.
• AlexNet
(ANet) : We utilize AlexNet provided in PyTorch for TinyImageNet, CUB, and Dogs as students.
• DSCNN :
We use DSCNN [60] for Google Speech Commands as the student.
It utilizes depth-separable convolution to accelerate training and inference.
We set the basic number of channels as 96.
Additionally, for CUB and Stanford Dogs, we use the corresponding pre-trained models provided in
PyTorch as initialization.
The number of parameters (“N.P”) and the training and test accuracies are in Tab. 5.
C.3 Training Details
Train Teachers
We ﬁrst train teacher networks on corresponding datasets and then save the checkpoints of them.
Train Students We train students via the loss function in Eq.
We set λ = 0 .
5 by default and tune τ ∈ { 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 12 . 0 , 16 . 0 } .
Additionally, we vary the student’s τ in two settings: (1) the same
[Table]
as the teacher’s; (2) 1 . 0 .
For our proposed KD method, the pair of ( τ 1 ,τ 2 ) in Eq.
5 is searched in {( 2 . 0 , 1 . 0 ) , ( 3 . 0 , 1 . 0 ) , ( 3 . 0 , 2 . 0 ) , ( 4 . 0 , 2 . 0 ) , ( 4 . 0 , 3 . 0 ) , ( 5 . 0 , 2 . 0 )} .
For our method, we keep the student’s temperature as 1 . 0 by default.
Compared Methods We then explain the compared methods in Tab. 2 and Tab. 3.
For CIFAR100/CIFAR-10, we use ResNet110, WRN28-8, and RNX29-64-4d as teachers.
For TinyImageNet,
CUB, and Stanford Dogs, we use WRN50-2 and ResNeXt101-32-8d as teachers.
These models are complex networks with a larger depth, or a larger widen factor, or more groups.
We have also explored the standard deviation of these KD learning methods.
KD performances are relatively stable among several training random seeds with the same hyper-parameter settings.
The standard deviations are also nearly the same across different KD methods, which are about 0 .
Hence, we do not report the standard deviations due to space limitation.
• NoKD : training the student without distillation.
• ST-KD : training the student under the guidance of a smaller teacher.
For CIFAR-100, we select the best performance from all smaller networks as shown in Fig.
For TinyImageNet, CUB, and Stanford Dogs, we correspondingly use WRN26-2, RNX50-32-4d, RNX50-32-4d as smaller teachers.
• KD : training the student under the guidance of the larger teacher.
• ESKD [6]: training the student under the guidance of the early-stopped teacher .
Speciﬁcally, we train the larger teachers only for { 60 , 120 , 150 } epochs using the cosine annealing learning rate and report the best accuracy of the student.
• TAKD [31]: training the student under the guidance of the teacher assistant .
Speciﬁcally, we vary the TA in {ResNet20, ResNet44} for ResNet110, {WRN28-2, WRN28-4} for WRN28-8, {RNX29-8-4d, RNX29-8-24d} for RNX29-64-4d, {WRN26-2} for WRN50-2, and {RNX50-32-4d} for RNX101-32-8d.
• SCKD [51]: training the student under the student customized teacher .
We use two KD losses, including the KL divergence of probabilities (Eq. 1) and the L2 feature distance loss as done in [36].
• Ens : training two students with different initializations two times and taking their ensemble.
• ResKD : training the student ﬁrst and then training the residual student to ﬁt the residual.
The ensemble of these two students is used for inference.
• KD+ATS :
training the student with the larger teacher’s guidance via the proposed ATS.
• KD+ATS+Ens : repeating the above two times and taking the ensemble results.
When repeating “KD+ATS” two times, we use the same hyperparameters and keep the fairness of comparisons.
C.4 Figure Details
We explain some ﬁgures in detail.
[Table]
[Figure]
• Fig.
2: for “ResNet → SFV1”, we use ResNet14 as the small teacher (“ST”) and ResNet as the large teacher (“LT”); for “WRN → MV2”, we use WRN28-1 as the small teacher (“ST”) and WRN28-8 as the large teacher (“LT”).
• Fig.
3: on CIFAR-100, we vary 18 teachers and 3 students as listed in Tab. 5; we also vary temperatures in {1.0, 2.0, 4.0, 8.0, 16.0}; the total pairs of distillation combinations are 18 × 3 × 5 = 270 ; for each pair, we calculate the teacher’s derived average and derived variance averaged across 50K training samples; the KD improvement ratio is calculated by ( Acc KD − Acc NoKD )/ Acc NoKD .
• Fig.
6 and Fig.
5: the former calculates f y (i.e., target logit) and σ ( g ) (i.e., standard deviation of wrong logits) across training samples and plot the bins; the latter calculates σ ( q ) (i.e., derived variance) and σ ( ˜q ) (i.e., inherent variance) and only report their mean and standard deviation across training samples.
• Fig.
4: for each pair of two teachers T 1 and T 2 , we obtain their softened labels on all training samples; for each sample, we calculate the four metrics and report the average across training samples.
• Fig.
7: given a temperature τ , we could obtain softened probabilities of all training samples; then we calculate σ ( q ) (i.e., derived variance) and σ ( ˜q ) (i.e., inherent variance) across all training samples and only report the average results.
• Fig.
9: For TS, we tune τ ∈ { 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 12 . 0 , 16 . 0 } and vary the student’s τ in two settings: (1) the same as the teacher’s, (2) 1 . 0 ;
For ATS, the pair of ( τ 1 ,τ 2 ) in Eq.is searched in {( 2 . 0 , 1 . 0 ) , ( 3 . 0 , 1 . 0 ) , ( 3 . 0 , 2 . 0 ) , ( 4 . 0 , 2 . 0 ) , ( 4 . 0 , 3 . 0 ) , ( 5 . 0 , 2 . 0 )} ; through adjusting the hyper-parameters and selecting the best results, we plot the performance curves.
7.5 10.0 Temperature 0 .. 0040 .
D e r
i v e d
A v e r a g e : e ( q )
WRN28-1:TS e ( q ) σ ( q ) 0.5 2.
7.5 10.0 Temperature
WRN28-8:TS e ( q ) σ ( q ) 0.5 2.
7.5 10.0 Temperature
WRN28-8:ATS e ( q ) 0 .. 00500 .
D e r
i v e d
V a r i a n ce : σ ( q ) σ ( q )
[Table]
D More Experimental Studies D.1 CIFAR-10 Results
We also compare our proposed methods with SOTA on CIFAR-10, and the results are listed in Tab. 6.
Because CIFAR-10 only contains 10 classes and is an slightly simpler benchmark, the performance improvement is not so obvious as other datasets.
D.2 Speech Data Results
ATS could also perform well on speech data, i.e., the Google Speech Commands benchmark as shown in Tab. 8, where we vary several types of KD loss.
Our proposed ATS does not depend on multiplying τ 2 or not in Eq.
D.3 Comparisons With Other KD Methods on CIFAR-
We also compare our proposed method with other KD methods, such as [44, 17, 34, 4, 16].
The previous work [44] provides a comprehensive experimental study on many KD methods.
For fair comparison, we directly cite their results and only report several scenes of “Larger Teacher →
Smaller Student”, i.e., “ResNet110 ( 74 . 31% ) → ResNet20 ( 69 . 06% )” and “VGG13 ( 74 . 64% ) →
VGG8 ( 70 . 36% )”.
For IE-KD [16], we use the proposed variant of IE-AT.
For IE-KD [16] and
ReviewKD [4], we utilize the public code that the authors provide.
We also compare with these two methods because they are the recently proposed SOTA KD methods.
The results are listed in Tab. 7.
Consistently, our method could improve the KD up to 1% and the results are comparable with CRD that utilizes more advanced techniques.
We also apply ATS to the cases that students have almost the same capacity as teachers, e.g., “WRN40-2 ( 75 . 61% ) → WRN16-2 ( 73 .
and “ResNet ( 72 . 34% )
→ ResNet20 ( 69 . 06% )”.
Under these scenes, our method becomes not so effective and the performance improvement over KD seems more like the beneﬁts of hyper-parameter tuning.
Hence, our proposed ATS may be more effective when applying a more complex teacher to guide the learning process of a smaller student.
D.4 More Results for the Observations
We also present additional results for the observations in Fig.
8 and Fig.
7 to verify that the observations and conclusions are not accidental.
Similar results are shown in Fig.
12 and Fig.