Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
Abstract
Knowledge Distillation (KD) aims at transferring the knowledge of a wellperformed neural network (the teacher ) to a weaker one (the student ).
A peculiar phenomenon is that a more accurate model doesn’t necessarily teach better, and temperature adjustment can neither alleviate the mismatched capacity.
To explain this, we decompose the efﬁcacy of KD into three parts: correct guidance , smooth regularization , and class discriminability .
The last term describes the distinctness of wrong class probabilities that the teacher provides in KD.
Complex teachers tend to be over-conﬁdent and traditional temperature scaling limits the efﬁcacy of class discriminability , resulting in less discriminative wrong class probabilities.
Therefore, we propose Asymmetric Temperature Scaling (ATS) , which separately applies a higher/lower temperature to the correct/wrong class.
ATS enlarges the variance of wrong class probabilities in the teacher’s label and makes the students grasp the absolute afﬁnities of wrong classes to the target class as discriminative as possible.
Both theoretical analysis and extensive experimental results demonstrate the effectiveness of ATS.
The demo developed in Mindspore is available at https://gitee.com/lxcnju/ats-mindspore and will be available at https: //gitee.com/mindspore/models/tree/master/research/cv/ats .
Introduction
Although large-scale deep neural networks have achieved overwhelming successes in many realworld applications [22, 11, 60], the vast capacity hinders them from being deployed on portable devices with limited computation and storage resources [3].
Some efﬁcient architectures, e.g.,
MobileNets [14, 37] and ShufﬂeNets [59, 29], have been proposed for lightweight deployment, while their performances are usually constrained.
Fortunately, knowledge distillation (KD) [46, 13] could transfer the knowledge of a more complex and well-performed network (i.e., the teacher ) to them.
The original KD [13] forces the student to mimic the teacher’s behavior via minimizing the KullbackLeibler (KL) divergence between their output probabilities.
Recent studies generalize KD to various types of knowledge [36, 57, 17, 12, 33, 1, 34, 44, 52, 27, 54, 45, 50, 26, 23] or various distillation schemes [61, 2, 58, 20].
An intuitive sense after the proposal of KD [13] is that larger teachers could teach students better because their accuracies are higher.
A recent work [6] ﬁrst points out that the teacher accuracy is a poor predictor of the student’s performance.
That is, more accurate neural networks don’t necessarily teach better.
Until now, this phenomenon is still counter-intuitive [51], surprising [31], and unexplored [24].
Different from some existing empirical studies and theoretical analysis [40, 18, 30, 35, 55, 63, 6, 28, 15], we investigate the miraculous phenomenon in detail and aim to answer the following questions: What’s the real reason that more complex teachers can’t teach
[Figure]
of KD
[Figure]
well?
Is it really impossible to make larger teachers teach better through simple operations, such as temperature scaling?
To answer the ﬁrst question, we focus on analyzing the distinctness of wrong class probabilities that a teacher provides in KD.
We decompose the teacher’s label into three parts (see Sect.
4.1): (I)
Correct Guidance : the correct class’s probability; (II) Smooth Regularization : the average probability of wrong classes; (III) Class Discriminability : the variance of wrong class probabilities (deﬁned as derived variance ).
The commonly utilized temperature scaling could control the efﬁcacy of these three terms (the left of Fig. 1).
More complex teachers are over-conﬁdent and assign a larger score for the correct class or less varied scores for the wrong classes.
If we use a uniform temperature to scale their logits, the class discriminability of the larger teacher is less effective (theoretically analyzed in
Sect. 4.2), i.e., the probabilities of wrong classes are less distinct (the right of Fig. 1).
As to the second question, we focus on enlarging the variance of wrong class probabilities (i.e., derived variance ) that a teacher provides to make the distillation process more discriminative.
To speciﬁcally enhance the distinctness of wrong class probabilities, we separately apply a higher/lower temperature to the correct/wrong class’s logit instead of a uniform temperature (see Sect. 4.3).
We name our method Asymmetric Temperature Scaling (ATS) , and abundant experimental studies verify that utilizing this simple operation could make larger teachers teach well again.
Related Works
KD with Larger Teacher : Although KD has been a general technique for knowledge transfer in various applications [13, 61, 42, 25], could any student learn from any teacher?
[6] ﬁrst studies the
KD’s dependence on student and teacher architectures.
They ﬁnd that larger models do not often make better teachers and propose the early-stopped teacher as a solution.
[31] introduces a multi-step
KD process, employing an intermediate-sized network (the teacher assistant ) to bridge the capacity gap.
[51] formulates KD as a multi-task learning problem with several knowledge transfer losses.
The transfer loss will be utilized only when its gradient direction is consistent with the cross-entropy loss.
[10, 24] deﬁne the knowledge gap as residual , which is utilized to teach the residual student , and then they take the ensemble of the student and residual student for inference.
These works attribute the worse teaching performance to capacity mismatch, i.e., weaker students can’t completely mimic the excellent teachers.
However, they don’t explain this peculiar phenomenon in detail.
Understanding of KD :
Quite a few works focus on understanding the advantages of KD from a principled perspective.
[28] uniﬁes KD and privileged information into generalized distillation .
[35, 18] utilize gradient ﬂow and neural tangent kernel to analyze the convergence property of KD under deep linear networks and inﬁnitely wide networks.
[5] explains KD via quantifying the task-
[Table]
f
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
relevant and task-irrelevant visual concepts.
[7] casts KD as a semiparametric inference problem and proposes corresponding enhancements.
Our work is more related to KD decompositions.
[9] treats the teacher’s correct/wrong outputs differently, respectively explaining them as importance weighting and class similarities.
[40] further decomposes the “dark knowledge” into universal knowledge, domain knowledge, and gradient rescaling.
[30] establishes a bias-variance tradeoff to quantify the divergence of a teacher with the Bayes teacher .
[63] utilizes bias-variance decomposition to analyze
KD and discovers regularization samples that could increase bias and decrease variance.
Our work is also related to Label smoothing (LS).
[55] points out that the regularization effect in KD is similar to
LS.
[32] ﬁnds that training a teacher with LS could degrade its teaching quality, and attributes this to the fact that LS erases relative information between teacher logits.
Recently, [38] further studies this problem and proposes a metric to measure the degree of erased information quantitatively.
Our work also decomposes KD into several effects to study why more complex teachers can’t teach well.
Detailed relatedness to these works is presented in Sect.
Background
We consider a C -class classiﬁcation problem with Y =
[ C ] = { 1 , 2 , ⋯ ,C } .
Given a neural network and a sample pair ( x ,y ) , we could obtain the “logits” as f ( x ) ∈ R C .
We denote the softmax function with temperature τ as SF (⋅ ; τ ) , i.e., p c ( τ ) =
exp ( f c ( x )/ τ )/
Z ( τ ) and Z ( τ ) =
∑ Cj = 1 exp ( f j ( x )/ τ ) , where p ( τ ) is the softened probability vector that a network outputs and c is the index of class.
Later, we may omit the dependence on x and τ if without any ambiguity.
We use f y and p y to denote the correct class ’s logit and probability, while we use g and q to represent the vector of wrong classes ’ logits and probabilities, i.e., g =
[ f c ] c ≠ y and q =
[ p c ] c ≠ y .
The notations could be found in Tab. 1.
The most standard KD [13] contains two stages of training.
The ﬁrst stage trains complex teachers, and then the second stage transfers the knowledge from teachers to a smaller student via minimizing the KL divergence between softened probabilities.
Usually, the loss function during the second stage (i.e., the student’s learning objective) is a combination of cross-entropy loss and distillation loss:
[Equation]
KD Loss , (1) where the upper script “T”/“S” denotes “Teacher”/“Student” respectively.
Commonly, a default temperature of 1 is utilized for the CE loss, and the student could also take a temperature of 1 for the
KD loss, e.g., p Sc ( τ = 1 ) [13, 31, 45, 44].
Suppose we have two teachers, denoted as T large and T small , and the larger teacher performs better on both training and test data.
If we use them to teach the same student S , we could ﬁnd that the student’s performance is worse when mimicking the larger teacher’s outputs.
Adjusting the temperature could neither make the larger teacher teach well.
The details of this phenomenon could be found in [6, 31] and Fig.
Obviously, p T large could differ a lot from p T small , which is the only difference in the loss function when teaching the student.
Hence, we focus on analyzing what probability distributions are tended to be provided by teachers with different capacities .
Proposed Methods
This section ﬁrst decomposes KD into three parts and deﬁnes several quantitative metrics.
Then, we present theoretical analysis to demonstrate why larger networks can’t teach well.
Finally, we propose a more appropriate temperature scaling approach as an alternative
[Table]
4.1 KD Decomposition
We omit the coefﬁcient of λτ 2 in Eq.
[Table]
and deﬁne e ( q T ( τ ))
= 1 C −
∑ Cj = 1 ,j ≠ y p Tj ( τ ) , where q T ( τ ) =
[ p Tc ( τ )] c ≠ y .
Then, we have the following decomposition:
[Equation]
[Table]
(2) (I) Correct Guidance : this term guarantees correctness during teaching.
The decomposition in [9] also contains this term, which is explained as importance weighting.
This term works similarly to the cross-entropy loss, which could be dealt with separately when applying temperature scaling.
(II) Smooth Regularization : some previous works [55, 62, 63] attribute the success of KD to the efﬁcacy of regularization and study its relation to label smoothing (LS).
The combination of this term with correct guidance works similarly to LS.
Notably, e ( q T ( τ )) differs across samples, implying that the strength of smoothing is instance-speciﬁc, which is similar to the analysis in [62].
(III) Class
Discriminability : this term tells the student the afﬁnity of wrong classes to the correct class.
Transferring the knowledge of class similarities to students has been the mainstream guess of the “dark knowledge” in KD [13, 38].
Ideally, a good teacher should be as discriminating as possible in telling students which classes are more related to the correct class.
Illustrations of the decomposition are presented in the left of Fig.
Obviously, an appropriate temperature should simultaneously contain the efﬁcacy of the three terms, e.g., the shaded row in
Fig.
A too high or too low temperature could lead to smaller class discriminability , making the guidance less different among wrong classes, which weakens the distillation performance in practical.
Among these three terms, we advocate that class discriminability is more fundamental in KD and present more discussions in Appendix A (veriﬁed in Fig.
2 and Fig. 3).
To measure these three terms quantitatively, we use the target class probability (i.e., p y ), the average of wrong class probabilities (i.e., e ( q ) = 1 C − 1 ∑ j ≠ y p j ), and the variance of wrong class probabilities (i.e., v ( q ) =
1 C − 1 ∑ j ≠ y ( p j − e ( q )) 2 ) as estimators.
e (⋅) and v (⋅) calculates the mean and variance of the elements in a vector.
In some cases, we use the standard deviation as an estimator for the third term, i.e., σ ( q ) =
v 1 / 2 ( q ) .
Because the latter two terms are calculated after applying softmax to the complete logit vector, we deﬁne them as Derived Average (DA) and Derived Variance (DV) , respectively.
In experiments, we calculate these metrics for all training samples and sometimes report the average or standard deviation across these samples.
4.2 Theoretical Analysis
We analyze the mean and variance of the softened probability vector, i.e., the teacher’s label p T ( τ ) used in KD.
We defer the proofs of Lemma 4.1 and Proposition 4.3, 4.4 to Appendix B.
Lemma 4.1 (Variance of Softened Probabilities) .
Given a logit vector f ∈ R C and the softened probability vector p =
SF ( f ; τ ) ,τ ∈ ( 0 , ∞) , v ( p ) monotonically decreases as τ increases.
As τ increases, p ( τ ) becomes more uniform, i.e., it’s entropy increases.
However, we especially focus on the wrong classes, where the mean and variance are more intuitive to calculate and analyze.
Assumption 4.2.
The target logit is higher than other classes’ logits, i.e., f y ≥ f c , ∀ c ≠ y .
Assumption 4.2 is rational because well-performed teachers could almost achieve a higher accuracy (e.g., >95%) on the training set, and most training samples meet this requirement.
Proposition 4.3.
Under Assumption 4.2, p y monotonically decreases as τ increases, and e ( q ) monotonically increases as τ increases.
As τ → ∞ , e ( q ) → 1 / C .
Proposition 4.3 implies that increasing temperature could lead to a higher derived average (empirically see Fig. 7) and strengthen the smooth regularization term in Eq. 2.
Before we analyze the class discriminability term, we deﬁne ˜
q ( τ ) as the result of applying softmax only to the wrong logits with temperature τ , i.e., ˜ q ( τ ) =
SF ( g ; τ ) .
For the element index c ′ of q , we have
[Equation]
Notably, ˜ q differs from q a lot.
Speciﬁcally, the former satisﬁes ∑ c ′ ˜ q c ′ =
1 , while the summation of the latter is ∑ c ≠ y p c = 1 − p y .
The former does not depend on the correct class’s logit while the latter does.
We name v ( ˜ q ) Inherent Variance (IV) because it only depends on wrong classes’ logits.
Proposition 4.4 (Derived Variance vs. Inherent Variance) .
The derived variance is determined by the square of derived average and the inherent variance via:
[Equation]
With τ increases, e ( q ) increases (Proposition 4.3) while v ( ˜ q ) decreases (Lemma 4.1), and hence, it is not so easy to judge the speciﬁc monotonicity of v ( q ) w.r.t.
τ .
Empirically, we observe that the derived variance ﬁrst increases and then decreases (see Fig. 7), which conforms to the change of the class discriminability as illustrated in Fig.
We could use Proposition 4.4 to clearly analyze why larger teacher networks can’t teach well.
Before this, we present another two properties and a corollary without detailed proof.
Remark 4.5 .
Fixing g and τ , a higher target logit f
y leads to a higher p y , i.e., a smaller derived average e ( q ) .
Remark 4.6 .
Fixing τ , less varied wrong logits g leads to less varied ˜ q , i.e., a smaller inherent variance v ( ˜ q ) .
Corollary 4.7.
Suppose we have two teachers T 1 and T 2 , and their logit vectors for a same sample are f T 1 and f T 2 .
• If f T 1 y ≥ f T 2 y while g T 1 and g T 2 are nearly the same, then p T 1 y ≥ p T 2 y (Remark 4.5) while v ( ˜ q T 1 )
≈ v ( ˜ q T 2 ) .
Hence, v ( q T 1 ) ≤ v ( q T 2 ) .
• If f T 1 y ≈ f T 2 y while v ( g T 1 ) ≤ v ( g T 2 ) , then p T 1 y ≈ p
T 2 y while v ( ˜ q T 1 ) ≤ v ( ˜ q T 2 )
(Remark 4.6).
Hence, v ( q T 1 ) ≤ v ( q T 2 ) .
This corollary explains why a larger teacher can’t teach better.
Because the larger teacher tends to be over-conﬁdent, the target logit f
y may be larger or the variance of wrong logits v ( g ) may be smaller.
These are illustrated in Fig.
1 and empirically veriﬁed in Fig.
Then the derived variance v ( q ) may be smaller, limiting the efﬁcacy of class discriminability in Eq. 2.
Empirical results are in Fig.
Notably, we focus on analyzing the variance of wrong class probabilities instead of all classes.
Maximizing the variance of all classes’ probabilities does not mean maximizing the variance of wrong classes’.
For example, although a very low temperature can maximize the variance of all classes’ probabilities, the generated teacher’s label is one-hot that shows no distinctness between wrong classes.
In other words, the effectiveness of KD should be more related to the distinctness between wrong classes rather than all classes.
However, traditional temperature scaling applies a uniform temperature for all classes, which cannot separately handle the wrong classes.
4.3 Asymmetric Temperature Scaling
We conclude the above analysis: if a larger teacher makes an over-conﬁdent prediction, the wrong class probabilities it provides could be not discriminative enough.
Utilizing a uniform temperature could not enlarge the derived variance as much as possible with the interference of the target class’s logit (see the middle of Fig. 7).
Thanks to the decomposition in Eq.
2, the correct guidance term works similarly to the cross-entropy loss and allows us to deal with it separately.
Hence, we propose a novel temperature scaling approach:
[Equation]
[Figure]
ResNet → SFV
[Figure] [Figure]
where we take τ 1 > τ 2 > 0 .
This approach is named Asymmetric Temperature Scaling (ATS) because we apply different temperatures to the logits of correct and wrong classes.
According to Eq. 4, ATS could bring such beneﬁts when the teacher is over-conﬁdent:
• If the teacher outputs a larger logit f y for the correct class, a relatively larger τ 1 could decrease it to a reasonable magnitude, i.e., decreasing p y and increasing e ( q ) , and ﬁnally increasing the derived variance v ( q ) ; • If the teacher outputs less varied logits g for wrong classes, a relatively smaller temperature τ 2 could make them more diverse,
i
[Figure]
e.
, increasing v ( ˜q ) , ﬁnally increasing the derived variance v ( q ) .
ATS is more ﬂexible in enlarging the derived variance (see the right of Fig. 7), i.e., it could generate more discriminative distillation guidance during teaching.
Take the demo in Fig.
1 as an example, the smaller and larger teacher captures the same relative class afﬁnities, e.g., they both know that the fourth/ﬁfth class (shaded cells) is the most/least relevant to the target class.
However, with a uniform temperature 4 . 0 , the smaller teacher provides probabilities ( 0 . 073 , 0 . 060 ) for these two classes, while over-conﬁdent larger teachers provide ( 0 . 040 , 0 . 033 ) or ( 0 . 074 , 0 . 067 ) .
Clearly, the absolute afﬁnities of the larger teachers are not so discriminative as the smaller teacher’s.
Utilizing ATS, we could respectively apply ( τ 1 = 4 .
67 ,τ 2 = 4 . 0 ) or ( τ 1 = 4 . 0 ,τ 2 = 2 . 0 ) to the over-conﬁdent teacher’s logits, generating the same probability vector as the smaller teacher’s.
ATS utilizes two temperatures and creates the wiggle room to make the distribution over wrong classes more discriminative.
Experiments
We use CIFAR-10/CIFAR-100 [21], TinyImageNet [43], CUB [47], Stanford Dogs [19], and
Google Speech Commands [48] as the datasets.
For teacher networks, we use different versions of ResNet [11], WideResNet [56], ResNeXt [49].
For student networks, we use VGG [39], ShufﬂeNetV1/V2 [59, 29], AlexNet [22], MobileNetV2 [37], and DSCNN [60].
We majorly follow the training settings in [44] 1 .
Except that the Google Speech Commands takes 50 epochs, we train networks on other datasets with 240 epochs.
We use the SGD optimizer with 0.9 momentum.
For VGG, AlexNet, ResNet, WideResNet, and ResNeXt, we set the learning rate as 0.05 (recommended by [44]).
For ShufﬂeNet and MobileNet, we use a smaller learning rate of 0.
(recommended by [44]).
We use the pre-trained models provided in PyTorch for CUB and Stanford
Dogs, and correspondingly, their learning rates are scaled by 0 .
1 × .
During training, we decay the learning rate by 0.1 every 30 epochs after the ﬁrst 150 epochs (recommended by [44]).
For Google
Speech Commands, we decay the learning rate via the cosine annealing.
We set the batch size asfor CIFAR data, 64 for other datasets.
Other dataset, network and training details are in Appendix C.
5.1 Observations
Class discriminability matters a lot in KD and correlates with the KD improvement.
To show the importance of class discriminability in KD, we omit the distinctness of wrong class probabilities during distillation.
Speciﬁcally, we only keep the ﬁrst two terms in Eq. 2, which works similarly to
[Figure]
E S
[Figure]
CIFAR
[Figure] [Figure]
f y
CIFAR-100: σ ( g )
[Figure]
E
[Figure]
[Figure] [Figure]
N
[Figure]
CIFAR
[Figure] [Figure]
f
[Figure]
CIFAR
[Figure] [Figure]
N
[Figure]
N
[Figure] [Figure]
[Figure] [Figure]
[Figure]
the instance-speciﬁc label smoothing (abbreviated as “ILS”).
Fig.
2 shows students’ performances on
CIFAR-100.
Without the third term in Eq. 2, both small teachers (“ST”) and larger teachers (“LT”) teach worse signiﬁcantly.
Then, we investigate the correlations of KD performance improvement (i.e., the test accuracy change ratio with KD w.r.t.
without KD) with smooth regularization and class discriminability under 270 pairs of “(teacher, student, temperature)”.
Details are in Appendix C.4.
Fig.
3 plots the scatters, where dots show pairs whose improvement is higher than 2% .
Clearly, teachers with a larger derived variance tend to guide better.
These observations show the rationality of the proposed KD decomposition and imply that enhancing derived variance is beneﬁcial .
Larger teachers provide a larger target logit or less varied wrong logits.
We ﬁrst compare the logit distributions provided by larger and smaller teachers on the training set.
Fig.
4 plots the histograms (200 bins) of the target logit (i.e., f y ) and the standard deviation of wrong logits (i.e., σ ( g ) ).
The left and right, respectively, show the results on CIFAR-100 and CIFAR-10.
Clearly, the ﬁrst column shows that ResNet110 tends to generate a larger target logit (i.e., E x
[ f y ]
≈ 15 . 0 )
than
ResNet14 (i.e., E x
[ f y ] ≈ 10 . 0 ).
On CIFAR-10, the smallest f y given by WRN28-8 is larger than
WRN28-1.
Furthermore, WRN28-8 gives smaller variance (the fourth column), i.e., smaller inherent variance .
These reveal speciﬁc manifestations of complex networks’ over-conﬁdence .
Larger and smaller teachers have similar inherent variance while different derived variance under traditional temperature scaling.
We use τ = 1 . 0 to soften the complete logits and only the wrong class logits , respectively, and then show the mean and standard deviation of derived variance and inherent variance across training samples in Fig.
Although the larger models’ inherent variance is smaller, the difference between RNX29-4-4d and RNX29-64-4d is only up to 1 .
3 × .
However, the derived variance differs a lot, where the smaller teacher’s variance is approximately 9 .
7 × as the larger teacher’s.
These observations imply that traditional temperature scaling could seriously decrease the derived variance of larger teachers though they have appreciable inherent variance .
Complex teachers know approximately the same as smaller teachers on relative class afﬁnities.
Only if the relative magnitudes of the wrong class probabilities are correct, it is valid to enlarge the discrimination between them.
Otherwise, if a teacher himself misunderstands the knowledge’s principles, it will be counterproductive to reinforce this knowledge to students.
Given two teachers
T 1 and T 2 , we ﬁrst calculate the Spearman correlation between p T 1 and p T 2 .
Second, the set overlap
[Figure] [Figure]
V GG
ResNet
KDTSKDATS 0 .
S hu ﬄ e N e t V
KDTSKDATS
R E S 14 R E S 20 R E S 32 R E S 44 R E S 56 R E S 110 0 .0 .
M o b il e N e t V
KDTSKDATS 0 .
WideResNet
KDTSKDATS 0 .
KDTSKDATS
W R N 28 - 1 W R N 28 - 2 W R N 28 - 3 W R N 28 - 4 W R N 28 - 6 W R N 28 - 8 0 .0 .
KDTSKDATS 0 .
ResNeXt KDTSKDATS 0 .
KDTSKDATS
R N X 29 - 4 - 4 d R N X 29 - 8 - 4 d R N X 29 - 16 - 4 d R N X 29 - 24 - 4 d R N X 29 - 32 - 4 d R N X 29 - 64 - 4 d 0 .0 .
KDTSKDATS
Teacher
[Table]
between the top-5 predictions is calculated
[Table]
i.e., ∣C T 1 ∩ C T 2 ∣/∣C T 1 ∪ C T 2 ∣ .
C denotes the set of toppredicted classes.
Third, we calculate Kendall’s τ between p T 1 and p T 2 , which directly shows the rank correlation of two teachers.
These metrics only depend on classes’ relative magnitudes.
The results are in Fig.
Excitingly, these metrics among teachers with different capacities do not vary a lot, and the Spearman correlations are almost all larger than 0 . 85 .
According to the interpretation of Kendall’s τ [8, 53], if the smaller teacher predicts that class i is more related to the target class than that of class j , then the larger teacher has a probability of ( 0 .
75 + 1 )/ 2 = 0 .
875 to give the same relative afﬁnity.
As a comparison, we also calculate the distillation distance deﬁned in [15], which utilizes L1 distance and depends on the absolute magnitudes of probabilities.
Using this metric, the distance increases quickly as the capacity gap increases.
These observations demonstrate that teachers know approximately the same about relative class afﬁnities while their absolute values differ signiﬁcantly under traditional temperature scaling .
The proposed ATS could enlarge the derived variance of larger teachers.
The above analysis veriﬁes that the over-conﬁdent teachers experience lower derived variance under traditional temperature scaling although they grasp the relative class afﬁnities well.
For an intuitive visualization, we plot the probabilities for a randomly sampled instance whose correct class is y = 1 from CIFAR-100.
The top two rows in Fig.
8 show the results of RNX29-4-4d and RNX29-64-4d under traditional temperature scaling (TS), where the latter really experiences a smaller derived variance .
Using ATS could enhance the derived variance as shown at the bottom row (the bars are more jagged).
Then, we study the change of derived average (DA) and derived variance (DV) as temperature increases.
Given a τ , we obtain the DA and DV for all training samples via softmax and then calculate the average.
For ATS, we use τ 1 = 1 .
25 τ and τ 2 = 0 .
75 τ .
The curves are shown in Fig.
According to
Proposition 4.3, e
[Table]
Notably, this term changes nearly the same between teachers with various capacities.
However, the derived variance v ( q ) differs a lot.
Empirically, v ( q ) ﬁrst increases and then decreases, and the maximal of the larger teacher’s DV is smaller, which veriﬁes the Corollary 4.7.
Because derived variance corresponds to the efﬁcacy of class discriminability , this shows why larger teachers can’t teach well.
Using the proposed ATS could enhance the derived variance , which equivalently improves the efﬁcacy of class discriminability in KD.
We conclude that traditional temperature scaling leads to distillation labels with less discriminative information among wrong class probabilities; our proposed
ATS could enhance the discrimination among wrong classes and beneﬁt the distillation process .
5.2 Performances
ATS makes larger teachers teach well again.
Previous studies ﬁnd that more accurate teachers can’t necessarily teach well [6, 31].
As shown in Fig.
9, although we tune temperatures in { 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 12 . 0 , 16 . 0 } , larger teachers still teach worse under traditional temperature scaling (the solid curves).
However, using ATS (the dashed curves) could make larger teachers teach well or better again.
The details are in Appendix C.4.
ATS surpasses previous methods with advanced techniques.
We compare with SOTA methods and list the results on CIFAR-100, TinyImageNet, CUB, and Dogs in Tab. 2 and Tab. 3.
The sota compared methods include ESKD [6], TAKD [31], SCKD [51], and ResKD [10, 24].
NoKD trains students without the teacher’s supervision.
ST-KD trains students under the guidance of a smaller teacher.
KD trains students under the guidance of the larger teacher.
More details of compared methods are in Appendix C.3.
The last column of these tables shows the average performance of corresponding rows.
The larger teacher could slightly improve the students’ performances via traditional KD, i.e., 68 .
24% → 69 . 21% and 57 .
94 → 58 . 89% .
However, using a smaller teacher (the row of “ST-KD”) could obtain about 2% improvement on average, i.e., 68 .
24% → 70 .
32% and 57 .
94% → 59 .
This again veriﬁes that larger teachers really teach worse on various datasets.
Taking advantage of the early-stopped teacher (ESKD), the teacher assistant (TAKD), and the student customized teacher (SCKD) only improves the larger teacher slightly, e.g., 69 .
21% → 69 .
86% and 58 .
89% → 59 .
These results do not even surpass the small teacher.
ResKD improves the students’ performances a lot via introducing the residual student and taking the two students’ ensemble, which surpasses the ensemble performances of two separately trained students under different initializations (the “Ens” row).
For a fair comparison, we also test the performances of our methods via repeating the “KD+ATS” two times and making predictions via the ensemble.
The results are almost 1% higher than ResKD.
Results on CIFAR-10 and speech data are in Appendix D.
Ablation Studies One might argue that the validity of ATS is due to the tuning from a larger hyperparameter space.
We vary τ 1 from 1.0 to 6.0, τ 2 from 1.0 to 5.0, and record the performances in
Fig.
Obviously, setting τ 1 > τ 2 could be better, and especially, we recommend the setting of τ 2 ∈ [ τ 1 − 2 ,τ 1 − 1 ] .
Although we introduce one more hyper-parameter, ATS is simple to implement and surpasses SOTA methods that take advanced techniques.
We achieve the goal of only utilizing simple operations to make larger teachers teach well again.
Then, we study the trade off of the KD τ 2 = 1 .
τ 2 = 2 .
τ 2 = 3 .
τ 2 = 4 .
τ 2 = 5 .τ
1 = 1 .τ 1 = 2 .τ
= 3 .τ 1 = 4 .τ
1 = 5 .τ
= 6 .69.
CIFAR-100:WRN28-8 → VGG     τ 2 = 1 .
τ 2 = 2 .
τ 2 = 3 .
τ 2 = 4 .
τ 2 = 5 .63.53 64.44 63.97 63.14 63.39 65.92 64.58 64.68 64.32 64.
CIFAR-100:RNX29-64-4d → MV    τ 2 = 1 .
τ 2 = 2 .
τ 2 = 3 .
τ 2 = 4 .
τ 2 = 5 .45.54 45.20 46.39 46.18 45.59 49.20 47.
TinyImageNet:WRN50-2 → SFV
1 ,τ 2 in Eq. 5).
λ
=0.
λ
=0.
λ
=0.
λ
=0.
λ
=0.
T e s t
A cc u r a c y
WRN28-8 → MV
KD TS: τ =2.0 KD ATS:
τ 1 =3.0, τ 2 =2.
loss and CE loss under different λ .
The results of “WRN28-8 → MV2” on CIFAR-100 (Fig. 11) veriﬁes that ATS could improve the performances under various λ .
We also compare ATS with other types of KD under various scenes, and the results are in Appendix D.3.
Conclusion
We study the miraculous phenomenon in KD that a more accurate model doesn’t necessarily teach better.
The proposed KD decomposition attributes the success of a better teacher to three factors, including correct guidance , smooth regularization , and class discriminability .
Through theoretical analysis, over-conﬁdent teachers could not release their potential abilities of the class discriminability under traditional temperature scaling.
As a simple yet effective solution, we propose Asymmetric
Temperature Scaling (ATS) to enhance the derived variance of larger teachers, making their distillation labels more discriminative when teaching students.
Extensive experimental results verify the superiorities of our proposed methods.
Broader Impact
We focus on the variance of wrong class probabilities to analyze why larger teachers cannot teach well and hope that our research could bring a new perspective to the KD ﬁeld.
Our work has no potential negative societal impacts.
Acknowledgements
This work is partially supported by National Natural Science Foundation of China (Grant No.
61921006, 41901270), and Natural Science Foundation of Jiangsu Province (Grant No. BK20190296).
Thanks to Huawei Noah’s Ark Lab NetMIND Research Team.
We gratefully acknowledge the support of Mindspore used for this reasearch.
Professor De-Chuan Zhan is the corresponding author.
A