Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Knowledge Distillation (KD) aims at transferring the knowledge of a wellperformed neural network (the teacher ) to a weaker one (the student ).
A peculiarphenomenon is that a more accurate model doesn’t necessarily teach better, andtemperature adjustment can neither alleviate the mismatched capacity.
To explainthis, we decompose the efﬁcacy of KD into three parts: correct guidance , smoothregularization , and class discriminability .
The last term describes the distinctnessof wrong class probabilities that the teacher provides in KD.
Complex teacherstend to be over-conﬁdent and traditional temperature scaling limits the efﬁcacy ofclass discriminability , resulting in less discriminative wrong class probabilities.
Therefore, we propose Asymmetric Temperature Scaling (ATS) , which separatelyapplies a higher/lower temperature to the correct/wrong class.
ATS enlarges thevariance of wrong class probabilities in the teacher’s label and makes the studentsgrasp the absolute afﬁnities of wrong classes to the target class as discriminative aspossible.
Both theoretical analysis and extensive experimental results demonstratethe effectiveness of ATS.
Introduction
Although large-scale deep neural networks have achieved overwhelming successes in many realworld applications [20, 10, 58], the vast capacity hinders them from being deployed on portabledevices with limited computation and storage resources [3].
Some efﬁcient architectures, e g.,
MobileNets [13, 35] and ShufﬂeNets [57, 27] have been proposed for lightweight deployment whiletheir performances are usually constrained.
Fortunately, knowledge distillation (KD) [44, 12] couldtransfer the knowledge of a more complex and well-performed network (i e., the teacher ) to them.
The original KD [12] forces the student to mimic the teacher’s behavior via minimizing the KullbackLeibler (KL) divergence between their output probabilities.
Recent studies generalize the logits-levelknowledge to various types of expertise, including the intermediate features [34, 55, 15, 11, 31, 1], theneuron or sample relationships [32, 42, 50, 25, 52, 43, 48], the parameters [24, 21], etc. Additionally,various distillation schemes have emerged, such as deep mutual learning [59], online distillation [2],self-distillation [56, 18], etc.
Although some empirical studies and theoretical analysis explain why
KD works [38, 16, 28, 33, 53, 61, 5, 26, 14], some miraculous phenomena in KD are still not wellexplained, e.g., more accurate neural networks don’t necessarily teach better [29, 5, 9, 22, 49]. Several fundamental questions have not been answered: What’s the real reason that more complexteachers can’t teach well?
Is it really impossible to make larger teachers teach better through simpleoperations, such as temperature scaling?
To answer the ﬁrst question, we focus on analyzing the distinctness of wrong class probabilitiesthat a teacher provides in KD.
We decompose the teacher’s label into three parts (see Sect.
4.1): (I)
Correct Guidance : the correct class’s probability; (II) Smooth Regularization : the average probability
[Figure]
of KD
[Figure]
of wrong classes; (III) Class Discriminability : the variance of wrong class probabilities (deﬁned asderived variance ).
The commonly utilized temperature scaling could control the efﬁcacy of thesethree terms (the left of Fig. 1).
More complex teachers are over-conﬁdent and assign a larger score forthe correct class or less varied scores for the wrong classes.
If we use a uniform temperature to scaletheir logits, the class discriminability of the larger teacher is less effective (theoretically analyzed in
Sect. 4.2), i.e., the probabilities of wrong classes are less distinct (the right of Fig. 1).
As to the second question, we focus on enlarging the variance of wrong class probabilities (i.e.,derived variance ) that a teacher provides to make the distillation process more discriminative.
Tospeciﬁcally enhance the distinctness of wrong class probabilities, we separately apply a higher/lowertemperature to the correct/wrong class’s logit instead of a uniform temperature (see Sect. 4.3).
Wename our method Asymmetric Temperature Scaling (ATS) , and abundant experimental studies verifythat utilizing this simple operation could make larger teachers teach well again.
Related Works
KD with Larger Teacher : Although KD has been a general technique for knowledge transfer invarious applications [12, 59, 40, 23], could any student learn from any teacher?
[5] ﬁrst studies the
KD’s dependence on student and teacher architectures.
They ﬁnd that larger models do not oftenmake better teachers and propose the early-stopped teacher as a solution.
[29] introduces a multi-step
KD process, employing an intermediate-sized network (the teacher assistant ) to bridge the capacitygap.
[49] formulates KD as a multi-task learning problem with several knowledge transfer losses.
The transfer loss will be utilized only when its gradient direction is consistent with the cross-entropyloss. [9, 22] deﬁne the knowledge gap as residual , which is utilized to teach the residual student , andthen they take the ensemble of the student and residual student for inference.
These works attributethe worse teaching performance to capacity mismatch, i.e., weaker students can’t completely mimicthe excellent teachers.
However, they don’t explain this peculiar phenomenon in detail.
Understanding of KD :
Quite a few works focus on understanding the advantages of KD from aprincipled perspective.
[26] uniﬁes KD and privileged information into generalized distillation .[33, 16] utilize gradient ﬂow and neural tangent kernel to analyze the convergence property of KDunder deep linear networks and inﬁnitely wide networks.
[4] explains KD via quantifying the taskrelevant and task-irrelevant visual concepts.
[6] casts KD as a semiparametric inference problem andproposes corresponding enhancements.
Our work is more related to KD decompositions.
[8] treatsthe teacher’s correct/wrong outputs differently, respectively explaining them as importance weightingand class similarities.
[38] further decomposes the “dark knowledge" into universal knowledge,domain knowledge, and gradient rescaling.
[28] establishes a bias-variance tradeoff to quantify thedivergence of a teacher with the Bayes teacher .
[61] utilizes bias-variance decomposition to analyze
KD and discovers regularization samples that could increase bias and decrease variance.
Our work isalso related to Label smoothing (LS).
[53] points out that the regularization effect in KD is similar to
LS.
[30] ﬁnds that training a teacher with LS could degrade its teaching quality, and attributes thisto the fact that LS erases relative information between teacher logits.
Recently, [36] further studiesthis problem and proposes a metric to measure the degree of erased information quantitatively.
Ourwork also decomposes KD into several effects to study why more complex teachers can’t teach well.
Detailed relatedness to these works is presented in Sect.
Background
We consider a C -class classiﬁcation problem with Y =
[ C ] = { 1 , 2 , · · · , C } .
Given a neural network 79 and a sample pair ( x , y ) , we could obtain the “logits" as f ( x ) ∈ R C .
We denote the softmax functionwith temperature τ as SF ( · ; τ ) , i.e., p c ( τ ) =
exp ( f c ( x ) /τ ) /Z
( τ ) and Z ( τ ) =
(cid:80) Cj =1 exp ( f j ( x ) /τ ) .
Later, we may omit the dependence on x and τ if without any ambiguity.
We use f y and p y to denotethe correct class ’s logit and probability, while we use g and q to represent the vector of wrong classes ’logits and probabilities, i.e., g =
[ f c ] c (cid:54) = y and q =
[ p c ] c (cid:54) = y .
Although there are many types of KD as introduced in Sect. 1, we focus on the most standard
KD proposed in [12], i.e., transferring knowledge from a larger teacher to a smaller student viaminimizing the KL divergence of softened probabilities.
Usually, the loss function is a combinationof cross-entropy loss and distillation loss:
[Equation]
where the upper script “T"/“S" denotes “Teacher"/“Student" respectively.
Suppose we have two teachers, denoted as T large and T small , and the larger teacher performs better onboth training and test data.
If we use them to teach the same student S , we could ﬁnd that the student’sperformance is worse when mimicking the larger teacher’s outputs.
Adjusting the temperature couldneither make the larger teacher teach well.
The details of this phenomenon could be found in [5, 29]and Fig.
Obviously, p T large could differ a lot from p T small , which is the only difference in the lossfunction when teaching the student.
Hence, we focus on analyzing what probability distributions aretended to be provided by teachers with different capacities .
Proposed Methods
This section ﬁrst decomposes KD into three parts and deﬁnes several quantitative metrics.
Then, wepresent theoretical analysis to demonstrate why larger networks can’t teach well.
Finally, we proposea more appropriate temperature scaling approach as an alternative.
4.1 KD Decomposition
We can decompose the KD loss 1 in Eq. 1 into three terms:
[Equation]
Correct Guidance
[Equation]
Smooth
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(I)
Correct Guidance this term guarantees correctness during teaching.
The decomposition in [8]also contains this term, which is explained as importance weighting.
This term works similarly to thecross-entropy loss, which could be dealt with separately when applying temperature scaling
(II) Smooth Regularization : some previous works [53, 60, ] attribute the success of KD to the
efﬁcacy of regularization and study its relation to label smoothing (LS).
The combination of this term
with correct guidance works similarly to LS.
Notably, E j (cid:54) = y [ p Tj ( τ )] differs across samples, implyingthat the strength of smoothing is instance-speciﬁc, which is similar to the analysis in [60].(III) Class Discriminability : this term tells the student the afﬁnity of wrong classes to the correctclass.
Transferring the knowledge of class similarities to students has been the mainstream guess ofthe “dark knowledge" in KD [12, 36].
Ideally, a good teacher should be as discriminating as possiblein telling students which classes are more related to the correct class.
Illustrations of the decomposition are presented in the left of Fig.
Obviously, an appropriatetemperature should simultaneously contain the efﬁcacy of the three terms, e.g., the shaded row in
Fig.
A too high or too low temperature could lead to smaller class discriminability , making theguidance less different among wrong classes, which weakens the distillation performance in practical.
Among these three terms, we advocate that class discriminability is more fundamental in KD andpresent more discussions in Appendix A (veriﬁed in Fig.
2 and Fig. 3).
To measure these three terms quantitatively, we use the target class probability (i.e., p y ), theaverage of wrong class probabilities (i.e., E [ q ] = 1 C − 1 (cid:80)
j (cid:54) =
y p j ), and the variance of wrong classprobabilities (i.e., V [ q ] = 1 C − 1 (cid:80)
j (cid:54) =
y ( p j − E [ q ]) 2 ) as estimators.
In some cases, we use thestandard deviation as an estimator for the third term, i.e., σ [ q ] = V 1 / 2 [ q ] .
Because the latter twoterms are calculated after applying softmax to the complete logit vector, we deﬁne them as Derived Average (DA) and Derived Variance (DV) , respectively.
In experiments, we calculate these metricsfor all training samples and sometimes report the average or standard deviation across these samples.
4.2 Theoretical Analysis
We analyze the mean and variance of the softened probability vector, i.e., the teacher’s label p T ( τ )used in KD.
We do not analyze the commonly used entropy because we will especially focus on thewrong classes, where the mean and variance statistics are more intuitive.
In the following, we deferthe proofs of Lemma 4.1 and Proposition 4.3, 4.4 to Appendix B.
Lemma 4.1 (Variance of Softened Probabilities) .
Given a logit vector f ∈ R C and the softenedprobability vector via softmax function, i.e., p = SF ( f ; τ ) , τ ∈ (0 , ∞ ) , V [ p ] monotonically decreasesas τ increases.
Assumption 4.2.
The target logit is higher than other classes’ logits, i.e., f y ≥ f c , ∀ c (cid:54) = y .
Assumption 4.2 is rational because well-performed teachers could almost achieve a higher accuracy(e.g., >95%) on the training set, and most training samples meet this requirement.
Proposition 4.3.
Under Assumption 4.2, p y monotonically decreases as τ increases, and E [ q ]monotonically increases as τ increases.
As τ → ∞ , E [ q ] → 1 /C .
Proposition 4.3 implies that increasing temperature could lead to a higher derived average (empiricallysee Fig. 7) and strengthen the smooth regularization term in Eq. 2.
Before we analyze the class discriminability term, we deﬁne ˜
q ( τ ) as the result of applying softmaxonly to the wrong logits with temperature τ , i.e., ˜ q
( τ ) =
SF ( g ; τ ) .
For c (cid:54) = y , its correspondingindex in q is c (cid:48) , we have
[Equation]
Notably, ˜ q differs from q a lot.
Speciﬁcally, the former satisﬁes (cid:80) c (cid:48) ˜ q c (cid:48)
= 1 , while the summationof the latter is (cid:80) c (cid:54) = y p c = 1 − p y .
The former does not depend on the correct class’s logit while thelatter does.
We name V [˜ q ] Inherent Variance (IV) because it only depends on wrong classes’ logits.
Proposition 4.4 (Derived Variance vs. Inherent Variance) .
The derived variance is determined bythe square of derived average and the inherent variance via:
[Equation]
With τ increases, E [ q ] increases (Proposition 4.3) while V [˜ q ] decreases (Lemma 4.1), and hence, itis not so easy to judge the speciﬁc monotonicity of V [ q ] w.r.t.
τ .
Empirically, we observe that the derived variance ﬁrst increases and then decreases (see Fig. 7), which conforms to the change of theclass discriminability as illustrated in Fig.
We could use Proposition 4.4 to clearly analyze why larger teacher networks can’t teach well.
Beforethis, we present another two properties and a corollary without detailed proof.
Remark 4.5 .
Fixing g and τ , a higher target logit f
y leads to a higher p y , i.e., a smaller derivedaverage E [ q ] .
Remark 4.6 .
Fixing τ , less varied wrong logits g leads to less varied ˜ q , i.e., a smaller inherentvariance V [˜ q ] .
Corollary 4.7.
Suppose we have two teachers T 1 and T 2 , and their logit vectors for a same sampleare f T 1 and f T 2 .
• If f T 1 y ≥ f T 2 y while g T 1 and g T 2 are nearly the same, then p T 1 y ≥ p T 2 y (Remark 4.5) whileV [˜ q T 1 ] ≈ V [˜ q T 2 ] .
Hence, V [ q T 1 ] ≤ V [ q T 2 ] .•
If f T 1 y ≈ f T 2 y while V [ g T 1 ] ≤ V [ g T 2 ] , then p T 1 y ≈ p
T
y while V [˜ q T 1 ] ≤ V [˜ q T 2 ] (Remark 4.6).
Hence, V [ q T 1 ] ≤ V [ q T 2 ] .
This corollary explains why a larger teacher can’t teach better.
Because the larger teacher tends to beover-conﬁdent, the target logit f
y may be larger or the variance of wrong logits V [ g ] may be smaller.
These are illustrated in Fig.
1 and empirically veriﬁed in Fig.
Then the derived variance V [ q ] maybe smaller, limiting the efﬁcacy of class discriminability in Eq.
Empirical results are in Fig.
Notably, we focus on analyzing the variance of wrong class probabilities instead of all classes.
Maximizing the variance of all classes’ probabilities does not mean maximizing the variance ofwrong classes’.
For example, although a very low temperature can maximize the variance of allclasses’ probabilities, the generated teacher’s label is one-hot that shows no distinctness betweenwrong classes.
In other words, the effectiveness of KD should be more related to the distinctnessbetween wrong classes rather than all classes.
However, traditional temperature scaling applies auniform temperature for all classes, which cannot separately handle the wrong classes.
4.3 Asymmetric Temperature Scaling
We conclude the above analysis: if a larger teacher makes an over-conﬁdent prediction, the wrongclass probabilities it provides could be not discriminative enough.
Utilizing a uniform temperaturecould not enlarge the derived variance as much as possible with the interference of the target class’slogit (see the middle of Fig. 7).
Thanks to the decomposition in Eq.
2, the correct guidance termworks similarly to the cross-entropy loss and allows us to deal with it separately.
Hence, we proposea novel temperature scaling approach:
[Equation]
where we take τ 1 > τ 2 > 0 .
This approach is named Asymmetric Temperature Scaling (ATS) becausewe apply different temperatures to the logits of correct and wrong classes.
According to Eq. 4, ATScould bring such beneﬁts when the teacher is over-conﬁdent:
• If the teacher outputs a larger logit f y for the correct class, a relatively larger τ 1 could
decrease it to a reasonable magnitude, i
e., decreasing p y and increasing E [ q ] , and ﬁnally
increasing the derived variance V [ q ] ;
• If the teacher outputs less varied logits g for wrong classes, a relatively smaller temperature
τ 2 could make them more diverse, i.e., increasing V [ ˜q ] , ﬁnally increasing the derived
variance V [ q ] .
ATS is more ﬂexible in enlarging the derived variance (see the right of Fig. 7), i.e., it could generatemore discriminative distillation guidance during teaching.
Take the demo in Fig.
1 as an example,the smaller and larger teacher captures the same relative class afﬁnities, e.g., they both know thatthe fourth/ﬁfth class (shaded cells) is the most/least relevant to the target class.
However, with auniform temperature 4 . 0 , the smaller teacher provides probabilities (0 . 073 , 0 . 060) for these twoclasses, while over-conﬁdent larger teachers provide (0 . 040 , 0 . 033) or (0 . 074 , 0 . 067) .
Clearly, the
[Figure]
ResNet → SFV
[Figure] [Figure]
absolute afﬁnities of the larger teachers are not so discriminative as the smaller teacher’s.
Utilizing ATS, we could respectively apply ( τ 1 = 4 .
67 , τ 2 = 4 . 0) or ( τ 1 = 4 . 0 , τ 2 = 2 . 0) to the over-conﬁdentteacher’s logits, generating the same probability vector as the smaller teacher’s
Experiments
We use CIFAR-10/CIFAR-100 [19], TinyImageNet [41], CUB [45], Stanford Dogs [17], and
Google Speech Commands [46] as the datasets.
For teacher networks, we use different versionsof ResNet [10], WideResNet [54], ResNeXt [47].
For student networks, we use VGG [37], ShufﬂeNetV1/V2 [57, 27], AlexNet [20], MobileNetV2 [35], and DSCNN [58].
Details are in Appendix C.
5.1 Observations
Class discriminability matters a lot in KD and correlates with the KD improvement.
To showthe importance of class discriminability in KD, we omit the distinctness of wrong class probabilitiesduring distillation.
Speciﬁcally, we only keep the ﬁrst two terms in Eq. 2, which works similarly tothe instance-speciﬁc label smoothing (abbreviated as “ILS").
Fig.
2 shows students’ performances on
CIFAR-100.
Without the third term in Eq. 2, both small teachers (“ST") and larger teachers (“LT")teach worse signiﬁcantly.
Then, we investigate the correlations of KD performance improvement(i.e.
, the test accuracy change ratio with KD w.r.t.
without KD) with smooth regularization and classdiscriminability under 270 pairs of “(teacher, student, temperature)".
Details are in Appendix C.4.
Fig.
3 plots the scatters, where dots show pairs whose improvement is higher than 2% .
Clearly,teachers with a larger derived variance tend to guide better.
These observations show the rationalityof the proposed KD decomposition and imply that enhancing derived variance is beneﬁcial .
Larger teachers provide a larger target logit or less varied wrong logits.
We ﬁrst compare thelogit distributions provided by larger and smaller teachers on the training set.
Fig.
4 plots thehistograms (200 bins) of the target logit (i.e., f y ) and the standard deviation of wrong logits (i.e.,σ [ g ] ).
The left and right, respectively, show the results on CIFAR-100 and CIFAR-10.
Clearly, theﬁrst column shows that ResNet110 tends to generate a larger target logit (i.e., E x
[ f y ]
≈ 15 . 0 )
than
ResNet14 (i.e., E x
[ f y ] ≈ 10 . 0 ).
On CIFAR-10, the smallest f y given by WRN28-8 is larger than
WRN28-1.
Furthermore, WRN28-8 gives smaller variance (the fourth column), i.e., smaller inherentvariance .
These reveal speciﬁc manifestations of complex networks’ over-conﬁdence .
Larger and smaller teachers have similar inherent variance while different derived varianceunder traditional temperature scaling.
We use τ = 1 . 0 to soften the complete logits and only thewrong class logits , respectively, and then show the mean and standard deviation of derived varianceand inherent variance across training samples in Fig.
Although the larger models’ inherent varianceis smaller, the difference between RNX29-4-4d and RNX29-64-4d is only up to 1 .
3 × .
However,the derived variance differs a lot, where the smaller teacher’s variance is approximately 9 .
7 × asthe larger teacher’s.
These observations imply that traditional temperature scaling could seriouslydecrease the derived variance of larger teachers though they have appreciable inherent variance .
Complex teachers know approximately the same as smaller teachers on relative class afﬁnities.
Only if the relative magnitudes of the wrong class probabilities are correct, it is valid to enlargethe discrimination between them.
Otherwise, if a teacher himself misunderstands the knowledge’sprinciples, it will be counterproductive to reinforce this knowledge to students.
Given two teachers
[Figure]
CIFAR
[Figure]
f
[Figure]
CIFAR
[Figure]
[ g ]
[Figure] [Figure]
[Figure] [Figure]
CIFAR
[Figure]
f
[Figure]
CIFAR
[Figure]
g
[Figure]
[Figure]
[Figure]
[Figure]
DV under TS is limited while ATS enlarges it
T 1 and T 2 , we ﬁrst calculate the Spearman correlation between p T 1 and p T 2 .
Second, the set overlapbetween the top-5 predictions is calculated, i.e., |C T 1 ∩ C T 2 | / |C T 1 ∪ C T 2 | .
C denotes the set of top-5 240 predicted classes.
Third, we calculate Kendall’s τ between p T 1 and p T 2 , which directly shows therank correlation of two teachers.
These metrics only depend on classes’ relative magnitudes.
Theresults are in Fig.
Excitingly, these metrics among teachers with different capacities do not vary alot, and the Spearman correlations are almost all larger than 0 . 85 .
According to the interpretationof Kendall’s τ [7, 51], if the smaller teacher predicts that class i is more related to the target classthan that of class j , then the larger teacher has a probability of (0 .
75 + 1) / 2 = 0 .
875 to give thesame relative afﬁnity.
As a comparison, we also calculate the distillation distance deﬁned in [14],which utilizes L1 distance and depends on the absolute magnitudes of probabilities.
Using this metric,the distance increases quickly as the capacity gap increases.
These observations demonstrate thatteachers know approximately the same about relative class afﬁnities while their absolute values differsigniﬁcantly under traditional temperature scaling .
The proposed ATS could enlarge the derived variance of larger teachers.
The above analysisveriﬁes that the over-conﬁdent teachers experience lower derived variance under traditional temperature scaling although they grasp the relative class afﬁnities well.
For an intuitive visualization, weplot the probabilities for a randomly sampled instance whose correct class is y = 1 from CIFAR-100.
The top two rows in Fig.
8 show the results of RNX29-4-4d and RNX29-64-4d under traditionaltemperature scaling (TS), where the latter really experiences a smaller derived variance .
Using ATScould enhance the derived variance as shown at the bottom row (the bars are more jagged).
Then,we study the change of derived average (DA) and derived variance (DV) as temperature increases.
Given a τ , we obtain the DA and DV for all training samples via softmax and then calculate theaverage.
For ATS, we use τ 1 = 1 .
25 τ and τ 2 = 0 .
75 τ .
The curves are shown in Fig.
According to
Proposition 4.3, E [ q ] increases as τ increases, which enhances the efﬁcacy of smooth regularization .
Notably, this term changes nearly the same between teachers with various capacities.
However, thederived variance V [ q ] differs a lot.
Empirically, V [ q ] ﬁrst increases and then decreases, and themaximal of the larger teacher’s DV is smaller, which veriﬁes the Corollary 4.7.
Because derivedvariance corresponds to the efﬁcacy of class discriminability , this shows why larger teachers can’tteach well.
Using the proposed ATS could enhance the derived variance , which equivalently improvesthe efﬁcacy of class discriminability in KD.
We conclude that traditional temperature scaling leads to
[Figure] [Figure]
V GG
ResNet
KDTSKDATS 0 .
S hu ﬄ e N e t V
KDTSKDATS
R E S 14 R E S 20 R E S 32 R E S 44 R E S 56 R E S 110 0 .0 .
M o b il e N e t V
KDTSKDATS 0 .
WideResNet
KDTSKDATS 0 .
KDTSKDATS
W R N 28 - 1 W R N 28 - 2 W R N 28 - 3 W R N 28 - 4 W R N 28 - 6 W R N 28 - 8 0 .0 .
KDTSKDATS 0 .
ResNeXt KDTSKDATS 0 .
KDTSKDATS
R N X 29 - 4 - 4 d R N X 29 - 8 - 4 d R N X 29 - 16 - 4 d R N X 29 - 24 - 4 d R N X 29 - 32 - 4 d R N X 29 - 64 - 4 d 0 .0 .
KDTSKDATS
[Table]
[Table] [Table]
[Table] [Table]
distillation labels with less discriminative information among wrong class probabilities; our proposed
ATS could enhance the discrimination among wrong classes and beneﬁt the distillation process .
5.2 Performances
ATS makes larger teachers teach well again.
Previous studies ﬁnd that more accurate teachers can’t necessarily teach well [5, 29].
As shown in Fig.
9, although we tune temperatures in{ 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 12 . 0 , 16 . 0 } , larger teachers still teach worse under traditional temperature scaling(the solid curves).
However, using ATS (the dashed curves) could make larger teachers teach well orbetter again.
The details are in Appendix C.4.
ATS surpasses previous methods with advanced techniques.
We compare with SOTA methodsand list the results on CIFAR-100, TinyImageNet, CUB, and Dogs in Tab. 1 and Tab. 2.
Details ofcompared methods are in Appendix C.3.
The last column of these tables shows the average performance of corresponding rows.
The larger teacher could slightly improve the students’ performancesvia traditional KD, i.e., 68 .
24% → 69 . 21% and 57 .
94 → 58 . 89% .
However, using a smaller teacher(the row of “ST-KD") could obtain about 2% improvement on average, i.e., 68 .
24% → 70 .
32% and57 .
94% → 59 .
This again veriﬁes that larger teachers really teach worse on various datasets.
Taking advantage of the early-stopped teacher (ESKD), the teacher assistant (TAKD), and the student customized teacher (SCKD) only improves the larger teacher slightly, e.g., 69 .
21% → 69 .
86%and 58 .
89% → 59 .
These results do not even surpass the small teacher.
ResKD improvesthe students’ performances a lot via introducing the residual student and taking the two students’ensemble, which surpasses the ensemble performances of two separately trained students underdifferent initializations (the “Ens" row).
For a fair comparison, we also test the performances of
[Table]
TinyImageNet:WRN50-2 → SFV      λ =0.
λ
=0.
λ
=0.
λ
=0.
λ
=0.
T e s t
A cc u r a c y
WRN28-8 → MV
KD TS: τ =2.0 KD ATS:
τ 1 =3.0, τ 2 =2.
our methods via repeating the “KD+ATS" two times and making predictions via the ensemble.
Theresults are almost 1% higher than ResKD.
Results on CIFAR-10 and speech data are in Appendix D.
Ablation Studies One might argue that the validity of ATS is due to the tuning from a larger hyperparameter space.
We vary τ 1 from 1.0 to 6.0, τ 2 from 1.0 to 5.0, and record the performances in
Fig.
Obviously, setting τ 1 > τ 2 could be better, and especially, we recommend the setting ofτ 2 ∈
[ τ 1 − 2 , τ 1 − 1] .
Although we introduce one more hyper-parameter, ATS is simple to implement 295 and surpasses SOTA methods that take advanced techniques.
We achieve the goal of only utilizingsimple operations to make larger teachers teach well again.
Then, we study the trade off of the KDloss and CE loss under different λ .
The results of “WRN28-8 → MV2" on CIFAR-100 (Fig. 11)veriﬁes that ATS could improve the performances under various λ .
We also compare ATS with othertypes of KD under various scenes, and the results are in Appendix D.3.
Conclusion
We study the miraculous phenomenon in KD that a more accurate model doesn’t necessarily teachbetter.
The proposed KD decomposition attributes the success of a better teacher to three factors,including correct guidance , smooth regularization , and class discriminability .
Through theoreticalanalysis, over-conﬁdent teachers could not release their potential abilities of the class discriminabilityunder traditional temperature scaling.
As a simple yet effective solution, we propose Asymmetric
Temperature Scaling (ATS) to enhance the derived variance of larger teachers, making their distillation labels more discriminative when teaching students.
Extensive experimental results verify thesuperiorities of our proposed methods.
Broader Impact
We focus on the variance of wrong class probabilities to analyze why larger teachers cannot teachwell and hope that our research could bring a new perspective to the KD ﬁeld.
Our work has nopotential negative societal impacts.