Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Knowledge Distillation (KD) aims at transferring the knowledge of a wellperformed neural network (the teacher ) to a weaker one (the student ).
A peculiarphenomenon is that a more accurate model doesn’t necessarily teach better, andtemperature adjustment can neither alleviate the mismatched capacity.
To explainthis, we decompose the efﬁcacy of KD into three parts: correct guidance , smoothregularization , and class discriminability .
The last term describes the distinctnessof wrong class probabilities that the teacher provides in KD.
Complex teacherstend to be over-conﬁdent and traditional temperature scaling limits the efﬁcacy ofclass discriminability , resulting in less discriminative wrong class probabilities.
Therefore, we propose Asymmetric Temperature Scaling (ATS) , which separatelyapplies a higher/lower temperature to the correct/wrong class.
ATS enlarges thevariance of wrong class probabilities in the teacher’s label and makes the studentsgrasp the absolute afﬁnities of wrong classes to the target class as discriminative aspossible.
Both theoretical analysis and extensive experimental results demonstratethe effectiveness of ATS.
Introduction
Although large-scale deep neural networks have achieved overwhelming successes in many realworld applications [22, 11, 60], the vast capacity hinders them from being deployed on portabledevices with limited computation and storage resources [3].
Some efﬁcient architectures, e g.,
MobileNets [14, 37] and ShufﬂeNets [59, 29] have been proposed for lightweight deployment whiletheir performances are usually constrained.
Fortunately, knowledge distillation (KD) [46, 13] couldtransfer the knowledge of a more complex and well-performed network (i e., the teacher ) to them.
The original KD [13] forces the student to mimic the teacher’s behavior via minimizing the KullbackLeibler (KL) divergence between their output probabilities.
Recent studies generalize KD to varioustypes of knowledge [36, 57, 17, 12, 33, 1, 34, 44, 52, 27, 54, 45, 50, 26, 23] or various distillationschemes [61, 2, 58, 20]. An intuitive sense after the proposal of KD [13] is that larger teachers couldteach students better because their accuracies are higher.
A recent work [6] ﬁrst points out that theteacher accuracy is a poor predictor of the student’s performance.
That is, more accurate neuralnetworks don’t necessarily teach better. Until now, this phenomenon is still counter-intuitive [51],surprising [31], and unexplored [24]. Different from some existing empirical studies and theoreticalanalysis [40, 18, 30, 35, 55, 63, 6, 28, 15], we investigate the miraculous phenomenon in detail andaim to answer the following questions: What’s the real reason that more complex teachers can’t teachwell?
Is it really impossible to make larger teachers teach better through simple operations, such astemperature scaling?
To answer the ﬁrst question, we focus on analyzing the distinctness of wrong class probabilitiesthat a teacher provides in KD.
We decompose the teacher’s label into three parts (see Sect.
4.1): (I)
[Figure]
of KD
[Figure]
Correct Guidance : the correct class’s probability; (II) Smooth Regularization : the average probabilityof wrong classes; (III) Class Discriminability : the variance of wrong class probabilities (deﬁned asderived variance ).
The commonly utilized temperature scaling could control the efﬁcacy of thesethree terms (the left of Fig. 1).
More complex teachers are over-conﬁdent and assign a larger score forthe correct class or less varied scores for the wrong classes.
If we use a uniform temperature to scaletheir logits, the class discriminability of the larger teacher is less effective (theoretically analyzed in
Sect. 4.2), i.e., the probabilities of wrong classes are less distinct (the right of Fig. 1).
As to the second question, we focus on enlarging the variance of wrong class probabilities (i.e.,derived variance ) that a teacher provides to make the distillation process more discriminative.
Tospeciﬁcally enhance the distinctness of wrong class probabilities, we separately apply a higher/lowertemperature to the correct/wrong class’s logit instead of a uniform temperature (see Sect. 4.3).
Wename our method Asymmetric Temperature Scaling (ATS) , and abundant experimental studies verifythat utilizing this simple operation could make larger teachers teach well again.
Related Works
KD with Larger Teacher : Although KD has been a general technique for knowledge transfer invarious applications [13, 61, 42, 25], could any student learn from any teacher?
[6] ﬁrst studies the
KD’s dependence on student and teacher architectures.
They ﬁnd that larger models do not oftenmake better teachers and propose the early-stopped teacher as a solution.
[31] introduces a multi-step
KD process, employing an intermediate-sized network (the teacher assistant ) to bridge the capacitygap.
[51] formulates KD as a multi-task learning problem with several knowledge transfer losses.
Thetransfer loss will be utilized only when its gradient direction is consistent with the cross-entropy loss.[10, 24] deﬁne the knowledge gap as residual , which is utilized to teach the residual student , andthen they take the ensemble of the student and residual student for inference.
These works attributethe worse teaching performance to capacity mismatch, i.e., weaker students can’t completely mimicthe excellent teachers.
However, they don’t explain this peculiar phenomenon in detail.
Understanding of KD :
Quite a few works focus on understanding the advantages of KD from aprincipled perspective.
[28] uniﬁes KD and privileged information into generalized distillation .[35, 18] utilize gradient ﬂow and neural tangent kernel to analyze the convergence property of KDunder deep linear networks and inﬁnitely wide networks.
[5] explains KD via quantifying the taskrelevant and task-irrelevant visual concepts.
[7] casts KD as a semiparametric inference problem andproposes corresponding enhancements.
Our work is more related to KD decompositions.
[9] treatsthe teacher’s correct/wrong outputs differently, respectively explaining them as importance weightingand class similarities.
[40] further decomposes the “dark knowledge" into universal knowledge,domain knowledge, and gradient rescaling.
[30] establishes a bias-variance tradeoff to quantify the divergence of a teacher with the Bayes teacher .
[63] utilizes bias-variance decomposition to analyze
KD and discovers regularization samples that could increase bias and decrease variance.
Our work isalso related to Label smoothing (LS).
[55] points out that the regularization effect in KD is similar to
LS.
[32] ﬁnds that training a teacher with LS could degrade its teaching quality, and attributes thisto the fact that LS erases relative information between teacher logits.
Recently, [38] further studiesthis problem and proposes a metric to measure the degree of erased information quantitatively.
Ourwork also decomposes KD into several effects to study why more complex teachers can’t teach well.
Detailed relatedness to these works is presented in Sect.
Background
We consider a C -class classiﬁcation problem with Y =
[ C ] = { 1 , 2 , ⋯ ,C } .
Given a neural networkand a sample pair ( x ,y ) , we could obtain the “logits" as f ( x ) ∈ R C .
We denote the softmax functionwith temperature τ as SF (⋅ ; τ ) , i.e., p c ( τ ) =
exp ( f c ( x )/ τ )/
Z ( τ ) and Z ( τ ) =
∑ Cj = 1 exp ( f j ( x )/ τ ) .p ( τ ) is the softened probability vector that a network outputs. c is the index of class.
Later, we mayomit the dependence on x and τ if without any ambiguity.
We use f y and p y to denote the correctclass ’s logit and probability, while we use g and q to represent the vector of wrong classes ’ logitsand probabilities, i.e., g =
[ f c ] c ≠ y and q =
[ p c ] c ≠ y .
The most standard KD [13] contains two stages of training.
The ﬁrst stage trains complex teachers,and then the second stage transfers the knowledge from teachers to a smaller student via minimizingthe KL divergence between softened probabilities.
Usually, the loss function during the second stage(i.e. , the student’s learning objective) is a combination of cross-entropy loss and distillation loss: [Equation] [Equation]
·„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„‚„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„¶ KD
[Equation]
where the upper script “T"/“S" denotes “Teacher"/“Student" respectively.
Commonly, a defaulttemperature of 1 is utilized for the CE loss, and the student could also take a temperature of 1 for the
KD loss, e.g., p Sc ( τ = 1 ) [13, 31, 45, 44].
Suppose we have two teachers, denoted as T large and T small , and the larger teacher performs better onboth training and test data.
If we use them to teach the same student S , we could ﬁnd that the student’sperformance is worse when mimicking the larger teacher’s outputs.
Adjusting the temperature couldneither make the larger teacher teach well.
The details of this phenomenon could be found in [6, 31]and Fig.
Obviously, p T large could differ a lot from p T small , which is the only difference in the lossfunction when teaching the student.
Hence, we focus on analyzing what probability distributions aretended to be provided by teachers with different capacities .
Proposed Methods
This section ﬁrst decomposes KD into three parts and deﬁnes several quantitative metrics.
Then, wepresent theoretical analysis to demonstrate why larger networks can’t teach well.
Finally, we proposea more appropriate temperature scaling approach as an alternative.
4.1 KD Decomposition
We omit the coefﬁcient of λτ 2 in Eq. 1, and deﬁne e ( q T ( τ )) =
[Equation]
1 ∑ Cj = 1 ,j ≠ y p Tj ( τ ) , where q T ( τ ) =[ p
Tc ( τ )] c ≠ y .
Then, we have the following decomposition:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
·„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„‚„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„¶ Class
[Table]
(I) Correct Guidance : this term guarantees correctness during teaching.
The decomposition in [9]
also contains this term, which is explained as importance weighting.
This term works similarly to the
cross-entropy loss, which could be dealt with separately when applying temperature scaling.
(II) Smooth Regularization : some previous works [55, 62, 63] attribute the success of KD to the
efﬁcacy of regularization and study its relation to label smoothing (LS).
The combination of this term
with correct guidance works similarly to LS.
Notably, e j ≠ y ( p Tj ( τ )) differs across samples, implying that the strength of smoothing is instance-speciﬁc, which is similar to the analysis in [ (III) Class Discriminability : this term tells the student the afﬁnity of wrong classes to the correct
class. Transferring the knowledge of class similarities to students has been the mainstream guess of the “dark knowledge" in KD [13, 38].
Ideally, a good teacher should be as discriminating as possible in telling students which classes are more related to the correct class.
Illustrations of the decomposition are presented in the left of Fig.
Obviously, an appropriatetemperature should simultaneously contain the efﬁcacy of the three terms, e.g., the shaded row in
Fig.
A too high or too low temperature could lead to smaller class discriminability , making theguidance less different among wrong classes, which weakens the distillation performance in practical.
Among these three terms, we advocate that class discriminability is more fundamental in KD andpresent more discussions in Appendix A (veriﬁed in Fig.
2 and Fig. 3).
To measure these three terms quantitatively, we use the target class probability (i.e., p y ), the averageof wrong class probabilities (i.e., e ( q )
=
1 C − 1 ∑ j ≠ y p j ), and the variance of wrong class probabilities(i.e.
, v ( q ) =
1 C − 1 ∑ j ≠ y ( p j − e ( q )) 2 ) as estimators.
e (⋅) and v (⋅) calculates the mean and varianceof the elements in a vector.
In some cases, we use the standard deviation as an estimator for thethird term, i.e., σ [ q ] = v 1 / 2 ( q ) .
Because the latter two terms are calculated after applying softmaxto the complete logit vector, we deﬁne them as Derived Average (DA) and Derived Variance (DV) ,respectively.
In experiments, we calculate these metrics for all training samples and sometimes reportthe average or standard deviation across these samples.
4.2 Theoretical Analysis
We analyze the mean and variance of the softened probability vector, i.e., the teacher’s label p T ( τ )used in KD.
We defer the proofs of Lemma 4.1 and Proposition 4.3, 4.4 to Appendix B.
Lemma 4.1 (Variance of Softened Probabilities) .
Given a logit vector f ∈ R C and the softenedprobability vector p = SF ( f ; τ ) ,τ ∈ ( 0 , ∞) , v ( p ) monotonically decreases as τ increases. As τ increases, p ( τ ) becomes more uniform, i.e., it’s entropy increases.
However, we especiallyfocus on the wrong classes, where the mean and variance are more intuitive to calculate and analyze.
Assumption 4.2.
The target logit is higher than other classes’ logits, i.e., f y ≥ f c , ∀ c ≠ y .
Assumption 4.2 is rational because well-performed teachers could almost achieve a higher accuracy(e.g., >95%) on the training set, and most training samples meet this requirement.
Proposition 4.3.
Under Assumption 4.2, p y monotonically decreases as τ increases, and e ( q )monotonically increases as τ increases.
As τ → ∞ , e ( q ) → 1 / C .
Proposition 4.3 implies that increasing temperature could lead to a higher derived average (empiricallysee Fig. 7) and strengthen the smooth regularization term in Eq. 2.
Before we analyze the class discriminability term, we deﬁne ˜
q ( τ ) as the result of applying softmaxonly to the wrong logits with temperature τ , i.e., ˜ q
( τ ) =
SF ( g ; τ ) .
For the element index c ′ of q , wehave
[Equation]
Notably, ˜ q differs from q a lot.
Speciﬁcally, the former satisﬁes ∑ c ′ ˜ q c ′ =
1 , while the summation ofthe latter is ∑ c ≠ y p c = 1 − p y .
The former does not depend on the correct class’s logit while the latterdoes.
We name v ( ˜ q ) Inherent Variance (IV) because it only depends on wrong classes’ logits.
Proposition 4.4 (Derived Variance vs. Inherent Variance) .
The derived variance is determined bythe square of derived average and the inherent variance via:
[Equation]
With τ increases, e ( q ) increases (Proposition 4.3) while v ( ˜ q ) decreases (Lemma 4.1), and hence, itis not so easy to judge the speciﬁc monotonicity of v ( q ) w.r.t.
τ .
Empirically, we observe that the derived variance ﬁrst increases and then decreases (see Fig. 7), which conforms to the change of theclass discriminability as illustrated in Fig.
We could use Proposition 4.4 to clearly analyze why larger teacher networks can’t teach well.
Beforethis, we present another two properties and a corollary without detailed proof.
Remark 4.5 .
Fixing g and τ , a higher target logit f
y leads to a higher p y , i.e., a smaller derivedaverage e ( q ) .
Remark 4.6 .
Fixing τ , less varied wrong logits g leads to less varied ˜ q , i.e., a smaller inherentvariance v ( ˜ q ) .
Corollary 4.7.
Suppose we have two teachers T 1 and T 2 , and their logit vectors for a same sampleare f T 1 and f T 2 .
• If f T 1 y ≥ f T 2 y while g T 1 and g T 2 are nearly the same, then p T 1 y ≥ p T 2 y (Remark 4.5) whilev ( ˜ q T 1 )
≈ v ( ˜ q T 2 ) .
Hence, v ( q T 1 ) ≤ v ( q T 2 ) .•
If f T 1 y ≈ f T 2 y while v ( g T 1 ) ≤ v ( g T 2 ) , then p T 1 y ≈ p
T
y while v ( ˜ q T 1 ) ≤ v ( ˜ q T 2 ) (Remark 4.6).
Hence, v ( q T 1 ) ≤ v ( q T 2 ) .
This corollary explains why a larger teacher can’t teach better.
Because the larger teacher tends to beover-conﬁdent, the target logit f
y may be larger or the variance of wrong logits v ( g ) may be smaller.
These are illustrated in Fig.
1 and empirically veriﬁed in Fig.
Then the derived variance v ( q ) maybe smaller, limiting the efﬁcacy of class discriminability in Eq. 2.
Empirical results are in Fig.
Notably, we focus on analyzing the variance of wrong class probabilities instead of all classes.
Maximizing the variance of all classes’ probabilities does not mean maximizing the variance ofwrong classes’.
For example, although a very low temperature can maximize the variance of allclasses’ probabilities, the generated teacher’s label is one-hot that shows no distinctness betweenwrong classes.
In other words, the effectiveness of KD should be more related to the distinctnessbetween wrong classes rather than all classes.
However, traditional temperature scaling applies auniform temperature for all classes, which cannot separately handle the wrong classes.
4.3 Asymmetric Temperature Scaling
We conclude the above analysis: if a larger teacher makes an over-conﬁdent prediction, the wrongclass probabilities it provides could be not discriminative enough.
Utilizing a uniform temperaturecould not enlarge the derived variance as much as possible with the interference of the target class’slogit (see the middle of Fig. 7).
Thanks to the decomposition in Eq.
2, the correct guidance termworks similarly to the cross-entropy loss and allows us to deal with it separately.
Hence, we proposea novel temperature scaling approach:
[Equation]
where we take τ 1 > τ 2 > 0 .
This approach is named Asymmetric Temperature Scaling (ATS) becausewe apply different temperatures to the logits of correct and wrong classes.
According to Eq. 4, ATScould bring such beneﬁts when the teacher is over-conﬁdent:
• If the teacher outputs a larger logit f y for the correct class, a relatively larger τ 1 could
decrease it to a reasonable magnitude, i
e., decreasing p y and increasing e ( q ) , and ﬁnally
increasing the derived variance v ( q ) ;
• If the teacher outputs less varied logits g for wrong classes, a relatively smaller temperature
τ 2 could make them more diverse, i.e., increasing v ( ˜q ) , ﬁnally increasing the derived
variance v ( q ) .
ATS is more ﬂexible in enlarging the derived variance (see the right of Fig. 7), i.e., it could generatemore discriminative distillation guidance during teaching.
Take the demo in Fig.
1 as an example, thesmaller and larger teacher captures the same relative class afﬁnities, e.g., they both know that thefourth/ﬁfth class (shaded cells) is the most/least relevant to the target class.
However, with a uniformtemperature 4 . 0 , the smaller teacher provides probabilities ( 0 . 073 , 0 . 060 ) for these two classes,while over-conﬁdent larger teachers provide ( 0 . 040 , 0 . 033 ) or ( 0 . 074 , 0 . 067 ) .
Clearly, the absoluteafﬁnities of the larger teachers are not so discriminative as the smaller teacher’s. Utilizing ATS, we
[Figure]
ResNet → SFV
[Figure] [Figure]
could respectively apply ( τ 1 = 4 .
67 ,τ 2 = 4 . 0 ) or ( τ 1 = 4 . 0 ,τ 2 = 2 . 0 ) to the over-conﬁdent teacher’slogits, generating the same probability vector as the smaller teacher’s.
ATS utilizes two temperaturesand creates the wiggle room to make the distribution over wrong classes more discriminative.
Experiments
We use CIFAR-10/CIFAR-100 [21], TinyImageNet [43], CUB [47], Stanford Dogs [19], and
Google Speech Commands [48] as the datasets.
For teacher networks, we use different versionsof ResNet [11], WideResNet [56], ResNeXt [49].
For student networks, we use VGG [39], ShufﬂeNetV1/V2 [59, 29], AlexNet [22], MobileNetV2 [37], and DSCNN [60].
Details are in Appendix C.
5.1 Observations
Class discriminability matters a lot in KD and correlates with the KD improvement.
To showthe importance of class discriminability in KD, we omit the distinctness of wrong class probabilitiesduring distillation.
Speciﬁcally, we only keep the ﬁrst two terms in Eq. 2, which works similarly tothe instance-speciﬁc label smoothing (abbreviated as “ILS").
Fig.
2 shows students’ performances on
CIFAR-100.
Without the third term in Eq. 2, both small teachers (“ST") and larger teachers (“LT")teach worse signiﬁcantly.
Then, we investigate the correlations of KD performance improvement(i.e.
, the test accuracy change ratio with KD w.r.t.
without KD) with smooth regularization and classdiscriminability under 270 pairs of “(teacher, student, temperature)".
Details are in Appendix C.4.
Fig.
3 plots the scatters, where dots show pairs whose improvement is higher than 2% .
Clearly,teachers with a larger derived variance tend to guide better.
These observations show the rationalityof the proposed KD decomposition and imply that enhancing derived variance is beneﬁcial .
Larger teachers provide a larger target logit or less varied wrong logits.
We ﬁrst compare thelogit distributions provided by larger and smaller teachers on the training set.
Fig.
4 plots thehistograms (200 bins) of the target logit (i.e., f y ) and the standard deviation of wrong logits (i.e.,σ [ g ] ).
The left and right, respectively, show the results on CIFAR-100 and CIFAR-10.
Clearly, theﬁrst column shows that ResNet110 tends to generate a larger target logit (i.e., E x
[ f y ]
≈ 15 . 0 )
than
ResNet14 (i.e., E x
[ f y ] ≈ 10 . 0 ).
On CIFAR-10, the smallest f y given by WRN28-8 is larger than
WRN28-1.
Furthermore, WRN28-8 gives smaller variance (the fourth column), i.e., smaller inherentvariance .
These reveal speciﬁc manifestations of complex networks’ over-conﬁdence .
Larger and smaller teachers have similar inherent variance while different derived varianceunder traditional temperature scaling.
We use τ = 1 . 0 to soften the complete logits and only thewrong class logits , respectively, and then show the mean and standard deviation of derived varianceand inherent variance across training samples in Fig.
Although the larger models’ inherent varianceis smaller, the difference between RNX29-4-4d and RNX29-64-4d is only up to 1 .
3 × .
However, thederived variance differs a lot, where the smaller teacher’s variance is approximately 9 .
7 × as the largerteacher’s.
These observations imply that traditional temperature scaling could seriously decrease thederived variance of larger teachers though they have appreciable inherent variance .
Complex teachers know approximately the same as smaller teachers on relative class afﬁnities.
Only if the relative magnitudes of the wrong class probabilities are correct, it is valid to enlargethe discrimination between them.
Otherwise, if a teacher himself misunderstands the knowledge’sprinciples, it will be counterproductive to reinforce this knowledge to students.
Given two teachers
[Figure]
CIFAR
[Figure]
f
[Figure]
CIFAR
[Figure]
[ g ]
[Figure] [Figure] [Figure]
[Figure] [Figure]
CIFAR
[Figure]
f [Figure] CIFAR
[Figure]
g
[Figure]
N
[Figure]
N
[Figure] [Figure] [Figure]
[Figure]
DV under TS is limited ATS enlarges it
T 1 and T 2 , we ﬁrst calculate the Spearman correlation between p T 1 and p T 2 .
Second, the set overlapbetween the top-5 predictions is calculated, i.e., ∣C T 1 ∩ C T 2 ∣/∣C T 1 ∪ C T 2 ∣ .
C denotes the set of top-5predicted classes.
Third, we calculate Kendall’s τ between p T 1 and p T 2 , which directly shows therank correlation of two teachers.
These metrics only depend on classes’ relative magnitudes.
Theresults are in Fig.
Excitingly, these metrics among teachers with different capacities do not vary alot, and the Spearman correlations are almost all larger than 0 . 85 .
According to the interpretationof Kendall’s τ [8, 53], if the smaller teacher predicts that class i is more related to the target classthan that of class j , then the larger teacher has a probability of ( 0 .
75 + 1 )/ 2 = 0 .
875 to give the samerelative afﬁnity.
As a comparison, we also calculate the distillation distance deﬁned in [15], whichutilizes L1 distance and depends on the absolute magnitudes of probabilities.
Using this metric,the distance increases quickly as the capacity gap increases.
These observations demonstrate thatteachers know approximately the same about relative class afﬁnities while their absolute values differsigniﬁcantly under traditional temperature scaling .
The proposed ATS could enlarge the derived variance of larger teachers.
The above analysisveriﬁes that the over-conﬁdent teachers experience lower derived variance under traditional temperature scaling although they grasp the relative class afﬁnities well.
For an intuitive visualization, weplot the probabilities for a randomly sampled instance whose correct class is y = 1 from CIFAR-100.
The top two rows in Fig.
8 show the results of RNX29-4-4d and RNX29-64-4d under traditionaltemperature scaling (TS), where the latter really experiences a smaller derived variance .
Using ATScould enhance the derived variance as shown at the bottom row (the bars are more jagged).
Then,we study the change of derived average (DA) and derived variance (DV) as temperature increases.
Given a τ , we obtain the DA and DV for all training samples via softmax and then calculate theaverage.
For ATS, we use τ 1 = 1 .
25 τ and τ 2 = 0 .
75 τ .
The curves are shown in Fig.
According to
Proposition 4.3, e ( q ) increases as τ increases, which enhances the efﬁcacy of smooth regularization .
Notably, this term changes nearly the same between teachers with various capacities.
However, thederived variance v ( q ) differs a lot.
Empirically, v ( q ) ﬁrst increases and then decreases, and themaximal of the larger teacher’s DV is smaller, which veriﬁes the Corollary 4.7.
Because derivedvariance corresponds to the efﬁcacy of class discriminability , this shows why larger teachers can’tteach well.
Using the proposed ATS could enhance the derived variance , which equivalently improvesthe efﬁcacy of class discriminability in KD.
We conclude that traditional temperature scaling leads to
[Figure] [Figure]
V GG
ResNet
KDTSKDATS 0 .
S hu ﬄ e N e t V
KDTSKDATS
R E S 14 R E S 20 R E S 32 R E S 44 R E S 56 R E S 110 0 .0 .
M o b il e N e t V
KDTSKDATS 0 .
WideResNet
KDTSKDATS 0 .
KDTSKDATS
W R N 28 - 1 W R N 28 - 2 W R N 28 - 3 W R N 28 - 4 W R N 28 - 6 W R N 28 - 8 0 .0 .
KDTSKDATS 0 .
ResNeXt KDTSKDATS 0 .
KDTSKDATS
R N X 29 - 4 - 4 d R N X 29 - 8 - 4 d R N X 29 - 16 - 4 d R N X 29 - 24 - 4 d R N X 29 - 32 - 4 d R N X 29 - 64 - 4 d 0 .0 .
KDTSKDATS
[Table]
[Table] [Table]
[Table] [Table]
distillation labels with less discriminative information among wrong class probabilities; our proposed
ATS could enhance the discrimination among wrong classes and beneﬁt the distillation process .
5.2 Performances
ATS makes larger teachers teach well again.
Previous studies ﬁnd that more accurate teachers can’t necessarily teach well [6, 31].
As shown in Fig.
9, although we tune temperatures in{ 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 12 . 0 , 16 . 0 } , larger teachers still teach worse under traditional temperature scaling(the solid curves).
However, using ATS (the dashed curves) could make larger teachers teach well orbetter again.
The details are in Appendix C.4.
ATS surpasses previous methods with advanced techniques.
We compare with SOTA methodsand list the results on CIFAR-100, TinyImageNet, CUB, and Dogs in Tab. 1 and Tab. 2.
Details ofcompared methods are in Appendix C.3.
The last column of these tables shows the average performance of corresponding rows.
The larger teacher could slightly improve the students’ performancesvia traditional KD, i.e., 68 .
24% → 69 . 21% and 57 .
94 → 58 . 89% .
However, using a smaller teacher(the row of “ST-KD") could obtain about 2% improvement on average, i.e., 68 .
24% → 70 .
32% and57 .
94% → 59 .
This again veriﬁes that larger teachers really teach worse on various datasets.
Taking advantage of the early-stopped teacher (ESKD), the teacher assistant (TAKD), and the student customized teacher (SCKD) only improves the larger teacher slightly, e.g., 69 .
21% → 69 .
86%and 58 .
89% → 59 .
These results do not even surpass the small teacher.
ResKD improvesthe students’ performances a lot via introducing the residual student and taking the two students’ensemble, which surpasses the ensemble performances of two separately trained students underdifferent initializations (the “Ens" row).
For a fair comparison, we also test the performances of
[Table]
TinyImageNet:WRN50-2 → SFV      λ =0.
λ
=0.
λ
=0.
λ
=0.
λ
=0.
T e s t
A cc u r a c y
WRN28-8 → MV
KD TS: τ =2.0 KD ATS:
τ 1 =3.0, τ 2 =2.
our methods via repeating the “KD+ATS" two times and making predictions via the ensemble.
Theresults are almost 1% higher than ResKD.
Results on CIFAR-10 and speech data are in Appendix D.
Ablation Studies One might argue that the validity of ATS is due to the tuning from a larger hyperparameter space.
We vary τ 1 from 1.0 to 6.0, τ 2 from 1.0 to 5.0, and record the performances in
Fig.
Obviously, setting τ 1 > τ 2 could be better, and especially, we recommend the setting ofτ 2 ∈
[ τ 1 − 2 ,τ 1 − 1 ] .
Although we introduce one more hyper-parameter, ATS is simple to implementand surpasses SOTA methods that take advanced techniques.
We achieve the goal of only utilizingsimple operations to make larger teachers teach well again.
Then, we study the trade off of the KDloss and CE loss under different λ .
The results of “WRN28-8 → MV2" on CIFAR-100 (Fig. 11)veriﬁes that ATS could improve the performances under various λ .
We also compare ATS with othertypes of KD under various scenes, and the results are in Appendix D.3.
Conclusion
We study the miraculous phenomenon in KD that a more accurate model doesn’t necessarily teachbetter.
The proposed KD decomposition attributes the success of a better teacher to three factors,including correct guidance , smooth regularization , and class discriminability .
Through theoreticalanalysis, over-conﬁdent teachers could not release their potential abilities of the class discriminabilityunder traditional temperature scaling.
As a simple yet effective solution, we propose Asymmetric
Temperature Scaling (ATS) to enhance the derived variance of larger teachers, making their distillation labels more discriminative when teaching students.
Extensive experimental results verify thesuperiorities of our proposed methods.
Broader Impact
We focus on the variance of wrong class probabilities to analyze why larger teachers cannot teachwell and hope that our research could bring a new perspective to the KD ﬁeld.
Our work has nopotential negative societal impacts. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Introducing one more hyper-
parameter.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Nopotential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]