I S L ABEL S MOOTHING T RULY I NCOMPATIBLE WITH K NOWLEDGE D ISTILLATION : A N E MPIRICAL S TUDY
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation (M¨uller et al., 2019).
We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits .
We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes.
Then we propose a metric to quantitatively measure the degree of erased information in sample’s representation.
After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classiﬁcation,
Binary Networks, and Neural Machine Translation.
Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness.
I NTRODUCTION
Label smoothing (Szegedy et al., 2016) and knowledge distillation (Hinton et al., 2015) are two commonly recognized techniques in training deep neural networks and have been applied in many state-of-the-art models, such as language translation (Vaswani et al., 2017; Tan et al., 2019;
Zhou et al., 2020), image classiﬁcation (Xie et al., 2019; He et al., 2019) and speech recognition (Chiu et al., 2018;
Pereyra et al., 2017; Chorowski & Jaitly, 2017).
Recently a large body of studies is focusing on exploring the underlying relationships between these two methods, for instance, M ¨ uller et al.
(M ¨ uller et al., 2019) discovered that label smoothing could improve calibration implicitly but will hurt the effectiveness of knowledge distillation.
Yuan et al.
(Yuan et al., 2019) considered knowledge distillation as a dynamical form of label smoothing as it delivered a regularization effect in training.
The recent study (Lukasik et al., 2020) further noticed label smoothing could help mitigate label noise, they showed that when distilling models from noisy data, the teacher with label smoothing is helpful.
Despite the massive and intensive researches, how to use label smoothing as well as knowledge distillation in practice is still unclear, divergent, and under-explored.
Moreover, it is hard to answer when and why label smoothing works well or not under a variety of discrepant circumstances.
View of incompatibility between label smoothing and knowledge distillation.
Recently, M¨uller et al. proposed a new standpoint that teachers trained with label smoothing distill inferior student compared to teachers trained with hard labels, even label smoothing improves teacher’s accuracy , as the authors found that label smoothing tends to “erase” information contained intra-class across individual examples, which indicates that the relative information between logits will be erased to some extent when the teacher is trained with label smoothing.
This rising idea is becoming more and more dominant and has been quoted by a large number of recent literatures (Arani et al., 2019;
Tang et al., 2020; Mghabbar & Ratnamogan, 2020; Shen et al., 2020; Khosla et al., 2020).
However, this seems reasonable observation basically has many inconsistencies in practice when adopting knowledge distillation with smoothing trained teachers.
Thus, we would like to challenge whether this perspective is entirely correct?
To make label smoothing and knowledge distillation less mysterious, in this paper, we ﬁrst systematically introduce the mechanism and correlation [Figure] between these two techniques.
We then present a novel connection of label smoothing to the idea of “erasing” relative information.
We expose the truth that factually the negative effects of erasing relative information only happens on the semantically different classes.
Intuitively, those classes are easy to classify as they have obvious discrepancies.
Therefore, the negative effects during distillation are fairly moderate.
On those semantically similar classes, interestingly, we observe that erasing phenomenon can enforce two clusters being away from each other and actually enlarge the central distance of clusters between classes , which means it makes the two categories easier for classifying, as shown in Fig.
These classes in traditional training procedure are difﬁcult to distinguish, so generally, the beneﬁts of using label smoothing on teachers outweigh the disadvantages when training in knowledge distillation.
Our observation in this paper supplements and consummates prior M¨uller et al.
’s
discovery essentially, demonstrates that label smoothing is compatible with knowledge distillation through explaining the erasing logits information on similar classes.
We further shed light on understanding the behavior and effects when label smoothing and knowledge distillation are applied simultaneously, making their connection more interpretable, practical and clearer for usage.
How to prove that their discovery is not judgmatic?
We clarify such widely accepted idea through the following exploratory experiments, and exhaustively evaluate our proposed hypothesis: (i) Standard ImageNet-1K
(Deng et al., 2009), ﬁne-grained CUB200-2011 (Wah et al., 2011b) and noisy iMaterialist product recognition; (ii) Binary neural networks (BNNs) (Rastegari et al., 2016); (iii)
Neural machine translation (NMT) (Vaswani et al., 2017).
Intriguingly, we observe that if the teacher is trained with label smoothing, the absolute values of converged distilling loss on training set are much larger than that the teacher is trained with hard labels, whereas, as we will discuss in detail later in Fig.
5 and 6, the accuracy on validation set is still better than that without label smoothing.
We explain this seemingly contradictory phenomenon through visualizing the teachers’ output probabilities with and without label smoothing, it suggests that the suppression of label smoothing for knowledge distillation only happens on training phase as the distributions from teachers with label smoothing is more ﬂattening, the generalization ability of networks on validation set is still learned during optimization.
That is to say, the dynamical soft labels generated by teacher networks can prevent learning process from overﬁtting to the training data, meanwhile, improving the generalization on the unseen test data.
Therefore, we consider this erasing relative information function within class from label smoothing as a merit to distinguish semantically similar classes for knowledge distillation, rather than a drawback.
Moreover, we also propose a stability metric to evaluate the degree of erased information by label smoothing, we found the proposed metric is highly aligned with model’s accuracy and can be regarded as a supplement or alternateness to identify good teachers for knowledge distillation.
Finally, we discuss several intriguing properties of label smoothing we observed on the long-tailed category distribution and rapidly-increased #class scenarios, as provided in Sec. 7.
More speciﬁcally, this paper aims to address the following questions:
• Does label smoothing in teacher networks suppress the effectiveness of knowledge distillation?
Our answer is No .
Label smoothing will not impair the predictive performance of students. Instead, we observe that a smoothing trained teacher can protect the student from overﬁtting on the training set, which means that with smoothing trained teachers in knowledge distillation, the training loss is always higher than that without smoothing, but the validation accuracy is still similar or even better.
[Figure]
• What will actually determine the performance of a student in knowledge distillation?
From our empirical study, we observe if the student architecture is settled, the dominating factor in knowledge distillation is the quality of supervision, i e., the performance of a teacher network.
Higher-accuracy teacher is particularly successful in distilling better students, regardless it is trained with or without label smoothing.
This observation is partially against the conclusion in (M ¨ uller et al., 2019) which stated “a teacher with better accuracy is not necessary to distill a better student”.
• When will the label smoothing indeed lose its effectiveness for learning deep neural networks?
Long-tailed class distribution and increased number of classes are two scenarios we observed wherein label smoothing will lose or impair its effectiveness.
We empirically verify the ﬁndings on iNaturalist 2019 (Van Horn et al., 2018), Place-LT (Liu et al., 2019) and curated ImageNet (Liu et al., 2019).
B ACKGROUND
In this section, we ﬁrst introduce the background of label smoothing and knowledge distillation through a mathematical description.
Given a dataset D =
( X, Y ) over a set of classes K , X is the input data and Y is the corresponding one-hot label with each sample’s label y ∈ { 0 , 1 } K , where the element y c is 1 for the ground-truth class and 0 for others.
Label smoothing replaces one-hot hard label vector
y with a mixture of weighted y and a uniform distribution:
[Equation]
where α is a small constant coefﬁcient for ﬂattening the one-hot labels.
Usually, label smoothing is adopted when the loss function is cross-entropy, and the network uses softmax function to the last layer’s logits z to compute the output probabilities p , so the gradient of each training sample with respect to z will be: ∇H ( p , y ) =
p − y
= K (cid:80)
[Equation]
where H
[Equation]
p
[Equation]
1 y
[Equation]
p c is the cross-entropy loss and z c is c -th logit in z .
[Figure]
To further understand the effects of label smoothing on loss function, Fig.
3 illustrates correction effects of smoothing on the binary cross-entropy loss ( K = 2 ).
We can observe that the standard logistic loss ( α = 0 ) vanishes for large and conﬁdent positive predictions, and becomes linear for large negative predictions.
Label smoothing will penalize conﬁdent predictions and involve a ﬁnite positive minimum as it aims to minimize the average perclass.
Generally, larger α values will produce larger loss values rebounding at positive predictions.
This is also the underlying reason that smoothed loss can ﬂatten the predictions of a network.
In knowledge distillation, we usually pre-train the teacher model T w on the dataset in advance.
The student model S w is trained over the same set of data, but utilizes labels generated by T w .
More speciﬁcally, we can regard this process as learning S w on a new labeled dataset ˜ D
=
( X, T w ( X )) .
Once the teacher network is trained, its parameters will be frozen in the whole distillation.
The student network S w is trained by minimizing the similarity between its output and two parts: the hard one-hot labels and the soft labels generated by the teacher network.
Letting p T w c ( X )
=
T w ( X )[ c ] , p S w c ( X ) =
S w ( X )[ c ] be the probabilities assigned to class c in the teacher model T w and student model S w .
The distillation loss can be formulated as λ H ( p S w , y ) + (1 − λ )
H ( p S w / T , p T w / T ) where T is the temperature scaling factor and λ is the trade-off coefﬁcient to balance the two terms.
T HE “E RASE I NFORMATION ” E FFECT BY L ABEL S MOOTHING
This section aims to explain the erasing information effect more thoroughly.
We start by reproducing the visualization of penultimate layer’s activations using the same procedure from (M¨uller et al., 2019).
We adopt ResNet-50 trained with hard and smoothed labels on ImageNet.
As shown in Fig.
1, we obtain similar distributions as (M¨uller et al., 2019).
Since examples in training set are the ones used for distillation, we mainly analyze the visualization from the training data.
The core ﬁnding in (M ¨ uller et al., 2019) is that if a teacher is trained with hard labels, representations of examples are distributed in broad clusters, which means that different examples from the same class can have different similarities ( D 1 and D 2 ) to other classes.
For a teacher trained with label smoothing, they observed the opposite behavior.
Label smoothing encourages examples to lie in tight equally separated clusters, so each example of one class has very similar proximities ( D 1 is closer to D 2 ) to examples of the other classes.
Our re-visualization also supports this discovery.
The authors derive the conclusion that a teacher with better accuracy is not necessarily to distill a better student.
This seems reasonable as the broad clusters can enable different examples from the same class to provide different similarities to other classes, which contains more information for knowledge distillation.
However, if refocusing on the two semantically similar classes, when label smoothing is applied, the clusters are much tighter because label smoothing encourages each example is to be equidistant from all other class’s templates, while, the tight cluster substantially promotes different class representations to be separate, i.e., the distance of clusters D c increases, which further indicates that different class examples obtain more distinguishable features.
This phenomenon is crucial as these difﬁcult classes are the key for boosting classiﬁcation performance.
Generally, it is not necessary to measure “how much a poodle is a particularly similar to a tench” since we have enough evidence to classify them, but it is critical to have information “how different is a toy poodle to a miniature poodle”.
Visualizations of teacher predictions.
We further visualization the mean distribution of different classes crossing examples, as shown in Fig.
We average all the probabilities after softmax layer if the examples belong to the same category, and show the ﬁrst 100 classes in ImageNet.
Usually, the probabilities have a major value (the bars in Fig.
4 (1)) that represents model’s prediction on category and other small values (i.e., minor predictions in Fig.
4 (2)) indicate that the input image is somewhat similar to those other categories, some discussions about this are given in Appendix F. Our purpose of this visualization is to make certain of what label smoothing really calibrates in a network and shed light on how it affects the network predictions.
We can observe in this ﬁgure that a model trained with label smoothing will generate more softened distributions, but the relations across different classes are still preserved.
We conjecture the softened supervision is also the reason why teachers with label smoothing produce larger training loss during knowledge distillation.
Consequently, label smoothing will both decrease the variance (veriﬁed by following stability metric) and mean predictive values within a class, but will not impair the relations crossing different classes.
3. A S IMPLE M ETRIC FOR M EASURING THE D EGREE OF E RASED I NFORMATION
Different from the visualization scheme (M¨uller et al., 2019) of ﬁnding an orthonormal basis of the plane that only studies this problem qualitatively, we further address the “erasing” phenomenon through a statistical metric that is simple yet effective, and can measure the degree of erasing operation quantitatively.
Our motivation behind it is simple and straight-forward: If label smoothing erases relative information within a class, the variance of intra-class probabilities will decrease accordingly, thus we can use such variance to monitor the erasing degree, since this metric evaluates the ﬂuctuation [Figure] of the representations, we can also call it the stability metric.
The deﬁnition is as follows:
[Equation]
where i is the index of images and n c is the #image in class c .
p T w { i,c } is the mean of p T w in class c .
This metric utilizes the probabilities of intra-class variance to measure the stability of a teacher’s prediction.
The results on various network architectures are shown in Sec.
5 and a PyTorch-like code for calculating this metric is given in Appendix C.
Such metric has at least two advantages: 1) It can measure the degree of erased information quantitatively and further help discover more interesting phenomena, e.g., we observe that data augmentation method like CutMix (Yun et al., 2019) with longer training erases the relative information on logits dramatically and can further be reinforced by label smoothing.
We found that the proposed metric is highly aligned with model accuracy, thus such metric can be used as a complement for accuracy to evaluate the quality of teacher’s supervision for knowledge distillation.
A C LOSE L OOK AT L ABEL S MOOTHING AND K NOWLEDGE D ISTILLATION
A few recent studies (Shen & Savvides, 2020; Shen et al., 2019) suggested supervised part H ( p S w , y )
(i.e. hard labels) is not necessary as soft prediction is adequate to provide crucial information for students, meanwhile, removing supervised part can avoid involving incorrect labels caused from random crop, multi-object circumstance or false annotations by humans.
Thus, here we only consider the soft part H ( p S w / T , p T w / T ) with the commonly used Kullback-Leibler divergence similarity.
KL-divergence measures the similarity of two probability distributions.
We train the student network
S θ by minimizing the KL-divergence between its output p S θ c ( X ) and the soft labels p T θ c ( X ) generated by the teacher network.
Following (M ¨ uller et al., 2019; Hinton et al., 2015) we set T = 1 as the temperature constant and it is omitted for simplicity, thus our loss function will be:
[Equation]
Here, E x ∼T w [ − log S w ( X )] is the cross-entropy between S w and T w (denoted H ( p S w , p T w ) (cid:1) .
The second term H ( T w ( X ))
=
E x ∼T w [ − log p T w ( x )] is the entropy of teacher T w and is constant with respect to T w .
We can remove it and simply minimize the loss as follows:
[Equation]
We can observe that Eq.
4 is actually a standard cross-entropy loss.
Then, we have:
Property 1.
If not consider hard labels in knowledge distillation, distillation loss and cross-entropy loss with label smoothing have the same optimizing objective, i.e., D KL ( T w (cid:107)S w )
=
H ( p S w , p T w ) .
This property shows that label smoothing and knowledge distillation have the same optimization objective, the sole difference between them is the mechanism of producing the soft labels.
Therefore, except for the neural machine translation, in this paper all of our knowledge distillation experiments are conducted without the hard labels, which means our student solely relies on the softened distribution from a teacher without the one-hot ground-truth.
This may challenge the common practice in knowledge distillation (Hinton et al., 2015; Romero et al., 2015) that adopted both hard and soft labels with the cross-entropy loss for distillation, while our surprisingly good results and previous studies (Shen & Savvides, 2020;
Shen et al., 2019; Bagherinezhad et al., 2018) indicate that knowledge distillation enabled by soft labels solely is not only an auxiliary regularization (Yuan et al., 2019) but can be the dominating supervisions, which further inspires us to carefully revisit the role of knowledge distillation and design better supervision/objective in training deep networks.
E MPIRICAL S TUDIES
Metric Evaluation.
Our results of stability metric are shown in Table 1, the second and third columns are results without label smoothing and the last two are with it.
We study the metric crossing a variety of different network architectures.
The gaps of S Stability using the same architecture measure the degree of erasing relative information.
We can observe that the variances (1- S Stability ) with label smoothing always have lower values than models trained without label smoothing, this proves that label smoothing will erase information and enforce intra-class representations of samples being similar.
Generally, the accuracy and stability have a positive correlation between them.
But the stability can even overcome some outliers, for example, Wide ResNet50 with label smoothing has lower accuracy, but the stability is still consistent to the tendency of predictive quality.
Moreover, models trained with more epochs and augmentation techniques like CutMix (Yun et al., 2019) can dramatically increase the stability, this means relative information will be erased signiﬁcantly by more augmentation together with longer training.
We emphasize that this discovery cannot be observed by the qualitative visualization method (M¨uller et al., 2019).
A PyTorch-like code is in Appendix C.
[Table]
Image Classiﬁcation.
We verify our perspective through investigating the effectiveness of knowledge distillation with label smoothing on the image classiﬁcation tasks.
We conduct experiments on three [Table]
} as teacher networks and ResNet- { 18/50/101 } as students, respectively.
More experimental settings are in Appendix A.
Results.
The visualizations of our distillation training and testing curves are shown in Fig.
A more detailed comparison is listed in Table 2 and 7.
From the visualization we found two interesting phenomena: On training set, the loss of teacher networks that trained with label smoothing is much higher than that of without label smoothing.
While on validation set the accuracy is comparable or even slightly better (The boosts on CUB is greater than those on ImageNet-1K, as shown in Table 2).
We also provide the experiments of combining hard and soft labels in Appendix E, the results still show the effectiveness of better teachers with label smoothing, which distill better students.
To make it clearer why this happens in distillation, we visualize the supervisions from teacher networks in Fig. 4 and the discussion is shown there.
It indicates that label smoothing ﬂattens teacher’s predictions which causes the enlarged training loss, while the student’s generalization ability is still preserved.
[Figure]
Binary Neural Networks (BNNs).
We then examine the effectiveness of knowledge distillation on
Binary Neural Networks.
BNN aims to learn a network that both weights and activations are discrete values in { -1, +1 } .
In the forward pass, real-valued activations are binarized by the sign function:
A b =
Sign( A r ) =
(cid:26) − 1 if A r < 0 , + otherwise .
where A r is the real-valued activation of the previous layers, produced by the binary or real-valued convolution operations.
A b is the binarized activation.
The real-valued weights are binarized by: W b = || W r ||
l 1 n
Sign( W r ) = (cid:40) − || W r ||
l 1 n if W r < 0 , + || W r || l 1 n otherwise .
where W r is the real-valued weights that are stored as latent parameters to accumulate the small gradients.
W b is the binarized weights.
We update binary weights through multiplying the sign of real-valued latent weights and the channel-wise absolute mean ( 1 n || W r || l 1 ).
Training BNNs is challenging as the gradient of optimization is approximated and the capacity of models is also limited.
We perform experiments on ImageNet-1K and results are shown in Fig.
Withal, the teacher network trained with one-hot labels (blue curve) is over-conﬁdent as the loss value is much smaller, which means that the teacher trained with label smoothing can prevent distillation process from being overconﬁdent on the training data, and obtain slightly better generalization and accuracy (63.108% vs.
63.002%) on the validation set.
These results still support our conclusion on knowledge distillation.
[Figure]
Neural Machine Translation (NMT).
Finally, we investigate our hypothesis of knowledge distillation on the German-to-English translation task using the Transformer architecture (Vaswani et al., 2017).
We utilize the distillation framework of (Tan et al., 2019) on IWSLT dataset and the pretraining/distillation curves are shown in Fig.
A consistent setting is imposed on all the two comparison experiments, except the teacher is trained with and without label smoothing.
We choose α = 0 .
1 for label smoothing as suggested by (Vaswani et al., 2017; Szegedy et al., 2016;
M¨uller et al., 2019), we use Adam (Kingma & Ba, 2014) as the optimizer, lr with 0.0005, dropout with drop rate as 0.3, weight-decay with 0 and max tokens with 4096, all of these hyper-parameters are following the original settings of (Tan et al., 2019).
Our results of Fig.
7 deliver two important conclusions: First sub-illustration (left one) proves the statement of Vaswani et al.
(Vaswani et al., 2017) that label smoothing ( α = 0 .
1 ) boosts the BLEU score of language model despite causing worse perplexity if comparing to a model is trained with one-hot/hard labels.
Second sub-illustration (right one) implies that on the machine translation task, a stronger teacher (trained with label smoothing) will still distill a higher BLEU student.
That is to say, label smoothing may not suppress the effectiveness of knowledge distillation in the NMT task.
W HAT I S A B ETTER T EACHER IN K NOWLEDGE D ISTILLATION ?
[Figure]
Better Supervision is Crucial for Distillation.
We further explore the effects of teacher’s accuracy on the student through ﬁxing the student structure and switching different teachers.
We perform two settings for this ablation study: using the same teacher structure with different training strategies and different teacher architectures.
All teacher models are re-trained with label smoothing.
The results are shown in Fig.
8 (Left) and Table 8, generally, teachers with higher accuracies can distill stronger students, but they are not linear related and is limited to the capability of the student itself.
To further support the argument that better teachers usually distill better students , we choose the state-of-the-art
FixRes model (Touvron et al., 2019) for both the teacher and student and perform our distillation training via Eq. 4.
The results are shown in Fig.
8 (Right) and our method is slightly better than the baseline and FixRes.
Note that the compared FixRes is already the state-of-the-art with ResNet and
ResNeXt architecture, so our result (under ResNet family) is a very competitive single-crop accuracy to date on ImageNet-1K.
W HAT C IRCUMSTANCES I NDEED W ILL M AKE LS L ESS E FFECTIVE ?
Long-Tailed Distribution.
We found the long-tailed datasets with imbalanced samples across classes will indeed suppress the effectiveness of label smoothing (LS) and hurt model performance.
The results on long-tailed ImageNet-LT, Places365-LT and iNaturalist 2019 are shown in Table 3.
Our results on these three datasets empirically verify this observation and support this conclusion.
Since LS has weight shrinkage and regularisation effects (Lukasik et al., 2020), we have following conjecture:
Conjecture 1.
Weight shrinkage effect (Regularisation) enabled by label smoothing is no longer effective on the long-tailed recognition circumstance and will further impair the performance.
The weight shrinkage effect (label smoothing regularization) has been proven in (Lukasik et al., 2020)
(Theorem 1) on the linear models.
As label smoothing will assign probabilities equally to all minor classes (i.e., (1 − α ) · I pos + αL − 1 · J neg ), this operation may be biased to many-shot classes in long-tailed scenario.
We derive the conclusion empirically that label smoothing is inapplicable when the class distribution is long-tailed.
More #Class.
This is another circumstance we found will impair the effectiveness of label smoothing.
The results are shown in Table 4 and Fig.
9, we average the gains of label smoothing crossing two different network architectures and compare the boosts between curated ImageNet-100/500 and 1K.
Generally, more classes will reduce the improvement produced by label smoothing.
[Table]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
C ONCLUSION
We empirically demonstrated that label smoothing could both decrease the variance (i.e., erase relative information between logits) and lower mean predictive values (i.e., make prediction less conﬁdent) within a category, but it does not impair the relation distribution crossing different categories.
Our results on image classiﬁcation, binary neural networks, and neural machine translation indicate that label smoothing is compatible with knowledge distillation and this ﬁnding encourages more careful to understand and utilize the relationships of label smoothing and knowledge distillation in practice.
We found through extensive experiments and analyses that the indeed circumstances label smoothing will lose its effectiveness are long-tailed distribution and increased number of classes.
Our study also suggests that, to ﬁnd a better teacher for knowledge distillation, accuracy of teacher network is one factor, the stability of supervision from teacher network is also an alternative indicator.
A PPENDIX
In this appendix, we provide details omitted in the main text, including:
• Section A: A introduction of datasets and experimental settings (Sec. 5 “Empirical Studies” of the main paper.)
• Section B: A discussion about what happens if teachers with label smoothing are inferior?
(Sec.
“What Is a Better Teacher in Knowledge Distillation?”
of the main paper.)
• Section C:
A PyTorch-like code for calculating stability metric (Sec. 3.1 “A Simple Metric” of the main paper.)
• Section D: (1) Table 7 is a detailed version of our Table 2 (Sec. 5 “Empirical Studies” of the main paper.)
(2) Table 8 is a detailed results for visualization of Fig.
8 (Sec.
“What Is a Better Teacher in Knowledge Distillation?” of the main paper.)
• Section E: Results of knowledge distillation by adopting both hard and soft labels (Sec. 5 “Empirical Studies” of the main paper.)
• Section F: Discussions of minor probabilities from teacher network.
(Sec.
“The Erase Information Effect by Label Smoothing” of the main paper.)
• Section G: Supplementary metric of inter-class variation.
(Sec. 5 “Metric Evaluation” of the main paper.)
A D ATASETS AND E XPERIMENTAL S ETTINGS A. F OR T RAINING T EACHER AND K NOWLEDGE D ISTILLATION M ODELS TO V ERIFY THE C OMPATIBILITY BETWEEN L ABEL S MOOTHING AND K NOWLEDGE D ISTILLATION
Standard ImageNet-1K Classiﬁcation (Deng et al., 2009)
ImageNet-1K contains ∼ 1.28 million images in 1000 classes for training and 50K images for validation.
For training teacher networks, we follow the standard training protocol (He et al., 2016; Goyal et al., 2017), i.e., total training epoch is 90, initial learning rate is 0.1 and decayed to 1/10 with every 30 epochs.
For distillation, as the supervision is a soft distribution and will dynamically change, we train with 200 epochs and the learning rate is multiplied by 0.1 at 80 and 160 epochs.
All models are trained from scratch.
Fine-grained Recognition on CUB200-2011 (Wah et al., 2011a)
This dataset contains 200 bird species and 11,788 images, and is a wildly-used ﬁne-grained classiﬁcation benchmark.
Following (Wah et al., 2011a), we use standard split with ∼ 30 samples of each breed for both training and testing.
As CUB200-2011 has limited training data, we ﬁne-tune our model from the ImageNet pre-trained parameters.
Both teachers and students are trained with 200 epochs and the learning rate is multiplied by 0.1 at 80 and 160 epochs.
Noisy Product Recognition (iMaterialist Challenge) 2 .
The training data consists of 1,011, images from 2,019 categories.
The validation data has 10,095 images (around 5 for each category).
The training data is collected from the Internet and contains some noise ( ∼ 30% of incorrect labels), the validation data has been cleaned by human annotators.
The training protocol is following CUB2002011.
While, as the training images in this dataset is sufﬁciently enough, all our models are trained from scratch.
A. F OR T RAINING T EACHERS O NLY TO E XPLORE THE E FFECTIVENESS OF L ABEL S MOOTHING
Long-tailed iNaturalist 2019 (Van Horn et al., 2018).
The iNaturalist Challenge 2019 dataset contains 1,010 species, with a combined training and validation set of 268,243 images, wherein, the validation set has a uniform distribution of three images in each category.
While, the training set is constructed with a long-tailed distribution of image numbers.
We conduct experiments using the same training protocol as the CUB200-2011 dataset.
we evaluate our results on validation set using
Top-1/Top-3 metrics.
Also, this protocol is applied to the Place-LT dataset.
Place-LT (Liu et al., 2019).
The training set of this dataset is constructed by a Pareto distribution (Reed, 2001) with a power value α = 6 .
It contains a total number of 184.5K images fromcategories.
There are 4,980 ∼ 5 images per class from the 365 classes of Places-365 dataset (Zhou et al., 2017).
20 images per class are randomly selected from the original Places-365 training set as a validation set.
The training protocol is the same as iNaturalist 2019, we also evaluate our results on validation set using Top-1/Top-3 metrics.
ImageNet-LT (Liu et al., 2019).
The long-tailed ImageNet is constructed by sampling a subset from the standard ImageNet (Deng et al., 2009) following the Pareto distribution with the power value α = 6 .
The whole dataset contains 115.8K images from 1000 categories, with maximal 1280 images and minimal 5 images per class.
20 training images per class are randomly selected from the origin training set of ImageNet as a validation set.
All our models are trained from scratch.
Curated ImageNet-100/500/1K. We construct this dataset by randomly selecting a subset of 100/500/1K categories in the standard ImageNet (Deng et al., 2009).
We use this dataset to explore the impact of #class for label smoothing.
We observe that label smoothing is more effective on the dataset with fewer numbers of classes.
[Table]
B W HAT HAPPENS IF TEACHERS WITH LABEL SMOOTHING ARE INFERIOR ?
In the main paper
[Table]
we discussed that the superior teachers (even trained with label smoothing) could distill better students.
Also, as we noted that long-tailed distribution will suppress the effectiveness of label smoothing.
In this section, we would like to explore what happens if we use these inferior teachers for knowledge distillation?
We conduct experiments on the long-tailed iNaturalist 2019 data and show more evidence to prove our conclusion that the quality of supervision is more crucial for knowledge distillation than the ways of training teachers (with or without label smoothing).
The results are shown in Table 6, it can be observed that if teachers with lower accuracies, the distilled students will also have poorer performance, regardless of the teacher is trained with label smoothing or not.
[Table]
C A P Y T ORCH - LIKE C ODE FOR C ALCULATING S TABILITY M ETRIC
Algorithm 1 PyTorch-like Code for Calculating Stability Metric.
[Table]
D T WO D ETAILED T ABLES OF R ESULTS
[Table]
E R ESULTS OF K NOWLEDGE D ISTILLATION WITH H ARD AND S OFT L ABELS
Here we verify whether label smoothing is still effective when adopting hard labels in knowledge distillation.
As we mentioned above, the traditional distillation loss can be formulated as
[Table] [Table]
λ H ( p S w , y ) + (1 − λ ) H ( p S w / T , p T w / T ) .
We use ResNet-50 as the teacher and ResNet-18 as the student and conduct experiments with three different ratios of λ : 0.3, 0.5, 0.7.
Our Top-1/ results are given in Table 9.
We can see the teacher networks with label smoothing still distills better students than the teacher without label smoothing.
Also, with a higher probability of hard labels, the performance declines.
F M INOR P ROBABILITIES IN T EACHER N ETWORK
The visualization of minor probabilities is shown in Fig.
4 (2), we have two interesting observations in it: 1)
In the enlarged area of Fig.
4 (2), we can observe that several classes have values are close to zero, this phenomenon means in ImageNet dataset, there are several categories that are very unique to other classes and the model will barely predict other classes to them; 2) We can also observe more “red” areas gather at the bottom of the bars.
It reﬂects that the probabilities from the model trained with label smoothing will be assigned to more classes with smaller values, this is also consistent with what the label smoothing operation actually does.
G S UPPLEMENTARY M ETRICS OF I NTER - CLASS /E NTIRE V ARIATIONS
To better understand the behavior of erased information by label smoothing across different classes, we also introduce the inter-class stability metric as follows:
[Equation]
where K is the number of classes.
p T w { all } is average of probability across the entire dataset.
p T w { c } is the mean of p T w in class c .
S interStability can be regarded as a supplement for our intra-class stability metric to measure the inter-class stability.
The average probability within each class p T w { c } is deﬁned as:
[Equation] [Equation]
where i is the index of images and n c is the #image in class c .
The average probability across entire dataset p T w { all } is deﬁned as:
[Equation]
[Table]
where N is the number of samples in the entire dataset.
This metric utilizes the probabilities of inter-class variance to measure the stability of a teacher’s prediction.
The results on various network architectures are shown in Table 10.