Original: R ETHINKING S OFT L ABELS FOR K NOWLEDGE D ISTILLATION : A B IAS –V ARIANCE T RADEOFF P ERSPECTIVE
Revised: R ETHINKING S OFT L ABELS FOR K NOWLEDGE D ISTILLATION : A B IAS -V ARIANCE T RADEOFF P ERSPECTIVE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network.
Revised: Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network.

Original: The outputs from the teacher network are used as soft labels for supervising the training of a new network.
Revised: The outputs from the teacher network are used as soft labels for supervising the training of a new network.

Original: Recent studies (M¨uller et al., 2019; Yuan et al., 2020) revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network.
Revised: Recent studies (M¨uller et al., 2019; Yuan et al., 2020) revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network.

Original: From the perspective of statistical learning, regularization aims to reduce the  variance while the original hard labels are for bias error reduction.
Revised: From the perspective of statistical learning, regularization aims to reduce the variance, however how bias and variance change is not clear for training with soft labels.

Original: Therefore, during training soft labels are used for both bias error and variance reduction.

Original: In this paper, we investigate the bias-variance tradeoff brought by distillation with soft labels.
Revised: In this paper, we investigate the bias-variance tradeoff brought by distillation with soft labels.

Original: Speciﬁcally, we observe that  bias-variance tradeoffs are sample-wise and the distillation performance is negatively associated with the number of  samples facing the tradeoff.
Revised: Speciﬁcally, we observe that during training the bias-variance tradeoff varies sample-wisely. Further, under the same distillation temperature setting, we observe that the distillation performance is negatively associated with the number of some speciﬁc samples, which are named as regularization samples since these samples lead to bias increasing and variance decreasing.

Original: Nevertheless, we empirically ﬁnd that  samples facing the tradeoff are still beneﬁcial for distillation.
Revised: Nevertheless, we empirically ﬁnd that completely ﬁltering out regularization samples also deteriorates distillation performance.

Original: Our discoveries inspired us to propose the novel weighted soft labels to help the network focus on the sample not facing the  tradeoff.
Revised: Our discoveries inspired us to propose the novel weighted soft labels to help the network adaptively handle the sample-wise biasvariance tradeoff.

Original: Experiments on standard evaluation benchmarks validate the effectiveness of our method.
Revised: Experiments on standard evaluation benchmarks validate the effectiveness of our method.

Original: Our code is available at *.
Revised: Our code is available in the supplementary.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For deep neural networks (Goodfellow et al., 2016), knowledge distillation  (Ba & Caruana, 2014; Hinton et al., 2015) refers to the technique that uses well-trained networks to guide the training of another network.
Revised: For deep neural networks (Goodfellow et al., 2016), knowledge distillation (KD) (Ba & Caruana, 2014; Hinton et al., 2015) refers to the technique that uses well-trained networks to guide the training of another network.

Original: Typically, the well-trained network is named as the teacher network while the network to be trained is named as the student network.
Revised: Typically, the well-trained network is named as the teacher network while the network to be trained is named as the student network.

Original: For distillation, the predictions from the teacher network are leveraged and referred to as the soft labels (Balan et al., 2015; M ¨ uller et al., 2019).
Revised: For distillation, the predictions from the teacher network are leveraged and referred to as the soft labels (Balan et al., 2015; M ¨ uller et al., 2019).

Original: Soft labels generated by the teacher network have been proven effective in large-scale empirical studies (Liang et al., 2019; Tian et al., 2020; Zagoruyko & Komodakis, 2017; Romero et al., 2015) as well as recent theoretical studies (Phuong & Lampert, 2019).
Revised: Soft labels generated by the teacher network have been proven effective in large-scale empirical studies (Liang et al., 2019; Tian et al., 2020; Zagoruyko & Komodakis, 2017; Romero et al., 2015) as well as recent theoretical studies (Phuong & Lampert, 2019).

Original: However, the reason why soft labels are beneﬁcial to the student network is still not well explained.
Revised: However, the reason why soft labels are beneﬁcial to the student network is still not well explained.

Original: Giving a clear theoretical explanation is challenging:
Revised: Giving a clear theoretical explanation is challenging:

Original: The optimization details of a deep network with the common one-hot labels are still not well-studied (Nagarajan & Kolter, 2019), not to mention training with the soft labels.
Revised: The optimization details of a deep network with the common one-hot labels are still not well-studied (Nagarajan & Kolter, 2019), not to mention training with the soft labels.

Original: Nevertheless, two recent studies (M¨uller et al., 2019; Yuan et al., 2020) shed light on the intuitions about how the soft labels work.
Revised: Nevertheless, two recent studies (M¨uller et al., 2019; Yuan et al., 2020) shed light on the intuitions about how the soft labels work.

Original: Speciﬁcally, label smoothing, which is a special case of soft labels based training, is shown to regularize the activations of the penultimate layer to the network (M¨uller et al., 2019).
Revised: Speciﬁcally, label smoothing, which is a special case of soft labels based training, is shown to regularize the activations of the penultimate layer to the network (M¨uller et al., 2019).

Original: The regularization property of soft labels is further explored in (Yuan et al., 2020).
Revised: The regularization property of soft labels is further explored in (Yuan et al., 2020).

Original: They hypothesize that in knowledge distillation, one main reason why the soft labels work is the regularization introduced by soft labels.
Revised: They hypothesize that in KD, one main reason why the soft labels work is the regularization introduced by soft labels.

Original: Based on the assumption, the authors design a teacher-free distillation method by turning the predictions of the student network into soft labels.
Revised: Based on the assumption, the authors design a teacher-free distillation method by turning the predictions of the student network into soft labels.

Original: The evidence of the regularization brought by soft labels drives us to rethink soft labels for knowledge distillation: Soft labels areessentially targets in supervised learning, indicating that soft labels are both supervisory signals and regularizers.
Revised: Considering that soft labels are targets for distillation, the evidence of the regularization brought by soft labels drives us to rethink soft labels for KD: Soft labels are both supervisory signals and regularizers.

Original: Meanwhile, it is known that there is a tradeoff between ﬁtting the data and regularizing the model complexity, i.e., the bias-variance dilemma, according to statistical learning theories (Kohavi & Wolpert, 1996; Bishop, 2006).
Revised: Meanwhile, it is known that there is a tradeoff between ﬁtting the data and imposing regularizations, i.e., the bias-variance dilemma (Kohavi & Wolpert, 1996; Bishop, 2006), but it is unclear how bias and variance change for distillation with soft labels.

Original: Recall that soft labels are supervision and regularizers, it can be inferred that soft labels can help reduce the bias error along with the variance. Since the bias-variance tradeoff is an important issue in statistical learning, we investigate  bias-variance tradeoffs for soft labels. We ﬁrst present an intuitive understanding of bias-variance tradeoff for soft labels, by exploring the semantics behind the regularization brought by soft labels.
Revised: Since the bias-variance tradeoff is an important issue in statistical learning, we investigate whether the bias-variance tradeoff exists for soft labels and how the tradeoff affects distillation performance. We ﬁrst compare the bias and variance decomposition of direct training with that of distillation with soft labels, noticing that distillation results in a larger bias error and a smaller variance.

Revised: Then, we rewrite distillation loss into the form of a regularization loss adding the direct training loss.

Original: Then, through inspecting the gradient from bias reduction and variance reduction, we notice that for soft labels, bias-variance tradeoffs are sample-wise.
Revised: Through inspecting the gradients of the two terms during training, we notice that for soft labels, the biasvariance tradeoff varies sample-wisely.

Original: Moreover, by looking into a conclusion from (M¨uller et al., 2019), we observe that  the number of samples facing bias-variance tradeoff is negatively associated with the distillation performance.
Revised: Moreover, by looking into a conclusion from (M¨uller et al., 2019), we observe that under the same temperature setting, the distillation performance is negatively associated with the number of some certain samples.

Original: As for soft labels the bias-variance tradeoff is sample-wise, we propose to avoid sample-wise tradeoff, instead of tuning the tradeoff. To achieve this, we ﬁrst examine the effectiveness of several straightforward solutions, leading us to think that  samplesfacing the tradeoff arestill useful for distillation but not fully exploited.
Revised: These samples lead to bias increase and variance decrease and we name them as regularization samples. To investigate how regularization samples affect distillation, we ﬁrst examine if we can design ad hoc ﬁlters for soft labels to avoid training with regularization samples. But completely ﬁltering out regularization samples also deteriorates distillation performance, leading us to speculate that regularization samples are not well handled by standard KD.

Original: In the light of these ﬁndings, we propose weighted soft labels for distillation to  adaptively assign a lower weight to the samples facing the tradeoff.
Revised: In the light of these ﬁndings, we propose weighted soft labels for distillation to handle the sample-wise bias-variance tradeoff, by adaptively assigning a lower weight to regularization samples and a larger weight to the others.

Original: To sum up, our contributions are:
Revised: To sum up, our contributions are:

Original: • For knowledge distillation, we analyze how the soft labels work from a perspective of biasvariance tradeoff.
Revised: • For knowledge distillation, we analyze how the soft labels work from a perspective of biasvariance tradeoff.

Original: • We discover that  bias-variance tradeoffs are sample-wise and the number of  samplesfacing the tradeoff is negatively associated with the distillation performance.
Revised: • We discover that the bias-variance tradeoff varies sample-wisely. Also, we discover that if we ﬁx the distillation temperature, the number of regularization samples is negatively associated with the distillation performance.

Original: • We design straightforward schemes to avoid the tradeoffs and then propose the novel weighted soft labels for distillation.
Revised: • We design straightforward schemes to alleviate negative impacts from regularization samples and then propose the novel weighted soft labels for distillation.

Original: Experiments on large scale datasets validate the effectiveness of the proposed weighted soft labels.
Revised: Experiments on large scale datasets validate the effectiveness of the proposed weighted soft labels.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: Hinton et al.
Revised: Hinton et al.

Original: (2015) proposed to distill outputs from large and cumbersome models into smaller and faster models, which is named as knowledge distillation.
Revised: (2015) proposed to distill outputs from large and cumbersome models into smaller and faster models, which is named as knowledge distillation.

Original: The outputs for large networks are averaged and formulated as soft labels.
Revised: The outputs for large networks are averaged and formulated as soft labels.

Original: Also, other kinds of soft labels have been widely used for training deep neural networks (Szegedy et al., 2016; Pereyra et al., 2017).
Revised: Also, other kinds of soft labels have been widely used for training deep neural networks (Szegedy et al., 2016; Pereyra et al., 2017).

Original: Treating soft labels as regularizers were pointed out in (Hinton et al., 2015) since a lot of helpful information can be carried in soft labels.
Revised: Treating soft labels as regularizers were pointed out in (Hinton et al., 2015) since a lot of helpful information can be carried in soft labels.

Original: More recently, M ¨ uller et al.
Revised: More recently, M ¨ uller et al.

Original: (2019) showed the adverse effect of label smoothing upon distillation.
Revised: (2019) showed the adverse effect of label smoothing upon distillation.

Original: It is a thought-provoking discovery for the reason that both label smoothing and distillation are exploiting the regularization property behind soft labels.
Revised: It is a thought-provoking discovery for the reason that both label smoothing and distillation are exploiting the regularization property behind soft labels.

Original: Yuan et al.
Revised: Yuan et al.

Original: (2020) further investigated the regularization property of soft labels and then proposed a teacher free distillation scheme.
Revised: (2020) further investigated the regularization property of soft labels and then proposed a teacher free distillation scheme.

Original: Distillation loss.
Revised: Distillation loss.

Original: One of our main contributions is that we improve the distillation loss.
Revised: One of our main contributions is that we improve the distillation loss.

Original: For adaptively adjusting the distillation loss, TODO:
Revised: For adaptively adjusting the distillation loss, Tang et al.

Revised: (2019) pays attention to hard-to-learn and hard-tomimic samples, and the latter is weighted based on the prediction gap between teacher and student.

Revised: However, it does not consider that the teacher may give an incorrect guide to the student, under which the prediction gap is still large and such a method may lead to the performance being hurt.

Original: Zhou: add here.
Revised: Saputra et al. (2019) transfers teacher’s guidance only on the samples where the performance of the teacher surpasses the student, while Wen et al. (2019) deals with the incorrect guidance by probability shifting strategy. Our approach is different from the above methods, in terms of motivations as well as the proposed solutions.

Original: Bias–variance tradeoff.
Revised: Bias-variance tradeoff.

Original: Bias–variance tradeoff is a well-studied topic in machine learning (Kohavi & Wolpert, 1996; Domingos, 2000; Bishop, 2006) and for neural networks (Geman et al., 1992; Neal et al., 2018; Belkin et al., 2019; Yang et al., 2020).
Revised: Bias-variance tradeoff is a well-studied topic in machine learning (Kohavi & Wolpert, 1996; Domingos, 2000; Bishop, 2006) and for neural networks (Geman et al., 1992; Neal et al., 2018; Belkin et al., 2019; Yang et al., 2020).

Original: Existing methods are mainly concerned with the variance brought by the choice of network models.
Revised: Existing methods are mainly concerned with the variance brought by the choice of network models.

Original: Our perspective is different from the previous methods since we focus on the behavior of samples during training.
Revised: Our perspective is different from the previous methods since we focus on the behavior of samples during training.

Original: Besides, our main contribution is not to study how to tune the tradeoff, but how to avoid sample-wise tradeoff.
Revised: Besides, our main contribution is not to study how to theoretically analyze the tradeoff, but how to adaptively tune the sample-wise tradeoff during training.

Original:  B IAS – VARIANCE TRADEOFF FOR SOFT LABELS
Revised:  B IAS - VARIANCE TRADEOFF FOR SOFT LABELS

Original: [Figure]
Revised: [Figure]

Original: Soft labels play the role of ground-truth labels and regularizations at the same time.
Revised: Soft labels play the role of supervisory signals and regularizations at the same time, which inspires us to rethink soft labels from the perspective of the bias-variance tradeoff.

Original: We begin our analysis with some mathematical descriptions.
Revised: We begin our analysis with some mathematical descriptions.

Original: For a sample X labeled as i -th class, let the groundtruth label be a one-hot vector y where y i = 1 and other entries are 0.
Revised: For a sample x labeled as i -th class, let the ground-truth label be a one-hot vector y where y i = 1 and other entries are 0.

Original: Then for X and softmax output temperature τ , the soft label predicted by the teacher network is denoted as T ( X , τ ) and the output from the student is denoted as S ( X , τ ) .
Revised: Then for x and softmax output temperature τ , the soft label predicted by the teacher network is denoted as T ( x, τ ) and the output from the student is denoted as S ( x, τ ) .

Original: The soft label T ( X , τ ) is then used for training the student by the distillation 
Revised: The soft label T ( x, τ ) is then used for training the student by the distillation loss, i.e.

Original: [Figure] loss, i.e.
Revised: L kd = −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: 2 (cid:80) k ˆ y tk,τ log ˆ y sk,τ , where ˆ y sk,τ , ˆ y tk,τ means the k -th element of the student’s output

Original: [Equation] [Equation] [Equation] [Equation]
Revised: S ( x, τ ) and the teacher’s output T ( x, τ ) , respectively.

Original:  log 
Revised: With the above notations, the cross-entropy loss for training with one-hot labels is L ce = − y k log ˆ y sk, 1 .

Original: [Equation]
Revised: We now present the bias-variance decomposition for L ce and L kd , based on the deﬁnition and notations from Heskes (1998).

Original: y sk,τ , where KL means the KullbackLeibler divergence and ˆ y sk,τ , ˆ y tk,τ means the k -th element of the student’s output S ( X , τ ) and the teacher’s output T ( X , τ ) , respectively.
Revised: First, we denote the output distribution on sample x of a student network trained with distillation as ˆ y kd and without distillation as ˆ y ce . Then, let the averaged output of ˆ y kd and ˆ y ce be ¯ y kd and ¯ y ce , that is,

Original: Following statistical learning theories (Hastie et al., 2009), we assume that there exists a function f such that y =
Revised: [Equation] where Z ce , Z kd are two normalization constant. Then we have [Equation]

Original: f
Revised: where D KL is the Kullback-Leibler divergence. For direct training, the variance term is caused by different training sets for the task. For training with distillation, the variance E [ D KL (¯ y kd , ˆ y kd )] is caused by both different training sets and different teacher networks. In Fig.

Original: ( X ) + (cid:15) , where the noise (cid:15) has zero mean and variance σ 2 . 
Revised: 1, we illustrate the bias and variance of difference teacher networks. Although for KD there are more sources causing variance, the variance brought by L kd is not necessarily higher than L ce . In fact, existing empirical results strongly suggest that the overall variance is smaller with KD. For example, students trained with soft labels are better calibrated than one-hot baselines (M¨uller et al., 2019) and KD makes the predictions of students more consistent when facing adversarial noise (Papernot et al., 2016).

Original: Also, denote the model as ˆ f , then for sample X and the squared-error loss is 
Revised: Here, we present these empirical evidence as an assumption: Assumption 1 The variance brought by KD is smaller than direct training, that is, E

Original: [Equation]
Revised: [Equation]

Original: The term ( E [ ˆ f ( X ) − f ( X )]) 2 is known as bias error, while the term E [ ˆ f ( X ) − E
Revised: Similar to Eq.

Original: [ ˆ f ( X )]] 2 is the variance.
Revised: (2), we write the decomposition for L kd as [Equation]

Original: Intuitively, if soft labels  is used, the bias error during training will become larger since ˆ f no longer targets at learning the one-hot label. Meanwhile, the variance will become smaller because the output becomes smoothed.
Revised: An observation here is that ¯ y ce converges to one-hot labels while ¯ y kd converges to soft labels, so ¯ y ce is closer to the one-hot ground-truth distribution y than ¯ y kd , i.e., E y (cid:104)

Original: An illustration about the bias error and variance is shown in Fig.
Revised: [Equation]

Original: As the original label y only contains the ground-truth information, we can view y as the part for reducing bias error.
Revised: (cid:17)(cid:105) (cid:62) 0 .

Original: If we rewrite L kd as L kd =
Revised: If we rewrite L kd as L kd =

Original: L kd − L ce + L ce , thenthe cross-entropy loss L ce = −  y k log  ˆ y sk, 1 is for bias reduction.
Revised: L kd − L ce + L ce , then L kd − L ce causes that the bias increases by E y (cid:104) log (cid:16) ¯ y ce ¯ y kd (cid:17)(cid:105) and the variance decreases by E [ D KL (¯ y ce , ˆ y ce )] − E [ D KL (¯ y kd , ˆ y kd )] .

Original: Therefore, the variance reduction realized by the distillation comes from the difference of the  two losses, i.e., L kd − L ce .
Revised: From the above analysis, we separate L kd into two terms, and L kd − L ce leads to variance reduction, and L ce leads to bias reduction.

Original: In this section, we ﬁrst give an intuitive analysis of the variance reduction by visualizing L kd − L ce .
Revised: In the following sections, we ﬁrst analyze how L kd − L ce links to the bias-variance tradeoff during training.

Original: Then we analyze the changes in the relative importance between bias reduction and variance reduction during training with soft labels.
Revised: Then we analyze the changes in the relative importance between bias reduction and variance reduction during training with soft labels.

Original: Next, we conduct experiments to analyze the relative importance of bias reduction and variance reduction.

Original: 3. I NTUITIVE UNDERSTANDING OF BIAS - VARIANCE TRADEOFF 
Revised: 3. T HE BIAS - VARIANCE TRADEOFF DURING TRAINING

Original: First of all, we dive into intuitively understanding bias–variance tradeoff of soft labels.
Revised: It is known that bias reduction and variance reduction are often in conﬂict and we cannot minimize bias and variance together. However, if we consider the change of bias and variance during the

Original: Bias reduction plainly helps the model ﬁt the trainingdata, while variance reduction of soft labels is complicated.
Revised: [Table] training process, the importance of tuning the tradeoff also changes during training. Speciﬁcally, shortly after the training of the network starts, the bias error dominates the total error and the variance is less important.

Original: As analyzed above, the variance reduction is achieved bythe loss term L kd − L ce . In other words, the term L kd − L ce represents the regularization brought by training with soft labels.
Revised: As training goes on, gradients of reducing the bias error (induced by L ce ) and reducing the variance (induced by L kd − L ce ) can be of the same scale for some samples, then we need to balance the tradeoff because reducing one term is likely to increase another one.

Original:  To have an intuitive understanding of how the regularization works, we propose to visualize its gradient on the logits of the student network.
Revised: Therefore for soft labels, we need to handle the bias-variance tradeoff in a sample-wise manner and take the training process into consideration. To study the bias-variance tradeoff during training, we consider the gradients of bias and variance reduction.

Original: That is, let z be the logits output of the student  and z i is i -th element of it, then we are interested invisualizing ∂ ( L kd − L ce )
Revised: Let z be the logits output of the student on input x and z i is i -th element of it, then we are interested in ∂ ( L kd − L ce )

Original: ∂z i .
Revised: ∂z i .

Revised: For simplifying analysis, we are concerned with the gradients on the ground-truth related logit, that is, the sample x is labeled as i -th class.

Original: Mathematically we have
Revised: Mathematically, for the gradients of variance reduction, we have

Original: [Equation]
Revised: [Equation]

Original: where y t i,τ denotes the i -th element of the teacher’s prediction, i.e., T ( X , τ ) .
Revised: where y ti,τ denotes the i -th element of the teacher’s prediction, i.e., T ( x, τ ) .

Original: The term L kd − L ce is easy to understand when τ = 1 since the gradient now becomes y i − ˆ y ti, 1 .
Revised: The term L kd − L ce is easy to understand when τ = 1 since the gradient now becomes y i − ˆ y ti, 1 .

Original: As y ti,τ is the output from the teacher network and computed by a linear mapping of the activations in the teacher’s penultimate layer, the regularization indicates that the student need to follow the learnt the resemblances between classes (Hinton et al., 2015; M¨uller et al., 2019).

Original: Still, two questions are unclear: 1) what the resemblances are and 2) whether the regularization still indicates resemblances if τ is set higher than 1, a widely adopted hyperparameter (Tian et al., 2020).

Original: [Table] Towards answering the questions we now visualize the value of gradient vector ∂ ( L kd − L ce ) ∂ z concerning each class. Speciﬁcally, on ImageNet (Deng et al., 2009) training set with τ = 3 , we calculate the average value of ∂ ( L kd − L ce ) ∂
Revised: Meanwhile, for the bias reduction, we have ∂L ce ∂z i

Original: z for each class. Let M be the matrix of values with the ij -th entry M ij means averaged ∂ ( L kd − L ce ) ∂ z for class j .
Revised: = ˆ y si, 1 − y i , so ∂L ce ∂z i and ∂ ( L kd − L ce ) ∂z i always have different signs, leading to a tradeoff.

Original: Since (cid:80) iM ij = 0 , diagonal elements are ignored for visualization. The results are visualized in Fig. We ﬁnd that plotting the common correlation matrix heatmap is ambiguous, because the matrix to be visualized is of large size ( 1000 × 1000 )with a large variance. By treating each entry M ij as a vertex and then constructing a mesh for the matrix, we apply subdivision (Loop, 1987) to the mesh for smoothing the extreme points and ﬁnally rendering the mesh by ray-tracing package PlotOptiX. We can observe the several facts from the ﬁgures: 1) Comparing the sub-ﬁgure (a) and (b), we can see that for distillation resemblances implied by regularizers are similar across different teacher-student pairs.
Revised: If ∂L ce ∂z i is much higher than ∂ ( L kd − L ce )

Original: The resemblances are consistent with the semantic similarity of image class names. In a word, the regularization brought by soft labels implies resemblances among labels, which are consistent with the semantic distance of class names. In the next section, we will analyze how bias-variance tradeoff changes when training with soft labels. 3. S AMPLE - WISE BIAS - VARIANCE TRADEOFF
Revised: ∂z i , the bias reduction dominates the overall optimization direction.

Original: Now that we are aware of what the variance reduction brought by soft labels is, a question then arises: How soft labels affect the ﬁnal prediction error in terms of bias-variance tradeoff? We discover that the sample-wise bias-variance tradeoff affects the ﬁnal prediction error. Strictly speaking, if bias reduction and variance reduction lead to different gradient for the network weights, we need to take care of the bias-variance tradeoff.
Revised: Instead, if ∂ ( L kd − L ce ) ∂z i becomes higher, the sample is used for variance reduction.

Original: Also, it is known that bias reduction and variance reduction are often in conﬂict and we need to balance bias and variance. However, during the early phase of training, if we simply choose only bias reduction, both bias and variance can decrease since the model just begins to ﬁt into the data. As traininggoes on, some samples start facing the bias-variance tradeoff if bias and variance cannot decrease at the same time.
Revised: Interestingly, we discover that under a ﬁxed distillation temperature, the ﬁnal performance is worse when more training samples are used for variance reduction, which will be introduced in the next section.

Original: In other words, training with soft labels requires us to handle sample-wise bias-variance tradeoff. To simplify our analysis, we consider an important tradeoff case that bias reduction and variance reduction come into conﬂict in the direction of optimizing the ground-truth related logit. More precisely, for a sample labeled as i -th class, if ∂L ce ∂z i and ∂ ( L kd − L ce ) ∂z i have a different sign, the bias reduction and variance reduction come into conﬂict and bias-variance dilemma arises. We discover that the ﬁnal prediction error is higher when more training samples are facing the bias-variance tradeoff.
Revised: 3. R EGULARIZATION SAMPLES

Original: Our analysis starts with a conclusion from M¨uller et al.
Revised: Our analysis starts with a conclusion from M¨uller et al.

Original: (2019): if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective.
Revised: (2019): if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective.

Original: Inspired by the[Figure] phenomenon, we calculate the number of conﬂicting examples with different  settings.
Revised: Inspired by the phenomenon, we gather the impact of bias and variance during training with different distillation settings.

Original: That is, let a = ∂L ce ∂z i and b = ∂ ( L kd − L ce )
Revised: Let a = ∂L ce ∂z i and b = ∂ ( L kd − L ce )

Original: ∂z i , then  we are interested in the number of instances with sign( ab ) =
Revised: ∂z i , then as introduced before, we use a and b to represent the impact of bias and variance, respectively.

Original: − 1 , when distilling from the teacher with or without label smoothing.
Revised: If we have b > a for a sample, we name the sample as a regularization sample since the variance dominates the optimization direction. From the collected data, we ﬁnd that the number of regularization samples is closely related to distillation performance.

Original: In Tab. 1, we present the count of  samplesfacing bias-variance tradeoff for a student network trained by knowledge distillation.
Revised: In Tab. 1, we present the count of regularization samples for a student network trained by knowledge distillation.

Original: For distillation with a temperature higher than 1, which is the common setting, we observe that if the teacher network is trained with label smoothing, more samples will face the bias-variance tradeoff.
Revised: For distillation with a temperature higher than 1, which is the common setting, we observe that if the teacher network is trained with label smoothing, more samples will be involved in [Figure] [Figure] variance reduction.

Original: Also, distillation from a teacher trained with label smoothing performs worse, which is consistent with M ¨ uller et al.
Revised: Also, distillation from a teacher trained with label smoothing performs worse, which is consistent with M ¨ uller et al.

Original: Considering that the performance of the student is negatively associated withthe number of conﬂicting samples, our answer to the question asked at the beginning of this section is: For soft labels,  samples facing bias-variance tradeoff affect the ﬁnal distillation performance.
Revised: Therefore, we conclude that for distillation with soft labels, the regularization samples during training affect the ﬁnal distillation performance.

Original: Moreover, we plot the number of  samplesfacing bias-variance tradeoff with respect to different training epochs in Fig.
Revised: Moreover, we plot the number of regularization samples with respect to different training epochs in Fig.

Original: As demonstrated in the plots, the number of such samples increases much faster when using the teacher trained with label smoothing for distillation.
Revised: As demonstrated in the plots, the number of such samples increases much faster when using the teacher trained with label smoothing for distillation.

Original: The gap of the number between with and without  smoothing becomes larger for more training epochs.
Revised: For regularization samples, the gap of their number between with and without label smoothing becomes larger for more training epochs.

Original: These observations verify our motivation that  bias-variance tradeoffs are sample-wise and evolve during the training process.
Revised: These observations verify our motivation that the bias-variance tradeoff varies sample-wisely and evolves during the training process.

Original: From the above results, we conclude that bias-variance tradeoff for soft labels is sample-wise, which makes tuning the tradeoff harder than those loss based bias-variance tradeoffs.
Revised: From the above results [Figure] we conclude that bias-variance tradeoff for soft labels varies sample-wisely, therefore the strategy for tuning the tradeoff should also be sample-wise.

Original: In the next section, we investigate whether we can artiﬁcially tune the soft labels such that bias reduction does not conﬂict with variance reduction.
Revised: In the next section, we set up ad hoc ﬁlters for soft labels and further investigate how regularization samples affect distillation.

Original: 3. S AMPLES FACING BIAS - VARIANCE TRADEOFF ARE USEFUL
Revised: 3. H OW REGULARIZATION SAMPLES AFFECT DISTILLATION

Original: The results presented in the last section suggest that we should avoid training with samples facing bias-variance tradeoff.
Revised: The results presented in the last section suggest that we should avoid training with regularization samples.

Original: Hence, we design two straightforward solutions and then ﬁnd that training with samples facing bias-variance tradeoff is beneﬁcial.
Revised: Hence, we design two straightforward solutions and then ﬁnd that totally ﬁltering out regularization samples deteriorates the distillation performance.

Original: The ﬁrst experiment we conduct is to manually resolve the conﬂicting gradient on the label related logit, as deﬁned in section 3.2.
Revised: The ﬁrst experiment we conduct is to manually resolve the conﬂicting gradient on the label related logit, as deﬁned in section 3.2.

Original: Speciﬁcally, we apply a mask to the distillation loss L kd such that ∂L kd ∂z i = [-
 if y-] i = 1 .
Revised: Speciﬁcally, we apply a mask to the distillation loss L kd such that ∂L kd ∂z i = [-
 if y-]0 where i is the label.

Original: Consequently, the loss for this sample now becomes L ∗ kd 
Revised: Consequently, the loss for this sample now becomes L ∗ kd =

Original: = (cid:80) k (cid:54) =
Revised: [Equation]

Original: i ˆ y t k,τ log ˆ y s k,τ .
Revised: ˆ y sk,τ .

Original: For knowledge transfer, the motivation behind the mask distillation loss is that  only knowledge contained in the soft part, which reﬂects resemblances, is distilled from the teacher network.
Revised: The motivation behind the masked distillation loss is that we only transfer the knowledge of resemblances among the labels.

Original: Another experiment is to ﬁgure out what role in distillation those conﬂicting samples will play.
Revised: Another experiment is to ﬁgure out what role in distillation those regularization samples will play.

Original: To investigate this, we carry out knowledge distillation on two subsets of samples: 1) L kd is  valid when sign( ab ) = − 1 and 2) L kd is validwhen sign( ab ) = 1 . In other words, knowledge distillation is performed on only conﬂicting samples or only not conﬂicting samples.
Revised: To investigate this, we carry out knowledge distillation on two subsets of samples: 1) L kd is not valid on regularization samples, and 2) L kd is valid only on regularization samples.

Original: [Figure] [Table] The results of the two experiments are presented in Tab. 2.
Revised: The results of the two experiments are presented in Tab. 2.

Original: We can observe that all of the three approaches are not as good as the baseline knowledge distillation performance, but are better than the direct training baseline.
Revised: We can observe that all of the three approaches are not as good as the baseline knowledge distillation performance, but are better than the direct training baseline.

Original: First, since masking L kd loss result in worse performance, sign( ab ) is just an indicator, rather than a sufﬁcient condition
Revised: First, since masking L kd loss on the label related logit results in worse performance compared to standard KD, we cannot resolve the tradeoff by applying a mask on the ground truth related logit. Then, from the second experiment, we can see that ﬁltering out regulariza- [Table]

Original: [Table]
Revised: [Table]

Original: for judging conﬂicting samples.
Revised: tion samples deteriorates the distillation performance.

Original: So we cannot directly resolve the tradeoff unless a sufﬁcient condition is theoretically derived. Moreover,from the other two experiments, we see that performing on the subsets are less effective. Thus, the conﬂicting samples are still valuable for distillation. 
Revised: Moreover, the result of the third experiment is higher than the direct training baseline, indicating that regularization samples are still valuable for distillation. The above results motivate us to think that regularization samples are not fully exploited by standard KD and we can tune the tradeoff to fulﬁll the potential of regularization samples.

Original:  W EIGHTED SOFT LABELS
Revised:  W EIGHTED SOFT LABELS

Revised: From the last section

Revised: [Table]

Original: From the last section, we realize that there exist sample-wise bias-variance tradeoffs and  the number of these samples is negatively associated with the ﬁnal distillation performance.
Revised: we realize that the bias-variance tradeoff varies sample-wisely during training and under ﬁxed distillation settings [Table] the number of regularization samples is negatively associated with the ﬁnal distillation performance.

Original: Yet, such samples are still useful for distillation.
Revised: Yet [Table] discarding regularization samples deteriorates distillation performance and distilling knowledge from these samples is better than the direct training baseline.

Original: The above evidence inspires us to lower the weight of the conﬂicting samples.
Revised: The above evidence inspires us to lower the weight of regularization samples

Original: However, the metric sign( ab ) used previously is binary, which brings difﬁculty to adaptively adjusting the weight. Recall that sign( ab ) is designed to see if the distillation loss is in conﬂict with the cross-entropy loss, we now need a soft metric achieving the same goal.
Revised: [Table] Recall that regularization samples are deﬁned by the relative value of a and b , we propose to assign importance weight to a sample according to a and b . However, since L kd is computed with the hyperparameter temperature, a and b are correlated with the temperature and thus bring difﬁculty to tuning the hyperparameter.

Original: Speciﬁcally, we use L t ce L [Equation]
Revised: To make the weighting scheme independent of the temperature hyperparameter, we compare a and b with temperature τ = 1 .

Original: ce =log ˆ y si, 1 log ˆ y ti, 1 as a substitution to sign( ab ) .
Revised: Note that when τ = 1 , a = ˆ y si, 1 − y i and b =

Original: In this way, if a student network is poorly trained on this sample and produces a large ratio, it is likely that this sample is not facing the bias-variance tradeoff, thus we can increase the weight of this sample. Otherwise, if the student performs better than the teacher on this sample, resulting in a small ratio, this sample will be assigned a small weight.
Revised: y i − ˆ y ti, 1 , so we compare ˆ y si, 1 and ˆ y ti, 1 instead.

Original: In the light of previous works that assign sample-wise weights (Lin et al., 2017) TODO: more paper? , we propose weighted soft labels for knowledge distillation, which is formally deﬁned as
Revised: Finally, in the light of previous works that assign sample-wise weights (Lin et al., 2017; Tang et al., 2019), we propose weighted soft labels for knowledge distillation, which is formally deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where i is the ground truth class of the sample.
Revised: where i is the ground truth class of the sample.

Original: The above equation means that a weight is assigned to each sample’s L kd .
Revised: The above equation means that a weighting factor is assigned to each sample’s L kd according to the predictions of the teacher and the student.

Revised: In this way, if compared to the teacher, a student network is relatively better trained on a sample, we have ˆ y si, 1 > ˆ y ti, 1 , then a smaller weight is assigned to this sample.

Original: In Fig.
Revised: In Fig.

Original: 4, the whole computational graph of knowledge distillation with the proposed weighted soft labels is demonstrated.
Revised: 3, the whole computational graph of knowledge distillation with the proposed weighted soft labels is demonstrated.

Original: Finally, we add L wsl and L ce together to supervise the network, i.e., L total =
Revised: Finally, we add L wsl and L ce together to supervise the network, i.e., L total =

Original: L ce + L wsl .
Revised: L ce + αL wsl , where α is a balancing hyperparameter.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate our weighted soft labels comprehensively, we ﬁrst conduct experiments with various teacher-student pair settings on CIFAR-100 (Krizhevsky et al., 2009).
Revised: To evaluate our weighted soft labels comprehensively, we ﬁrst conduct experiments with various teacher-student pair settings on CIFAR-100 (Krizhevsky et al., 2009).

Original: Next, we compare our method with current state-of-the-art distillation methods on ImageNet (Deng et al., 2009).
Revised: Next, we compare our method with current state-of-the-art distillation methods on ImageNet (Deng et al., 2009).

Original: To validate the effectiveness of our method in terms of handling  bias-variance tradeoffs, we conduct ablation experiments by applying weighted soft labels on different subsets.
Revised: To validate the effectiveness of our method in terms of handling the bias-variance tradeoff, we conduct ablation experiments by applying weighted soft labels on different subsets.

Original: Finally, we analyze the impact of hyperparameter deﬁned for the weighted soft labels in Eq.

Original: 5. D ATASET AND HYPERPARAMETER SETTINGS
Revised: 5. D ATASET AND HYPERPARAMETER SETTINGS

Original: The datasets used in our experiments are CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).
Revised: The datasets used in our experiments are CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).

Original: CIFAR-100 contains 50K training and 10K test images of size 32 × 32 . 
Revised: CIFAR-100 contains 50K training and 10K test images of size 32 × 32 . ImageNet [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: ImageNet contains 1.2 million training and 50K validation images.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: contains 1.2 million training and 50K validation images.

Original: Except the loss function, training settings like learning rate or training epochs are the same with Tian et al.
Revised: Except the loss function, training settings like learning rate or training epochs are the same with Tian et al.

Original: (2020) for CIFAR-100 and Heo et al.
Revised: (2020) for CIFAR-100 and Heo et al.

Original: for ImageNet.
Revised: for ImageNet.

Original: For distillation, we set the temperature τ =  2 . For loss function, we set
Revised: For distillation, we set the temperature τ = 4 for CIFAR and τ = 2 for

Original: TODO:
Revised: ImageNet.

Original:  α = 2 , β
Revised: For loss function, we set α = 2 .

Original: = 1 for distillation on CIFAR and TODO:
Revised: 25 for distillation on CIFAR and α = 2 .

Original: α = 2 , β = 1 for ImageNet via grid search.
Revised: 5 for ImageNet via grid search.

Revised: The teacher network is well-trained previously and ﬁxed during training.

Original: For comparison, the following recent state-of-the-art methods are chosen:
Revised: For comparison, the following recent state-of-the-art methods are chosen:

Original: FitNet (Romero et al., 2015), AT (Zagoruyko & Komodakis, 2017), SP (Tung & Mori, 2019), CC (Peng et al., 2019), VID (Ahn et al., 2019), RKD (Park et al., 2019), PKT (Passalis & Tefas, 2018), AB (Heo et al., 2019), FT (Kim et al., 2018), FSP (Yim et al., 2017), NST (Huang & Wang, 2017).
Revised: FitNet (Romero et al., 2015), AT (Zagoruyko & Komodakis, 2017), SP (Tung & Mori, 2019), CC (Peng et al., 2019), VID (Ahn et al., 2019), RKD (Park et al., 2019), PKT (Passalis & Tefas, 2018), AB (Heo et al., 2019), FT (Kim et al., 2018), FSP (Yim et al., 2017), NST (Huang & Wang, 2017), Overhaul (Heo et al., 2019) and CRD (Tian et al., 2020).

Original: 5. M ODEL COMPRESSION 
Revised: 5. M ODEL COMPRESSION Results on CIFAR

Original: Results on CIFAR-100 In Tab. 3, we present the Top-1 classiﬁcation accuracy of our method and comparison methods.
Revised: In Tab. 3, we present the Top-1 classiﬁcation accuracy of our method and comparison methods.

Original: The results of comparison methods are quoted from Tian et al.
Revised: The results of comparison methods are quoted from Tian et al.

Original: Teacher-student pairs of the same and different architecture styles are considered, which again follows the setting in Tian et al.
Revised: Teacher-student pairs of the same and different architecture styles are considered.

Original: For pairs of same architecture style, we use wide residual networks (Zagoruyko & Komodakis, 2017) and residual networks (He et al., 2016).
Revised: For pairs of same architecture style, we use wide residual networks (Zagoruyko & Komodakis, 2017) and residual networks (He et al., 2016).

Original: For pairs of different architecture style, residual networks and ShufﬂeNet (Zhang et al., 2018) pairs are chosen for experiments.
Revised: For pairs of different architecture style, residual networks and ShufﬂeNet (Zhang et al., 2018) pairs are chosen for experiments.

Original: As shown in the table, for distillation with both same and different architecture style, our method reached new state-of-the-art results.
Revised: As shown in the table, for distillation with both same and different architecture style, our method reached new state-of-the-art results.

Revised: Speciﬁcally, our method outperforms standard KD by a large margin, which veriﬁes the effectiveness of our method.

Original: Results on ImageNet In Tab. 4, we compare our method with current SOTA methods on ImageNet.
Revised: Results on ImageNet In Tab. 4, we compare our method with current SOTA methods on ImageNet.

Original: Note that for the ResNet34 → ResNet-18 distillation setting, the result of CRD+KD is trained 10 more extra epochs while ours is the same as other methods.
Revised: Note that for the ResNet34 → ResNet-18 distillation setting, the result of CRD is trained 10 more extra epochs while ours is the same as other methods.

Revised: For ResNet-50 → MobileNet-v distillation setting, NST and FSP are not chosen for comparison as the two methods require too large GPU memories, so we include the accuracy of FT and AB reported in Heo et al.

Original:  Our results outperform all the existing methods, verifying the practical value of our method.
Revised: for comparison. Our results outperform all the existing methods, verifying the practical value of our method.

Original: 5. A BLATION STUDIES
Revised: 5. A BLATION STUDIES

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Weighted soft labels on different subsets.
Revised: Weighted soft labels on different subsets.

Original: Recall that we propose weighted soft labels for sample-wise biasvariance tradeoff, it is still unclear whether the improvements come from a well-handled sample-wise biasvariance tradeoff.
Revised: Recall that we propose weighted soft labels for tuning samplewise bias-variance tradeoff, it is still unclear whether the improvements come from a well-handled samplewise bias-variance tradeoff.

Original: To investigate this issue, we compare the performance gain of weighted soft labels on different training subsets.
Revised: To investigate this issue, we compare the performance gain of weighted soft labels on different training subsets.

Original: Similar to the settings used in Tab. 2, we apply weighted soft labels on two different subsets:  the  samplesfacing tradeoff and the samples not facing tradeoff.
Revised: Similar to the settings used in Tab. 2, we apply weighted soft labels on two different subsets: only the regularization samples and excluding regularization samples.

Original: In Tab. 5,  the conﬂicting subset means the subset of  samples facing tradeoff, which is judged by the metric sign( ab ) =
Revised: In Tab. 5, we show the results on subsets of only regularization samples and excluding regularization samples.

Original: − 1 .
Revised: From the signiﬁcant improvements, we can see that our method can not only improve performance on the RS subset, the improvements on excluding RS subset is also signiﬁcant.

Original: From the signiﬁcant improvements, we conclude that weighted soft labels can tackle the samplewise bias-variance tradeoff. trained with label smoothing (denoted as LS? ). [Table]
Revised: We conclude that weighted soft labels can tune sample-wise bias-variance tradeoff globally and lead to an improved distillation performance.

Original: [Table]
Revised: [Table]

Original: Distillation with label smoothing trained teacher.
Revised: Distillation with label smoothing trained teacher.

Original: Our exploration of bias-variance tradeoff starts with the conclusion made in M¨uller et al.
Revised: Our exploration of bias-variance tradeoff starts with the conclusion made in M¨uller et al.

Original: (2019): a teacher network trained with the label smoothing trick is less effective for distillation.
Revised: (2019): a teacher network trained with the label smoothing trick is less effective for distillation.

Original: It is worthwhile to study whether the[Table] [Table] conclusion remains true for distillation with our weighted soft labels.
Revised: It is worthwhile to study whether the conclusion remains true for distillation with our weighted soft labels.

Original: As discussed before, we hold the opinion that more samplesfacing tradeoff make the distillation less effective.
Revised: As discussed before, we hold the opinion that too many regularization samples make the distillation less effective.

Original: Since our weighted soft label is proposed to mitigate the negative effects of the tradeoff, we conduct comparison experiments in Tab. 6 to see if the negative effects still exist.
Revised: Since our weighted soft label is proposed to mitigate the negative effects of the regularization samples, with the same settings from Tab. 1, we conduct comparison experiments in Tab. 6 to see if the negative effects still exist.

Original: It is evident that weighted soft labels signiﬁcantly improve the distillation performance, especially for distillation from the teacher trained with label smoothing.
Revised: It is evident that weighted soft labels signiﬁcantly improve the distillation performance, especially for distillation from the teacher trained with label smoothing.

Original: Although the performance gap between with and without label smoothing is considerably narrowed, using the teacher trained with label smoothing  performs worse than that without label smoothing, which again verifying the conclusion drawn by M¨uller et al.
Revised: Besides, using the teacher trained with label smoothing still performs worse than that without label smoothing, which again veriﬁes the conclusion drawn by M¨uller et al.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Recent studies (M¨uller et al., 2019; Yuan et al., 2020) point out that one important reason behind the effectiveness of distillation is the regularization effect brought by being soft.
Revised: Recent studies (M¨uller et al., 2019; Yuan et al., 2020) point out that one important reason behind the effectiveness of distillation is the regularization effect brought by being soft.

Original: In this paper, we rethink the soft labels for distillation from a bias-variance tradeoff perspective.
Revised: In this paper, we rethink the soft labels for distillation from a bias-variance tradeoff perspective.

Revised: The tradeoff varies sample-wisely and we propose weighted soft labels to handle the tradeoff, of which the effectiveness is veriﬁed with experiments on standard evaluation benchmarks.

Revised: A A PPENDIX A. V ISUALIZATION OF THE RESEMBLANCES INTRODUCED BY SOFT LABEL REGULARIZER

Revised: In Sec 3.

Original: The tradeoff exists sample-wise
Revised: we propose to study ∂ ( L kd − L ce ) ∂z i during the training process. When τ = 1 , we show that ∂ ( L kd − L ce ) ∂z i equals y i −

Revised: ˆ y ti, 1 .

Revised: As y ti,τ is the output from the teacher network and computed by a linear mapping of the activations in the teacher’s penultimate layer, the regularization indicates that the student should follow the learned the resemblances between classes (Hinton et al., 2015;

Revised: M ¨ uller et al., 2019).

Revised: Still, two questions are unclear: 1) what the resemblances are and 2) whether the regularization still indicates resemblances if τ is set to 4, a widely adopted hyperparameter (Tian et al., 2020).

Revised: Towards answering the questions, we visualize the value of gradient vector ∂ ( L kd − L ce )

Revised: ∂

Revised: z concerning each class.

Revised: Speciﬁcally, on ImageNet (Deng et al., 2009) training set and τ = 4 , we calculate the average value of ∂ ( L kd − L ce ) ∂

Revised: z for each class.

Original: and we propose weighted soft labels to handle the tradeoff
Revised: Let M be the matrix of values with the ij -th entry M ij means averaged ∂ ( L kd − L ce ) ∂z i for class j . Since (cid:80) i M ij = 0 , diagonal elements are ignored for visualization.

Revised: The results are visualized in Fig.

Revised: We ﬁnd that plotting the common correlation matrix heatmap is ambiguous, because the matrix to be visualized is of large size ( 1000 × 1000 ) with a large variance.

Revised: By treating each entry M ij as a vertex and then constructing a mesh for the matrix, we apply subdivision (Loop, 1987) to the mesh for smoothing the extreme points and ﬁnally rendering the mesh by ray-tracing package PlotOptiX. We can observe the several facts from the ﬁgures: 1) Comparing the sub-ﬁgure (a) and (b), we can see that for distillation resemblances implied by regularizers are similar across different teacher-student pairs.

Revised: 2) Comparing (ab) with (cd), we can see that the resemblances are consistent with the semantic similarity of image class names.

Revised: In a word, for τ = 4 , the variance reduction brought by soft labels still implies resemblances among labels, which are consistent with the semantic distance of class names.

Revised: In the next section, we will analyze how bias-variance tradeoff changes when training with soft labels.

Revised: [Figure]