B ETTER S UPERVISORY S IGNALS BY O BSERVING L EARNING P ATHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Better supervision might lead to better performance.
In this paper, we ﬁrst clarify what makes supervision good for a classiﬁcation problem, and then explain two existing label reﬁning methods: label smoothing and knowledge distillation, in terms of our proposed criterion.
To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the network’s predicted distribution, for each training sample during training.
We surprisingly ﬁnd that the model can spontaneously reﬁne “bad” labels with through a “zig-zag” learning path, which occurs on both toy and real datasets.
Observing the learning path not only provides a new perspective for understanding knowledge distillation, overﬁtting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks.
Inspired by this, we propose a new algorithm, Filter-KD, which further enhances classiﬁcation performance of knowledge distillation, as veriﬁed by experiments in various settings.
I NTRODUCTION
In multi-class classiﬁcation problems, we usually supervise our model with “one-hot” labels: label vectors y which have y i = 1 for one i , and 0 for all other dimensions.
Over time, however, it has gradually become clear that this “default” setup is not always the best choice in practice, as measured by the generalization performance of the resulting model.
1 One alternative is to summarize a distribution of human annotations, as Peterson et al. (2019) did on CIFAR-10.
An alternative approach is label smoothing (e.g. Szegedy et al., 2016), mixing between a one-hot label and the uniform distribution.
Knowledge distillation (KD), ﬁrst training a teacher network on the training set and then a student network on the teacher’s output probabilities, was originally proposed for model compression (Hinton et al., 2015) but can also be thought of as reﬁning the supervision signal: it provides “soft” teacher outputs rather than hard labels to the student.
Knowledge distillation is promising because it requires no additional annotation effort, but, unlike label smoothing, can still provide sample-speciﬁc reﬁnement.
Perhaps surprisingly, knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student ; this is known as self-distillation (Furlanello et al., 2018; Zhang et al., 2019).
There have been many recent attempts to explain the success of knowledge distillation and speciﬁcally self-distillation (e.g. Menon et al., 2021; Allen-Zhu & Li, 2020; Tang et al., 2020), from both optimization and supervision perspectives. We focus on the latter area, where it is usually claimed that the teacher provides useful “dark knowledge” to the student through its labels.
Inspired by this line of work, we further explore why and how this improved supervisory signal emerges during the teacher’s one-hot training.
Speciﬁcally, we ﬁrst clarify that given any input sample x , good supervision signals should be close (in L 2 distance) to the ground truth categorical distribution, i.e., p ∗ ( y | x ) .
We then show that a neural network (NN) can automatically reﬁne “bad labels”, those where p ∗ ( y | x ) is far from the one-hot vector from the training set.
During one-hot training, the model prediction on such a sample ﬁrst moves towards p ∗ ( y | x ) , and then slowly converges to its supervisory label, following a “zig-zag” pattern.
A well-trained teacher, one that does not overﬁt to particular training labels, can thus provide supervisory signals closer to p ∗ ( y | x ) .
We ﬁnd this phenomenon is quite common in gradient descent training, and provide some analytical justiﬁcation (Section 3.3).
Based on our explanations, we point out that this signal can be better achieved if we take a moving average of the teacher’s prediction, an algorithm we term Filter-KD. This approach provides better supervision and hence better ﬁnal performance, especially when there are more bad labels.
S UPERVISION I NFLUENCES G ENERALIZATION
We will begin by establishing the classiﬁcation setting, and making clearer how the choice of supervisory signal affects the learned model.
2. C HOICES OF S UPERVISION S IGNAL
In K -way classiﬁcation, our goal is to learn a mapping f : X → ∆ K that can minimize the risk
[Equation]
where L ( y, f ( x )) is the loss function measuring the gap between model’s prediction and the true label (e.g. cross-entropy or square loss).
The true label y is an integer ranging from 1 to K , and the input signal x is usually high dimensional, e.g., an image or a sequence of word embeddings.
The joint distribution of ( x , y ) is P , whose density 3 can be written as p ( x ) p
( y | x ) .
In practice, as P is unknown, we instead (approximately) minimize the empirical risk
[Equation]
where 1 ( y n = k ) is an indicator function which equals 1 if y n = k or 0 otherwise, and e y n ∈ { 0 , 1 } K is its one-hot vector form.
L ( f ( x n ))
=
( L (1 , f ( x n )) , . . .
, L ( K, f ( x n )))
∈ R K is the loss for each possible label.
In R emp , the N training pairs D (cid:44) { ( x n , y n ) }
Nn =1 are sampled i.i.d.
from P .
Comparing (2) to (1), we ﬁnd that p ( x ) is approximated by an uniform distribution over the samples, which is reasonable.
However, using an indicator function (i.e., one-hot distribution) to approximate p ( y | x ) bears more consideration.
For example, if a data point x is quite vague and its true p ( y | x ) is ﬂat or multimodal, we might hope to see x multiple times with different label y during training.
But actually, most datasets have only one copy of each x , so we only ever see one corresponding e y .
Although R emp is an unbiased estimator for R , if we used a better (e.g. lower-variance) estimate of p ( y | x ) , we could get a better estimate for R and thus, hopefully, better generalization.
Speciﬁcally, suppose we were provided a “target” distribution p tar ( y | x ) (written in vector form as p tar ( x ) ) for each training point x , as D (cid:48)
= { ( x n , p tar ( x n ))
} Nn =1 .
Then we could use
R tar ( f, D (cid:48) )
(cid:44)
[Equation]
Standard training with R emp is a special case of R tar , using p tar ( x n ) =
e y n .
The CIFAR-10H dataset (Peterson et al., 2019) is one attempt at a different p tar , using multiple human annotators to estimate p tar .
Label smoothing (e.g. Szegedy et al., 2016) sets p tar to a convex combination of e y and the constant vector 1 K 1 .
In knowledge distillation (KD; Hinton et al., 2015), a teacher is ﬁrst trained on D , then a student learns from D (cid:48) with p tar based on the teacher’s outputs.
All three approaches yield improvements over standard training with R emp .
[Figure]
M EASURING THE Q UALITY OF S UPERVISION
Choosing a different p tar , then, can lead to a better ﬁnal model.
Can we characterize which p tar will do well?
We propose the following, as a general trend.
Hypothesis 1.
Suppose we train a model supervised by p tar , that is, we minimize R tar ( f, D (cid:48) ) .
Then, smaller average L 2 distance between p tar and the ground truth p ∗ on these samples, i.e. small E x [ (cid:107) p tar ( x ) − p ∗ ( x ) (cid:107) 2 ] , will in general lead to better generalization performance.
This hypothesis is suggested by Proposition 3 of Menon et al. (2021), which shows that for any predictor f and loss bounded as L ( y, ˆ y ) ≤ (cid:96) ,
[Equation]
[Equation] When N is large, the second term will dominate the right-hand side, implying smaller average (cid:107) p tar − p ∗ (cid:107) will lead to R tar being a better approximation of the true risk R . Minimizing a better
R tar should then lead to a better model, suggesting that the quality of the supervision signal can be roughly measured by its L 2 distance to the ground truth p ∗ .
To further support this hypothesis, we conduct experiments on a synthetic Gaussian problem (Figure 1 (a); details in Appendix A), where we can easily calculate p ∗ ( y | x ) for each sample.
We ﬁrst generate several different p tar by adding noise 4 to the ground truth p ∗ , then train simple 3-layer NNs under that supervision.
We also show ﬁve baselines: one-hot training (OHT), label smoothing (LS), KD, early-stopped KD (ESKD), and ground truth (GT) supervision (using p ∗ ).
KD refers to a teacher trained to convergence, while ESKD uses a teacher stopped early based on validation accuracy.
We early-stop the student’s training in all settings.
From Figure 1 (b-c), it is clear that smaller (cid:107) p tar − p ∗ (cid:107) 2 leads to better generalization performance, as measured either by accuracy (ACC) or expected calibration error (ECE) 5 on a held-out test set. Appendix A has more detailed results.
I NSIGHTS F ROM THE L EARNING P ATH
In the toy example of Section 2, we see that ESKD outperforms other baselines in accuracy by a substantial margin (and all baselines are roughly tied in ECE).
We expect that supervision with smaller (cid:107) p tar − p ∗ (cid:107) 2 leads to better generalization performance, but it is not clear how better p tar emerges from when the teacher in ESKD is trained using one-hot labels.
This section will answer this, by observing the learning paths of training samples.
3. P AY M ORE A TTENTION TO H ARDER S AMPLES
For a ﬁner-grained understanding of early stopping the teacher, we would like to better understand how the teacher’s predictions evolve in training.
Assuming Hypothesis 1, the main factor is E x (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 , where q is the teacher’s output probability distribution.
We expect, though, that this term will vary for different x , in part because some samples are simply more difﬁcult to learn; as a proxy for this, we deﬁne base difﬁculty as (cid:107) e y − p ∗ ( x ) (cid:107) 2 . This measure is high if: • x is ambiguous: p ∗ has several large components, so is
one-hot label near p ∗ . • x is not very ambiguous (there is a one-hot label near p ∗ ), but the sample was “unlucky” and drew y from a low-probability class.
Figure 2 shows these two quantities at three points in training: initialization, the point where ESKD’s teacher stops, and convergence.
At initialization, most points 6 have large (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 .
By the point of early stopping, most q ( x ) values are roughly near p ∗ .
At convergence, however, q ( x ) ≈ e y , as the classiﬁer has nearly memorized the training inputs, leading to a diagonal line in the plot.
It is clear the biggest contributors to E x (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 when training a model to convergence are the points with high base difﬁculty.
Per Hypothesis 1, these are the points we should most focus on.
3. L EARNING P ATH OF D IFFERENT S AMPLES
To better understand how q changes for each sample, we track the model’s outputs of all training samples at each training step.
We plot the vectors of three probabilities as points on the simplex (details in Appendix E), and visualize four samples with different base difﬁculty in Figure 3.
The easy samples very quickly move to the correct location near e y (as indicated by the light color until reaching the corner).
The medium sample takes a less direct route, drifting off slightly towards p ∗ , but still directly approaches e y .
The hard sample, however, does something very different: it ﬁrst approaches p ∗ , but then veers off towards e y , giving a “zig-zag” path.
In both the medium and hard cases, there seems to be some “unknown force” dragging the learning path towards p ∗ ; in both cases, the early stopping point is not quite perfect, but is noticeably closer to p ∗ than the ﬁnal converged point near e y .
In other words, during one-hot training, the NNs can spontaneously reﬁne the “bad labels.”
Under Hypothesis 1, this partly explains the superior performance of ESKD to KD with a converged teacher.
We only show four points, because they are quite representative of all samples, under different toydataset settings.
We also provide a “zig-zagness score” to further illustrate the correlation between base difﬁculty and the learning path patterns (refer to Appendix C for more details).
3. E XPLANATION OF PATTERNS IN THE LEARNING PATH
Where does this “unknown force” pushing q towards p ∗ come from?
The following decomposition, proved in Appendix G, will help elucidate its source:
Proposition 1.
Let z t ( x ) (cid:44) f
( w t , x ) denote the NN output logits with parameters w t , and q t ( x ) =
Softmax( z t ( x ))
the corresponding probabilities.
Let w t +1 (cid:44) w t − η ∇ w (cid:0) p tar ( x u ) T L ( q t ( x u )) (cid:1) be the result of applying one step of SGD to w t on the data point ( x u , p tar ( x u )) with learning rate η .
Then, the change in predictions for a particular observed sample, x o , is
[Equation]
where A t ( x o )
= ∇ z q t ( x o ) and K t ( x o , x u ) =
( ∇ w z ( x o ) | w t ) ( ∇ w z ( x u ) | w t ) T are K × K matrices.
The matrix K t is known as the empirical neural tangent kernel , which we can think of roughly as a notion of “similarity” between x o and x u based on the network’s representation at time t . In general, K t changes during training in potentially complex ways, but in very wide networks with appropriate random initialization, it is nearly invariant throughout training and nearly independent of the initialization (Jacot et al., 2018; Arora et al., 2019).
For more details, see Appendix G.
Figure 4 shows the learning path of a hard sample during one-hot training, say x o , where the label e y o is far from p ∗ ( x o ) .
In each epoch, w will receive N − 1 updates based on the loss of x u (cid:54) =
x o (the small blue path), and one update based on x o (the big red path).
At any time t , “dissimilar” samples will have small K t ( x o , x u ) (as measured, e.g., by its trace), and hence only slightly affect the predicted label for x o .
Similar samples will have large K t ( x o , x u ) , and hence affect its updates much more; because p ∗ is hopefully similar for similar x values, it is reasonable to expect that the mean of e y for data points with similar x will be close to p ∗ ( x o ) .
Thus the net effect of updates for x u (cid:54) =
x o should be to drag q ( x o ) towards p ∗ ( x o ) .
This is the “unknown force” we observed earlier.
The other force affecting q ( x o ) during training is, of course, the update based on x o , driving q ( x o ) towards e y o .
How do these two forces balance over the course training?
Early on, (cid:107) e y u
−
q
t
( x u ) (cid:107) 2 is relatively large for nearly any x u , because near initialization q t ( x u ) will be relatively ﬂat.
Hence the size of the updates for “similar” x u and the update for x o should be comparable, meaning that, if there are at least a few “similar” training points, q t ( x o ) will move towards p ∗ ( x o ) .
Throughout training, some of these similar samples will become well-classiﬁed, so that (cid:107) e y u − q t ( x u ) (cid:107)becomes small, and their updates will no longer exert much force on q ( x o ) .
Thus, the x o updates begin to dominate, causing the zig-zag pattern as the learning path turns towards e y o .
For easy samples, where p ∗ and e y o are in the same direction, these forces agree and lead to fast convergence.
On samples like the “medium” point in Figure 3, the two forces broadly agree early on, but the learning path deviates slightly towards p ∗ en route to e y o .
L EARNING P ATHS O N R EAL T ASKS
Our analysis in Sections 2 and 3 demonstrate that the model can spontaneously reﬁne the bad labels during one-hot training: the zig-zag learning path ﬁrst moves towards the unknown p ∗ .
But is this also true on real data, for more complex network architectures?
We will now show that they do, although seeing the patterns requires a little more subtlety.
We then propose an algorithm called Filter-KD to take further advantage of these observations for improved generalization.
To verify this explanation on real tasks – we use CIFAR10 and CIFAR100 as examples – the ﬁrst conundrum is that we can no longer calculate the base difﬁculty (cid:107) e y − p ∗ ( x ) (cid:107) 2 of a sample, because p ∗ is unknown.
Furthermore, as the labels of CIFAR dataset are relatively clean, we cannot reliably ﬁnd many samples with bad labels to see whether the pattern is representative.
Recall that there are two cases for a label to be bad: ambiguous sample with ﬂat p ∗ , and unambiguous samples with “wrong” labels.
It might be more persuasive to see that the model can spontaneously “correct” those wrong labels by early stopping at the inﬂection of the zig-zagging path.
Thus, we choose a random 1,000 of the 50,000 training samples, and ﬂip their labels to a random y (cid:48) (cid:54) =
y .
With these changes, we visualize the learning path of data points while training a ResNet-18 (He et al., 2016) on CIFAR10 for 200 epochs as an example.
The ﬁrst panel of Figure 5 shows the learning path of an easy sample with good label.
We can see that q t converges quickly towards the left corner, the one-hot distribution for the correct class, because the color of the scattered points in that ﬁgure are quite light.
At the early stopping epoch, q t has already converged to e y .
However, for a sample with a bad label, it is very difﬁcult to observe any patterns from the raw path (blue points): there are points almost everywhere in this plot.
This is likely caused by the more complex network and dataset, as well as high learning rate in early training.
To ﬁnd the hidden pattern in this high-variance path, we treat q t as a time series signal with many high-frequency components. Thus we can collect its low-frequency components via a low-pass ﬁlter and then draw it, i.e., taking a moving average of the q t (red points).
This makes it clear that, overall, the pattern zig-zags, ﬁrst moving towards the unknown true label before eventually turning to memorize the wrong label.
[Figure]
This smoothing method not only helps us observe the zig-zag pattern, but also improves the predictions. Figure 6 tracks how often the most likely prediction of the network, argmax q t , recovers the true original label, rather than the provided random label, for the corrupted data points. At the beginning of the model’s training, the initialized model randomly guesses labels, getting about 10% of the 1,000 ﬂipped labels correct.
During training, the model spontaneously corrects some predictions, as the learning path ﬁrst moves towards p ∗ . Eventually, though, the model memorizes all its training labels.
Training with the predictions from the 400th epoch corresponds to standard KD (no corrected points).
Early stopping as in ESKD would choose a point with around 700 corrected labels. The smoothed path, though, performs best, with over 800 corrections. Appendix D.
S MOOTHING AS A M ETHOD FOR K NOWLEDGE D ISTILLATION
Figure 6 gives a clear motivation for a new knowledge distillation method, which we call Filter-KD (Algorithm 1): train the student from the smoothed predictions of the teacher network.
Speciﬁcally, we maintain a look-up table q smooth ∈ R N × K to store a moving average of q t for each training sample.
Note that in one epoch, each q smooth ( x n ) will be updated only once.
We check the early stopping criterion with the help of a validation set.
Afterwards, the teaching supervision q smooth is ready, and we can train a student network under its supervision.
This corresponds to using a moving average of the teacher model “in function space,” i.e. averaging the outputs of the function over time.
Compared to ESKD, Filter-KD can avoid the extremely high variance of q t during training.
Note that unlike Figure 5 which plots q t after every iteration of training, most practical implementations only consider early-stopping at the end of each epoch.
This is equivalent to down-sampling the noisy learning path (as in Figure 5 (c)), further exacerbating the variance of q t .
Thus ESKD will likely select a bad p tar for many data points.
Filter-KD, by contrast, has much more stable predictions.
We will show the effectiveness of this algorithm shortly, but ﬁrst we discuss its limitations.
Compared to ESKD, the running time of Filter-KD might be slightly increased (but not too much), because we only do a few extra multiplications and summations for each batch.
More importantly, compared to the teaching model in ESKD, the Filter-KD requires a teaching table q smooth , which will require more memory when the dataset is large.
One alternative avoiding the need for this table would be to instead take an average “in parameter space,” like e.g. the momentum parameters updating as in Ruppert (1988); Szegedy et al. ; He et al.
We empirically ﬁnd that, although this helps the model converge faster, it does not lead to a better teacher network; see Appendix H.
Thus, although Filter-KD has clear drawbacks, we hope that our explanations here may lead to better practical algorithms for large datasets in the future.
Input: Dataset { ( x n , y n ) } Nn =1 , where I n = { 1 , 2 , ..., N } is the index for each pair
# Train the teacher
Initialize a network model, initialize an N × K matrix called q smooth
Go through the entire dataset, calculate q smooth [ n ] = Softmax(model( x n )) for epoch
= 1 , 2 , ..., T do for n ∈ { 1 , 2 , . . . , N } in random order do ˆ p =
Softmax(model( x n )) q smooth [ n ] =
(1 − α ) · q smooth [ n ] + α · ˆ p loss =
CrossEntropy( ˆ p, y n )
Backpropagate and update the parameters end for
Check the early stopping criterion; stop training if satisﬁed end for # Train the student
For each input x n , set p tar = q smooth [ n ] ; train the network under the supervision of p tar
Algorithm 1: Filter-KD.
α controls the cut-off frequency of low-pass ﬁlter (0.05 here).
5. Q UANTITATIVE R ESULTS OF F ILTER -KD
We now compare the performance of Filter-KD and other baselines on a real dataset.
We focus on self-distillation and a ﬁxed temperature τ = 1 (except Table 2), as we want the only difference among these methods to be p tar .
Thus we can conclude the improvement we achieved purely comes from the reﬁned supervision.
Appendix B for discussion and results related to the choice of temperature; τ = 1 is reasonably good for our current setting.
The ﬁrst task we consider is noisy-label classiﬁcation, where we train and validate the model on a dataset with some labels randomly ﬂipped.
After training, all the models are evaluated on a clean held-out test set.
The experiments are conducted on CIFAR (Figure 7) and TinyImageNet (Table 1), under different noise ratios σ : σ = 0 . 1 means 10% of the labels are ﬂipped.
In Figure 7, an interesting trend can be observed: the enhancement brought by Filter-KD is not signiﬁcant when σ is too small or too large. That is reasonable because for small σ , the few samples have bad labels, 0.8 Noisy ratio 0. T e s t a cc u r a c y
CIFAR
OHTESKDFilter 0.
0.8 Noisy ratio 0. T e s t a cc u r a c y CIFAR OHTESKDFilter
[Table]
[Table]
thus the possible enhancement might not be large.
When σ is too high, the labels of similar points become less reliable, and the learning path will no longer head as reliably towards p ∗ .
Thus, for very high noise ratios, the performance of Filter-KD decays back towards that of OHT.
Filter-KD also outperforms other methods in both accuracy and calibration on the clean dataset, as illustrated in Table 3. This remains true in the more common KD case when the teacher network is bigger than the student; see results in Table 2. Note that in Table 2, we no longer keep τ = 1 , as the baselines are more sensitive to τ when teacher is large (see Appendix B for more discussion); instead we optimize τ for each setting. Here “Efﬁ” is short for EfﬁcientNet-B1 (Tan & Le, 2019), “Res” is short for ResNet18, “Mobi” is short for MobileNetV2 (Sandler et al., 2018), “VGG” is short for VGG-11 (Simonyan & Zisserman, 2015).
[Table]
R ELATED W ORK
Human label reﬁnement.
Peterson et al.
(2019) use the distribution of annotations from multiple annotators to replace the one-hot label in training, and ﬁnd that doing so enhances both the generalization performance of trained models and their robustness under adversarial attacks.
However, this approach is relatively expensive, as it requires more annotation effort than typical dataset creation techniques; several experienced annotators are required to get a good estimate of p ∗ .
It is also unclear in general how well the distribution of decisions by human annotators corresponds with the true variability under the data distribution.
Appendix D has experiments based on this dataset.
Label smoothing.
Another popular method is label smoothing (Szegedy et al., 2016), discussed in the previous sections.
We believe that suitable smoothing can decrease (cid:107) p tar − p ∗ (cid:107) 2 to some extent, but the lower bound of this gap should be large, because label smoothing treats each class and each sample in the same way. Much prior research (e.g. M¨uller et al., 2019) shows that KD usually outperforms label smoothing as well.
KD and ESKD.
Knowledge distillation, by contrast, can provide a different p tar for each training sample.
KD was ﬁrst proposed by Hinton et al.
(2015) for model compression, under the assumption that it would help to have a more complex teacher extract more information from the dataset.
Later,
Furlanello et al.
; Cho & Hariharan (2019); Zhang et al.
; Yuan et al.
(2020) found that distillation can bring enhancement even when teacher and student have identical structure.
These surprising ﬁndings sparked studies of self-distillation; our work applies to both self-distillation and larger-teacher cases.
Another active direction in research about KD is ﬁnding explanations for its success, which is often still considered somewhat mysterious.
Tang et al.
(2020) decompose the “dark knowledge” provided by the teacher into three parts: uniform, intra-class and inter-class.
This explanation also overlaps with ours: if we observe the samples with different difﬁculty in the same class, we are discussing intra-class knowledge; if we observe samples with two semantically similar classes, we then have inter-class knowledge.
Among previous work, the most relevant study is that of Menon et al.
(2021), who proposes a statistical explanation of KD.
Our Hypothesis 1 is suggested by the main claim of that paper, which focuses on the question of what is a good p tar .
Our work builds off of theirs by helping explain why this good p tar emerges, and how to ﬁnd a better one .
Besides, by looking deeper into the learning path of samples with different difﬁculty, our work might shed more light on KD, anti-noisy learning, supervised learning, overﬁtting, etc.
Noisy label task.
Learning useful information from noisy labels is a long-standing task (Angluin & Laird, 1988; Natarajan et al., 2013).
There are various ways to cope with it; for instance, Menon et al.
(2019) use gradient clipping, Patrini et al.
(2017) use loss correction, Huang et al.
(2020) change the supervision during training, and Zhang et al.
(2020) take advantage of extra information to achieve good performance.
It is possible to combine KD-based methods with traditional ones to further enhance performance; our perspective of learning paths also provides further insight into why KD can help in this setting.
D ISCUSSION
In this paper, we ﬁrst claim that better supervision signal, i.e., p tar that is closer to p ∗ , leads to better generalization performance; this is supported by results of Menon et al.
(2021) and further empirical suggestions given here.
Based on this hypothesis, we explain why LS and KD outperform one-hot training using experiments on a toy Gaussian dataset.
To further understand how such better supervision emerges, we directly observe the behavior of samples with different difﬁculty by projecting them on a 2-D plane.
The observed zig-zag pattern is well-explained by considering the tradeoff between two forces, one pushing the prediction to be near that of similar inputs, the other pushing the prediction towards its training label; we give an intuitive account based on Propositionof why this leads to the observed zig-zag pattern.
To apply these ﬁndings on real tasks, in which the data and network are more complex, we conduct low-pass ﬁlter on the learning path, and propose Filter-KD to further enhance p tar .
Experimental results on various settings (datasets, network structures, and label noise) not only show the advantage of the proposed method as a method for a teacher in knowledge distillation methods, but also help verify our analysis of how generalization and supervision work in training neural network classiﬁers.
B T EMPERATURE IN DIFFERENT KD METHODS
Temperature is an important hyper-parameter in different kinds of KD methods, which might inﬂuence the performance a lot.
In this appendix, we will discuss why we prefer τ = 1 , and why this is a reasonably good empirical choice in our settings.
Run
Run Run3 τ =0.
τ = τ = τ = τ =0. τ = τ = τ = τ =
[Table]
τ = [Table] τ = τ =10 OHT 94. - - - ESKD 95.
[Table]
[Table]
[Table] [Table] FilterKD [Table]
OHT
[Table] [Table] [Table]
[Table] [Table] FilterKD [Table]
The role played by τ : In general, the loss function in KD has the form:
[Equation]
) where β ∈ [0 , 1] is another hyper-parameter to trade-off the importance between one-hot label and teacher’s predictions. Furthermore, for this loss, the gradient of L to logits z is: [Equation] Why we use τ = 1 :
The most important reason for us to choose τ = 1 (as well as β = 1 ), even with the risk of providing sub-optimal performance, is that we want to observe how much enhancement is brought by reﬁning the label. In our mind, KD’s success comes from the following two aspects: (a) better label (i.e., p tar is better than e y ); (b) better learning dynamics (soften q to q τ make the training easier). The ﬁrst one provides the student with more useful knowledge, and the second one helps the student to extract it.
The focus of our paper is the improvement in labels. When the temperature is not 1, both (a) and (b) are inﬂuenced, as we are using q τ (i.e., the smoothed student output) to match p τ tar (i.e., the smoothed target) during training and using q to inference during testing. If we ﬁx τ = 1 , the only difference between Filter-KD, ESKD, KD, LS, and OHT training is p tar . Under such a condition, we believe the fact that Filter-KD/ESKD outperforms OHT (strictly better in each run) is enough to conclude KD methods can provide better labels.
Furthermore, as Filter-KD is proposed to mitigate the high-variance issue in ESKD, and we can directly observe the zig-zag learning path of samples with bad labels, we believe it is reasonable to make the conclusion in the paper. Furthermore, using τ (cid:54) = 1 doesn’t substantially change our analysis. As illustrated in Equation (7), the gradient shows that q τ moves towards p τ tar in each update. Hence our analysis in propositionstill holds, which means the trade-off between the two forces still exists.
But considering τ (cid:54) =makes the explanation awkward because the prediction of the model is q , not q τ : readers might feel confused why we visualize q and p tar in the learning path but in fact, the gradient comes from q τ and p τ tar .
If we directly visualize q τ and p τ tar , how could we read the value of q from the ﬁgure? So, to make the paper easier to follow, we prefer τ = 1 . τ = is
reasonably good in our settings:
In fact, the optimal τ depends on many settings, e.g., teacher’s size, optimizer, learning rate (scheduler), and etc: there is still no consensus on what is the best choice of τ for all settings.
So for the experiments in the main context, we ﬁx τ and ﬁne-tune other hyper-parameters to achieve good results.
However, to verify this choice of τ is not terrible, we performed a coarse grid search on both
CIFAR and toy examples.
As illustrated in Table 4 and Figure 10, choosing τ ∈ [1 , 2] seems to be good choice: the performance doesn’t signiﬁcantly decay until τ is too large.
[Figure] T e s t [Figure] u r a c
[Figure]
Some readers might also curious about why our results don’t match the common notion that the optimal τ should be 4 , which is ﬁrst provided in Hinton et al.
and followed by much later work (Gao et al., 2018; Zagoruyko & Komodakis, 2017; Cho & Hariharan, 2019, e.g.).
Some works also use a grid search to conclude that a temperature as high as 20 should be the best choice (Yuan et al., 2020; Kim et al., 2019). We ﬁnd this mismatch comes from the relative difference between the network size of teacher and student. In short, when distilling from a large teacher to a small student (as most of the cases in aforementioned works), high τ is preferred.
When conducting self-distillation, τ need not be that high.
For example, Zhang et al. and Roth et al. (2021) claim τ =is the best choice, while Zhang & Sabuncu (2020) claim τ ≈ 2 is the best. To further verify this, we compare the temperature trend between two cases on the toy dataset.
In Figure 11, the ﬁrst row is distilling a 10-layer, 256-width MLP to a 3-layer, 32-width MLP; the second row is self-distillation between 3-layer, 32-width MLPs.
At the same time, we also run a grid search the smoothing factor α of our Filter-KD in Table 5. [Table] C H OW R EPRESENTATIVE IS THE Z IG -Z AG P ATTERN ? In the main paper, we only visualize the zig-zag learning path of a few samples in Figures 3 and 5.
The readers might then wonder whether this pattern is representative across the whole dataset. To verify this, we deﬁne a quantitative metric called zig-zag score . Speciﬁcally, we ﬁrst calculate the integral of each dimension of the prediction: [Equation]
We then use the highest Q i among i ∈ { 1 , ..., K } \ { y } , where y is the label’s class, as the zig-zag score. In other words, we focus on the behavior of q on those dimensions that are not the training label. If this score is large, we might expect the neighbouring samples exert high inﬂuence on the [Figure]
path, and vice versa.
Note that as we have (cid:80) i Q i = constant for any sample (as q t is a K-simplex), this zig-zag score might correlated with C-score of Jiang et al.
However, their focuses are different: C-score is similar to − Q i = y and focuses more on how fast the prediction converge to the label; zig-zag score, on the other hand, is more about how much the path deviates towards a different class, possibly the label close to p ∗ . With this score, we test the correlation between base difﬁculty and zig-zag score on toy datasets under different settings. From Figure 13, it is safe to conclude that samples with higher base difﬁculty will have a more zig-zagging path during training.
We also compare the expected zig-zag score of the 1000 samples with ﬂipped label in CIFAR10 (recall the experiment in Figure 6).
In Figure 12, it is clear that the zig-zagness of these wrong label samples (which we are quite have high base difﬁculty) is signiﬁcantly higher than the average.
Finally, we also randomly select some data samples in each class of CIFAR10 and visualize their learning paths.
Figure 14 shows random samples for the noisy label experiment.
It is clear that the learning path of easy samples with correct labels converge fast while that of samples with wrong labels is zig-zag.
In Figure 15, which shows the learning path with high zig-zag score when the training set is clean, we can still observe some samples with zig-zag path.
These samples might be ambiguous, with a quite ﬂat p ∗ .
However, as we do not know the true p ∗ of these samples, it is impossible to provide a result like Figure 12.
D D ISCUSSION OF CIFAR10H
As previously mentioned, Peterson et al.
(2019) propose a CIFAR10H dataset to reﬁne the one-hot label by human annotators.
However, we decided not to use this baseline in the main body of the paper.
We have the following three reasons:
• CIFAR10H only annotate the test set of CIFAR10, which make it hard to compare with other standard baselines; • We cannot control the difﬁculty of the task like what we do for the toy dataset; • We believe the labels provided by human annotators is not a perfect estimation of p ∗ .
The ﬁrst two reasons are straightforward, we explain why we don’t believe the labels in CIFAR10H is the precise p ∗ .
From our deﬁnition, p ∗ (cid:44) p ∗ ( y | x ) , which is related to the generating process of the image.
Although Peterson et al.
(2019) show that training using this label can bring enhancement
[Figure]
[Figure]
and hence make the prediction more robust, we still believe that there is a bias of human’s cognitive system in it.
In short, labels from CIFAR10H is a good estimation that is far better than one-hot, but not perfect.
To verify it, we conduct the following two experiments. First, we add noise to the labels of CIFAR10H and use them to train a student network. We then evaluate these models on the original CIFAR10 training set (Peterson et al. (2019) also applies this strategy to make an evaluation).
The results are provided in the ﬁrst two panels of Figure 16, which is similar to what we do for the toy dataset in Figure 8.
For the ﬁrst panel, when the added noise are high enough to ﬂip some labels, we see a clear negative correlation between (cid:107) p tar − p cifarh (cid:107)and test accuracy.
However, when we add small amount of noise to p cifarh , as in the second panel in the ﬁgure, the test accuracy will increase.
This means p cifarh might be a little conservative when estimating p ∗ : adding noise and hence making p tar smoother can slightly enhance the generalization ability.
However, in the toy dataset where p ∗ is precisely the generating probability, learning from p ∗ always performs the best.
Furthermore, we also compare the supervisions provided by Fitler-KD and p cifarh .
As illustrated in the last panel in Figure 16, Filter-KD can achieve higher accuracy than CIFAR10H case, which we believe is another evidence for our concern.
E S OME B ACKGROUNDS
Calculation of ECE: expected calibration error is a measurement about how well the predicted conﬁdence represent the true correctness likelihood (Guo et al., 2017).
For example, if our model gives 100 predictions, each with conﬁdence, say, q ( y = k | x ) ∈ [0 . 7 , 0 . 8] ).
Then we might expect there are 70 ∼ 80 correct predictions among those 100 ones.
To calculate this, we ﬁrst uniformly divide [0 , 1] into M bins, with each bin represented by I m ∈ (cid:0) m − 1 M , mM (cid:3) and B m is all the x samples whose conﬁdence falls into I m .
Then, ECE is calculated as:
[Equation]
(9) where acc ( B m ) = 1 | B m | (cid:80) i ∈
B m 1 (ˆ y i = k i ) , conf ( B m )
= 1 | B m | (cid:80) i ∈
B m q ( y = k
i | x i ) , k i is the true label and ˆ y i =
argmax [ q ( y | x i )] is the model’s prediction.
All the ECE mentioned in this paper is calculated by setting M = 10 .
Barycentric coordinate system: when visualizing the learning path, it is problematic to directly choose two dimensions and draw them in the Cartesian coordinate system, as illustrated in the ﬁrst panel in Figure 17.
In geometry, a suitable way to project a 3-simplex vector onto a 2-D plane is converting it to a point in Barycentric coordinate system.
Speciﬁcally, we have three basis vectors: v 0 =
[0 , 0] T , v 1 =
[1 , 0] T , and v 2 = (cid:104) 12 , √ 32 (cid:105) T , which are the corner and two edges of an equilateral triangle respectively.
Then the 2D-coordinate is calculated as ( x, y ) =
[ v 0 ; v 1 ; v 2 ] · q .
So every points in the left corner of a Cartesian system plane can be converted to the triangle in Barycentric system, as illustrated in the last panel in Figure 17.
F DISTANCE GAP UNDER DIFFERENT SUPERVISIONS
Figure 2 provides the distance gap between q and corresponding p ∗ for each training sample in different training stages, under the supervision of one-hot label e y .
From the results in this ﬁgure, we notice that the behavior of hard samples contributes more to the success of ESKD.
Here we further visualize how these gaps changes when the model is trained under different types of supervision, i.e., ground truth, LS and ESKD.
In the ﬁrst row, which is the result of LS training, we see the (cid:107) p tar − p ∗ (cid:107) 2 (i.e., red dots) has a ‘V’- shape.
The vertex of it depends on the smoothing parameter we choose.
However, as we claimed in the section 3.1 and section 6, the LS method has only one parameter to control p tar , which is like using a linear model to ﬁt a high-order function.
So, although a proper smoothing value can bring better supervision than one-hot label, the upper bound of it might be limited.
Regarding the training dynamics, we can see a similar trend as the results shown in the one-hot case, i.e., all the samples ﬁrst move down and then converge to p tar .
From the middle panel, we might expect the LS training outperforms the one-hot case, because the scatters are closer to the x-axis, which represents the ground truth p ∗ .
The second and the third rows demonstrate the KD and ESKD case.
Results in KD case is quite similar to the onehot case in Figure 2, because the converged p tar is close to e y .
However, we can still expect KD outperforms one-hot training, because the p tar is closer to p ∗ than e y does.
The last panel in this row also demonstrates that (cid:107) q − p ∗ (cid:107) 2 might be smaller than (cid:107) p tar − p ∗ (cid:107) 2 , which can be considered as an explanation for why iterated self-distillation, e.g., Born Again Network proposed in (Furlanello et al., 2018), can improve the performance.
For the ESKD case, we see the overﬁtting almost disappear: the distribution of the blue points do not change too much after the early stopping criterion is satisﬁed.
In the last row, which illustrates the training under the supervision of p ∗ , we ﬁnd all the blue points move toward the x-axis, i.e., their p tar = p ∗ , and ﬁnally converge to it.
There is also no overﬁtting in this case.
From the last two panels, we see the disperse of blue points is signiﬁcantly smaller than all other settings, which means the network’s prediction is quite close to p ∗ .
Hence the performance of this case is the best.
G MORE ABOUT THE DECOMPOSITION AND NTK MODEL
Proof of Proposition 1: to observe how q t ( x o ) changes when the network’s parameters update after observing x u [Equation] we can directly write down ∆ t ( x o ) =
q t +1 ( x o ) − q t ( x o ) and simplify it (recall z ( x ) =
f ( w , x ) is the logits of NN and q = Softmax ( z ) is the output probability):
[Equation]
(cid:15) [-
-][Equation]
w t +1 − w t (cid:107) 2 = η 2 · (cid:107)∇ w L (cid:107) 2 .
Here we have (cid:107)∇ w L (cid:107) 2 ≤ (cid:107)∇ z L (cid:107) 2 · (cid:107)∇ w z (cid:107) 2 .
In the right hand set, (cid:107)∇ z L (cid:107) =
(cid:107) p tar − q t (cid:107) is bounded because they are probabilities.
The term (cid:107)∇ w z (cid:107) is also bounded if we use gradient clipping during training.
Thus in most of the case, we have
O ( (cid:15) 2 ) =
O ( η 2 ) , which can be ignored when the learning rate η is sufﬁciently small.
The ﬁrst line is a Taylor expansion, where ∆ t ( x o ) is a K × 1 vector, ∇ w q t ( x o ) | w t is a K × d matrix, and w is a d × 1 vector ( d is the number of parameters).
The second line follows the chain rule, where ∇ z q t ( x o ) | w t is a K × K matrix and ∇ w z ( x o ) | w t is a K × d matrix.
The third line applies the deﬁnition of SGD, where ∇ w L ( x u ) | w t is a 1 × d vector.
The fourth line also follows the chain rule, where ∇ z L ( x u ) | w t is a 1 × K vector and ∇ w z ( x u ) | w t is a K × d matrix.
The ﬁfth and the sixth lines are rearrangments of different terms, where both A t and K t are K × K matrixes.
In this expression,
[Equation]
[Equation]
which is a symmetric positive semi-deﬁnite (PSD) matrix 9 with trace tr ( A t ( x o ))
= 1 − (cid:80)
Ki =
1 q 2 i .
As we have (cid:80) i q i = 1 , it is easy to check the trace of this matrix is larger at the beginning of training (when q tend to be ﬂat) than that at the end of training ( q tend to be peaky), as illustrated by most panels in Figure 19.
Given that the trace of a matrix equals the summation of its eigenvalues and A t ( x o ) is PSD, smaller tr ( A t ( x o ))
= 1 − (cid:80)
Ki =
1 q 2 i means this matrix will shrink the vector it multiplied more.
Hence the step size of ∆ t ( x o ) tend to decrease when q t becomes more peaky.
The second term in that expression, i.e., K t ( x o , x u ) is the inner product of gradients at x o and x u .
Intuitively, if their gradients has similar directions, this term might be large, and vice versa.
As this term also depends on t , it is hard to analyze its property when there are two inﬂuencing factors.
So following the results about NTK, which claims that an inﬁnitely wide NN can be approximated by a linear kernel model, we study K 0 instead of K t to focus more on the inﬂuence of x u .
Neural Tangent Kernel: researchers ﬁnd that the weights of a NN tend to be ‘lazy’ during training (Jacot et al., 2018) when the NN is wide enough and the learning rate is sufﬁciently small.
In other words, the offset of each dimension of w 0 and w T is small.
Hence, it is reasonable to use ﬁrst-order
Taylor expansion on w 0 to approximate the function indicated by the model:
[Equation]
[Equation]
[Equation]
[Equation]
where we use φ ( x ) T and b to simplify the notation.
Then, a 1-layer inﬁnitely wide NN can be approximated by the aforementioned linear model, where φ ( x ) plays a similar role as the design matrix in a linear regression model.
The NTK of that model is deﬁned as K 0 ( x i , x j ) (cid:44) φ ( x i )
T φ ( x j ) , which is useful when analyzing the learning dynamics of the model.
Learning Dynamics of q ( x o ) : in proposition 1, we decompose ∆ t ( x o ) into three parts and analyze what the learning path of a training sample might be like in section 3.3.
Here we will provide more detailed illustration of the two groups of force imposed on q ( x o ) .
Speciﬁcally, ∆ t ( x o ) will be inﬂuenced by two variables, i.e., updating sample x u and time t .
We discuss their effects separately.
For x u , only the last two terms, i.e., K 0 ( x o , x u ) and ( p tar ( x u ) − q t ( x u )) depends on it.
In section 3.3, we claim that if x o and x u are similar, the norm of K 0 ( x o , x u ) might be large, and vice versa.
Here we empericially verify this using a toy Gaussian dataset, as illustrated in
Figure 19.
The ﬁgure shows how the similarity between x o and x u correlates with tr ( K 0 ( x o , x u )) .
Each panel represents one x o , which is claimed in the title of the subﬁgures.
The x-axis represents the rank of the cosine similarity between observed x o and all N training samples (including itself).
The y-axis is the trace of K 0 ( x o , x u ) .
The color of each scatter point is the true label of x u .
From the ﬁgure, we can observe a clear decreasing trend, which means smaller similarity leads to larger tr ( K 0 ( x o , x u )) .
Additionally, the term ( p tar ( x u ) − q t ( x u )) provides a direction that q t ( x o ) should move towards.
We claim in section 3.3 that at any point in the input space, the labels of these input samples might follow the ground truth distribution, i.e., p ∗ ( y | x ) .
Hence most of the neighbouring x might pull q ( x o ) towards its ground truth p ∗ ( y | x o ) .
We will discuss the norm of this term when discussing the inﬂuence of t .
In short, at any time, the neighbouring x u will impose stronger force on x o and the direction of the force roughly points to the ground truth p ∗ ( y | x o ) .
Another factor to consider is t , which inﬂuence ∆ t ( x o ) through the term A t ( x o ) and term ( p tar ( x u ) − q t ( x u )) .
However, these two terms do not have an analytical expression w.r.t t , as q t is quite complicated.
So we can only provide some intuitions through the behavior of q t and then experimentally verify them.
In Figure 19, we show how tr ( A t ( x o )) depends on q t : ﬂat q t leads to larger trace.
A similar trend also exists in the norm of ( p tar ( x u ) − q t ( x u )) .
As the initialized q tend to be ﬂat, updates of any samples will inﬂuence network’s parameters a lot.
When the training progress, those easy samples converge fast, so their (cid:107) p tar ( x u ) − q t ( x u ) (cid:107) 2 and tr ( A t ) become small.
However, as the q t for the hard sample is still far away from its e y , the large (cid:107) p tar ( x u ) − q t ( x u ) (cid:107)and tr ( A t ) will ﬁnally drag q t back towards the one-hot distribution, as illustrated in Figure 3.
H LOW PASS FILTER ON NETWORK PARAMETERS
In Section 4, we point out the high variance issue of the traditional KD methods after observing the learning path of those hard samples.
We then propose a Filter-KD algorithm to smooth the output of each training sample during training.
Such a low pass ﬁltering method is quite similar to the momentum mechanism used in self-supervised learning, e.g., MOCO in He et al.
(2020), which conduct low pass ﬁltering on each parameter of the network.
As mentioned before, the proposed
Filter-KD algorithm might require more memory when the dataset becomes larger, because q smooth would record every training sample’s prediction.
Conducting low pass ﬁltering on network parameters might be a good way to solve this.
To verify whether this method works, we train a ResNet18 on CIFAR100, using one-hot supervision.
At the beginning of training, we copy the training model and call it tracking model.
At the end of each update (i.e., each batch), the parameters of training model is updated as usual while the tracking model’s parameters are updated with momentum.
Speciﬁcally, for the training model, w t +1 train =
w t train + η ∇ L , for the tracking model, w t +1 track =
(1 − α ) w t track + α w t +1 train .
We train the model for 150 epochs, and show the learning curve of test accuracy in the left panel of Figure 20.
We see the performance of the tracking model converges faster than the training model, which is reasonable because ﬁltering parameters is regularizing the training process.
However, the converging accuracy of this two models are the same.
At the same time, we ﬁnd this tracking model is not as good as q smooth when teaching a new model, as illustrated in the right panel of Figure 20.
As this paper mainly focuses on the behavior of the model’s prediction, the discussion and experiments of ﬁltering in parameter space is limited.
However, as many exciting works demonstrates the effectiveness of this momentum mechanism, we would try to understand it in our future work.
[Table]
Run Run Run
Run
Run Run Run Run
Run
[Table] OHT 95. KD 95. ESKD 95.
FilterKD 95.
OHT 78. KD 78. ESKD 78.
FilterKD 79.