B ETTER S UPERVISORY S IGNALS BY O BSERVING L EARNING P ATHS
A BSTRACT
Better-supervised models might have better performance.
In this paper, we ﬁrst clarify what makes for good supervision for a classiﬁcation problem, and then explain two existing label reﬁning methods, label smoothing and knowledge distillation, in terms of our proposed criterion.
To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the model’s predictions during training, for each training sample.
We ﬁnd that the model can spontaneously reﬁne “bad” labels through a “zig-zag” learning path, which occurs on both toy and real datasets.
Observing the learning path not only provides a new perspective for understanding knowledge distillation, overﬁtting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks.
Inspired by this, we propose a new knowledge distillation scheme, Filter-KD, which improves downstream classiﬁcation performance in various settings.
I NTRODUCTION
In multi-class classiﬁcation problems, we usually supervise our model with “one-hot” labels: label vectors y which have y i = 1 for one i , and 0 for all other dimensions.
Over time, however, it has gradually become clear that this “default” setup is not always the best choice in practice, in that other schemes can yield better performance on held-out test sets.
One such alternative is to summarize a distribution of human annotations, as Peterson et al.
(2019) did for CIFAR10.
An alternative approach is label smoothing (e.g. Szegedy et al., 2016), mixing between a one-hot label and the uniform distribution.
Knowledge distillation (KD), ﬁrst training a teacher network on the training set and then a student network on the teacher’s output probabilities, was originally proposed for model compression (Hinton et al., 2015) but can also be thought of as reﬁning the supervision signal: it provides “soft” teacher outputs rather than hard labels to the student.
Knowledge distillation is promising because it requires no additional annotation effort, but – unlike label smoothing – can still provide sample-speciﬁc reﬁnement.
Perhaps surprisingly, knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student and trained on the same data; this is known as self-distillation (Furlanello et al., 2018; Zhang et al., 2019).
There have been many recent attempts to explain knowledge distillation and speciﬁcally self-distillation (e.g. Menon et al., 2021; Allen-Zhu & Li, 2020; Tang et al., 2020), from both optimization and supervision perspectives.
We focus on the latter area, where it is usually claimed that the teacher provides useful “dark knowledge” to the student through its labels.
Inspired by this line of work, we further explore why and how this improved supervisory signal emerges during the teacher’s one-hot training.
Speciﬁcally, we ﬁrst clarify that given any input sample x , good supervision signals should be close (in L 2 distance) to the ground truth categorical distribution, i.e., p ∗ ( y | x ) .
We then show that a neural network (NN) can automatically reﬁne “bad labels”, those where p ∗ ( y | x ) is far from the training set’s one-hot vector.
During one-hot training, the model prediction on such a sample ﬁrst moves towards p ∗ ( y | x ) , and then slowly converges to its supervisory label, following a “zig-zag” pattern.
A well-trained teacher, one that does not overﬁt to particular training labels, can thus provide supervisory signals closer to p ∗ ( y | x ) .
We justify analytically (Section 3.3) that this pattern is common in gradient descent training.
Our explanations cause us to recognize that this signal can be better identiﬁed by taking a moving average of the teacher’s prediction, an algorithm we term Filter-KD.
This approach yields better supervision and hence better downstream performance, especially when there are many bad labels.
After completing this work, we became aware of an earlier paper (Liu et al., 2020) studying almost the same problem in a similar way.
We discuss differences between the papers throughout.
S UPERVISION INFLUENCES GENERALIZATION
We begin by clarifying how the choice of supervisory signal affects the learned model.
2. C HOICES OF SUPERVISION SIGNAL
In K -way classiﬁcation, our goal is to learn a mapping f : X → ∆ K that can minimize the risk
[Equation]
Here the label y is an integer ranging from 1 to K , and f gives predictions in the probability simplex ∆ K (a nonnegative vector of length K summing to one); L ( y, f ( x )) is the loss function, e.g. crossentropy or square loss.
The input signal x is usually high dimensional, e.g., an image or a sequence of word embeddings.
The joint distribution of ( x , y ) is P , whose density 2 can be written as p ( x ) p
( y | x ) .
In practice, as P is unknown, we instead (approximately) minimize the empirical risk
[Equation]
, (2) where 1 ( y n = k ) is an indicator function which equals {+
+} if y n = k or 0 otherwise, and e y n ∈ { 0 , 1 } K is its one-hot vector form.
L ( f ( x n ))
=
( L (1 , f ( x n )) , . . .
, L ( K, f ( x n )))
∈ R K is the loss for each possible label.
In R emp , the N training pairs D (cid:44) { ( x n , y n ) }
Nn =1 are sampled i.i.d.
from P .
Comparing (2) to (1), we can see p ( x ) is approximated by an uniform distribution over the samples, which is reasonable.
However, using an indicator function (i.e., one-hot distribution) to approximate p ( y | x ) bears more consideration.
For example, if a data point x is quite vague and its true p ( y | x ) is ﬂat or multimodal, we might hope to see x multiple times with different label y during training.
But actually, most datasets have only one copy of each x , so we only ever see one corresponding e y .
Although R emp is an unbiased estimator for R , if we used a better (e.g. lower-variance) estimate of p ( y | x ) , we could get a better estimate for R and thus, hopefully, better generalization.
Speciﬁcally, suppose we were provided a “target” distribution p tar ( y | x ) (written in vector form as p tar ( x ) ) for each training point x , as D (cid:48)
= { ( x n , p tar ( x n ))
} Nn =1 .
Then we could use
R tar [Equation] f, D
[Equation]
) (cid:44)
[Equation]
Standard training with R emp is a special case of R tar , using p tar ( x n ) =
e y n .
The CIFAR10H dataset (Peterson et al., 2019) is one attempt at a different p tar , using multiple human annotators to estimate p tar .
Label smoothing (e.g. Szegedy et al., 2016) sets p tar to a convex combination of e y and the constant vector 1 K 1 .
In knowledge distillation (KD; Hinton et al., 2015), a teacher is ﬁrst trained on
D , then a student learns from D (cid:48) with p tar based on the teacher’s outputs.
All three approaches yield improvements over standard training with R emp .
[Figure]
M EASURING THE Q UALITY OF S UPERVISION
Choosing a different p tar , then, can lead to a better ﬁnal model.
Can we characterize which p tar will do well?
We propose the following, as a general trend.
Hypothesis 1.
Suppose we train a model supervised by p tar , that is, we minimize R tar ( f, D (cid:48) ) .
Then, smaller average L 2 distance between p tar and the ground truth p ∗ on these samples, i.e. small E x [ (cid:107) p tar ( x ) − p ∗ ( x ) (cid:107) 2 ] , will in general lead to better generalization performance.
This hypothesis is suggested by Proposition 3 of Menon et al.
(2021), which shows (tracking constants omitted in their proof) that for any predictor f and loss bounded as L ( y, ˆ y ) ≤ (cid:96) ,
[Equation]
When N is large, the second term will dominate the right-hand side, implying smaller average (cid:107) p tar − p ∗ (cid:107) will lead to R tar being a better approximation of the true risk R ; minimizing it should then lead to a better learned model.
This suggests that the quality of the supervision signal can be roughly measured by its L 2 distance to the ground truth p ∗ . Appendix B slightly generalizes the result of Menon et al.
with bounds based on total variation ( L 1 ) and KL divergences; we focus on the L 2 version here for simplicity.
To further support this hypothesis, we conduct experiments on a synthetic Gaussian problem (Figure 1 (a); details in Appendix C), where we can easily calculate p ∗ ( y | x ) for each sample.
We ﬁrst generate several different p tar by adding noise 3 to the ground truth p ∗ , then train simple 3-layer
NNs under that supervision.
We also show ﬁve baselines: one-hot training (OHT), label smoothing (LS), KD, early-stopped KD (ESKD), and ground truth (GT) supervision (using p ∗ ).
KD refers to a teacher trained to convergence, while ESKD uses a teacher stopped early based on validation accuracy.
We early-stop the student’s training in all settings.
From Figure 1 (b-c), it is clear that smaller (cid:107) p tar − p ∗ (cid:107) 2 leads to better generalization performance, as measured either by accuracy (ACC) or expected calibration error (ECE) 4 on a held-out test set.
Appendix C has more detailed results.
I NSIGHTS FROM THE LEARNING PATH
In the toy example of Section 2, we see that ESKD outperforms other baselines in accuracy by a substantial margin (and all baselines are roughly tied in ECE).
We expect that supervision with smaller (cid:107) p tar − p ∗ (cid:107) 2 leads to better generalization performance, but it is not clear how better p tar emerges from when the teacher in ESKD is trained using one-hot labels.
This section will answer this, by observing the learning paths of training samples.
mean
We provide more results about the NNs trained under different supervisions using this fashion in Appendix F
3. P AY MORE ATTENTION TO HARDER SAMPLES
For a ﬁner-grained understanding of early stopping the teacher, we would like to better understand how the teacher’s predictions evolve in training.
Assuming Hypothesis 1, the main factor is E x (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 , where q is the teacher’s output probability distribution.
We expect, though, that this term will vary for different x , in part because some samples are simply more difﬁcult to learn. As a proxy for this, we deﬁne base difﬁculty as (cid:107) e y − p ∗ ( x ) (cid:107) 2 , which is large if:
• x is ambiguous: p ∗ has several large components, so there is no one-hot label near p ∗ .
• x is not very ambiguous (there is a one-hot label near p ∗ ), but the sample was “unlucky” and drew y from a low-probability class.
Figure 2 shows these two quantities at three points in training: initialization, the point where ESKD’s teacher stops, and convergence.
At initialization, most points 5 have large (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 .
By the point of early stopping, most q ( x ) values are roughly near p ∗ .
At convergence, however, q ( x ) ≈ e y , as the classiﬁer has nearly memorized the training inputs, leading to a diagonal line in the plot.
It is clear the biggest contributors to E x (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 when training a model to convergence are the points with high base difﬁculty.
Per Hypothesis 1, these are the points we should most focus on.
3. L EARNING P ATH OF D IFFERENT S AMPLES
To better understand how q changes for each sample, we track the model’s outputs on all training samples at each training step.
Figure 3 shows four samples with different base difﬁculty, with the vectors of three probabilities plotted as points on the simplex (details in Appendix A).
The two easy samples very quickly move to the correct location near e y (as indicated by the light color until reaching the corner).
The medium sample takes a less direct route, drifting off slightly towards p ∗ , but still directly approaches e y .
The hard sample, however, does something very different: it ﬁrst approaches p ∗ , but then veers off towards e y , giving a “zig-zag” path.
In both the medium and hard cases, there seems to be some “unknown force” dragging the learning path towards p ∗ ; in both cases, the early stopping point is not quite perfect, but is noticeably closer to p ∗ than the ﬁnal converged point near e y .
In other words, during one-hot training, the NNs can spontaneously reﬁne the “bad labels.”
Under Hypothesis 1, this partly explains the superior performance of ESKD to
KD with a converged teacher.
These four points are quite representative of all samples under different toy-dataset settings.
In Appendix E, we also deﬁne a “zig-zagness score” to numerically summarize the learning path shape, and show it is closely correlated to base difﬁculty.
E XPLANATION OF PATTERNS IN THE LEARNING PATH
The following decomposition will help explain the “unknown force” pushing q towards p ∗ .
Proposition 1.
Let z t ( x ) (cid:44) f
( w t , x ) denote the network output logits with parameters w t , and q t ( x ) =
Softmax( z t ( x )) the probabilities.
Let w t +1 (cid:44) w t − η ∇ w (cid:0) p tar ( x u ) T L ( q t ( x u )) (cid:1) be the result of applying one step of SGD to w t using the data point ( x u , p tar ( x u )) with learning rate η .
Then the change in network predictions for a particular sample x o is
[Equation]
[Equation]
where A t ( x o )
= ∇ z q t ( x o ) and K t ( x o , x u ) =
( ∇ w z ( x o ) | w t ) ( ∇ w z ( x u ) | w t ) T are K × K matrices.
The matrix K t is the empirical neural tangent kernel , which we can think of roughly as a notion of “similarity” between x o and x u based on the network’s representation at time t , which can change during training in potentially complex ways.
In very wide networks, though, K t is nearly invariant throughout training and nearly independent of the initialization (Jacot et al., 2018; Arora et al., 2019). The gradient norm can be controlled by gradient clipping, or bounded with standard SGD analyses when optimization doesn’t diverge.
Appendix G has more details and the proof.
Figure 4 shows the learning path of a hard sample during one-hot training, say x o , where the label e y o is far from p ∗ ( x o ) .
In each epoch, w will receive N − 1 updates based on the loss of x u (cid:54) =
x o (the small blue path), and one update based on x o (the big red path).
At any time t , “dissimilar” samples will have small K t ( x o , x u ) (as measured, e.g., by its trace), and hence only slightly affect the predicted label for x o .
Similar samples will have large K t ( x o , x u ) , and hence affect its updates much more; because p ∗ is hopefully similar for similar x values, it is reasonable to expect that the mean of e y for data points with similar x will be close to p ∗ ( x o ) .
Thus the net effect of updates for x u (cid:54) =
x o should be to drag q ( x o ) towards p ∗ ( x o ) .
This is the “unknown force” we observed earlier.
The other force affecting q ( x o ) during training is, of course, the update based on x o , driving q ( x o ) towards e y o .
How do these two forces balance over the course training?
Early on, (cid:107) e y u
−
q
t
( x u ) (cid:107) 2 is relatively large for nearly any x u , because near initialization q t ( x u ) will be relatively ﬂat.
Hence the size of the updates for “similar” x u and the update for x o should be comparable, meaning that, if there are at least a few “similar” training points, q t ( x o ) will move towards p ∗ ( x o ) .
Throughout training, some of these similar samples will become well-classiﬁed, so that (cid:107) e y u − q t ( x u ) (cid:107) 2 becomes small, and their updates will no longer exert much force on q ( x o ) .
Thus, the x o updates begin to dominate, causing the zig-zag pattern as the learning path turns towards e y o .
For easy samples, where p ∗ and e y o are in the same direction, these forces agree and lead to fast convergence.
On samples like the “medium” point in Figure 3, the two forces broadly agree early on, but the learning path deviates slightly towards p ∗ en route to e y o .
Liu et al.
(2020) prove that a similar pattern occurs while training a particular linear model on data with some mislabeled points, and speciﬁcally that stopping training at an appropriate early point will give better predictions than continuing training.
L EARNING PATHS ON REAL TASKS
Our analysis in Sections 2 and 3 demonstrate that the model can spontaneously reﬁne the bad labels during one-hot training: the zig-zag learning path ﬁrst moves towards the unknown p ∗ .
But is this also true on real data, for more complex network architectures?
We will now show that they do, although seeing the patterns requires a little more subtlety.
We visualize the learning path of data points while training a ResNet18 (He et al., 2016) on CIFAR for 200 epochs as an example.
The ﬁrst panel of Figure 5 shows the learning path of an easy sample.
We can see that q t converges quickly towards the left corner, the one-hot distribution for the correct class, because the color of the scattered points in that ﬁgure are quite light.
At the early stopping epoch, q t has already converged to e y .
However, for a hard sample, it is very difﬁcult to observe any patterns from the raw path (blue points): there are points almost everywhere in this plot.
This is likely caused by the more complex network and dataset, as well as a high learning rate in early training.
To ﬁnd the hidden pattern in this high-variance path, we treat q t as a time series signal with many high-frequency components.
Thus we can collect its low-frequency components via a lowpass ﬁlter and then draw that, i.e., taking a exponential moving average (EMA) on q t (red points).
This makes it clear that, overall, the pattern zig-zags, ﬁrst moving towards the unknown true label before eventually turning to memorize the wrong label.
This ﬁltering method not only helps us observe the zig-zag pattern, but can also reﬁne the labels.
We use the following two experiments to verify this.
First, we train the model on CIFAR10H using one-hot labels. As CIFAR10H provides p hum , which can be considered as an approximation of p ∗ , we track the mean distance between q and p hum during training. In the ﬁrst panel of Figure 6, which averages the distance of all 10k training samples, the difference between the blue q kd curve and red q ﬁlt curve is quite small. However, we can still observe that the red curve is lower than the blue one before overﬁtting: ﬁltering can indeed reﬁne the labels.
This trend is more signiﬁcant when considering the most difﬁculty samples, as in the second panel: the gap between curves is larger.
To further verify the label reﬁnement ability of ﬁltering, we randomly choose 1,000 of the 50,
CIFAR10 training samples, and ﬂip their labels to a different y (cid:48) (cid:54) =
y .
The last panel in Figure 6 tracks how often the most likely prediction of the network, argmax q t , recovers the true original label, rather than the provided random label, for the corrupted data points.
At the beginning of the model’s training, the initialized model randomly guesses labels, getting about 10% of the 1,000 ﬂipped labels correct.
During training, the model spontaneously corrects some predictions, as the learning path ﬁrst moves towards p ∗ .
Eventually, though, the model memorizes all its training labels.
Training with the predictions from the 400th epoch corresponds to standard KD (no corrected points).
Early stopping as in ESKD would choose a point with around 70% of labels corrected.
The ﬁltered path, though, performs best, with over 80% corrected.
F ILTERING AS A METHOD FOR KNOWLEDGE DISTILLATION
Figure 6 gives a clear motivation for a new knowledge distillation method, which we call Filter-KD (Algorithm 1): train the student from the smoothed predictions of the teacher network.
Speciﬁcally, we maintain a look-up table q smooth ∈ R N × K to store a moving average of q t for each training sample.
Note that in one epoch, each q smooth ( x n ) will be updated only once.
We check the early stopping criterion with the help of a validation set.
Afterwards, the teaching supervision q smooth is ready, and we can train a student network under its supervision.
This corresponds to using a moving average of the teacher model “in function space,” i.e. averaging the outputs of the function over time.
Compared to ESKD, Filter-KD can avoid the extremely high variance of q t during training.
Unlike Figure 5, which plots q t after every iteration of training, most practical implementations only consider early-stopping at the end of each epoch.
This is equivalent to down-sampling the noisy learning path (as in the last panel of Figure 5), further exacerbating the variance of q t .
Thus ESKD will likely select a bad p tar for many data points.
Filter-KD, by contrast, has much more stable predictions.
We will show the effectiveness of this algorithm shortly, but ﬁrst we discuss its limitations.
Compared to ESKD, the running time of Filter-KD might be slightly increased.
Furthermore, compared to the teaching model in ESKD, the Filter-KD requires a teaching table q smooth , which will require substantial memory when the dataset is large.
One alternative avoiding the need for this table would be to instead take an average “in parameter space,” like e.g. momentum parameter updating as in
Szegedy et al.
We empirically ﬁnd that, although this helps the model converge faster, it does not lead to a better teacher network; see
Appendix I. Thus, although Filter-KD has clear drawbacks, we hope that our explanations here may lead to better practical algorithms in the future. Similarly inspired by this spontaneous label reﬁning mechanism (or early stopping regularization), Liu et al. and
Huang et al.
(2020) each propose algorithms aiming at the noisy label problem.
We discuss the relationship between these three methods in Appendix H. 5. Q UANTITATIVE RESULTS OF F ILTER -KD We now compare the performance of Filter-KD and other baselines on a real dataset. We focus on self-distillation and a ﬁxed temperature τ = 1 (except Table 2 and last panel in Figure 7),
as we
[Table]
[Table]
want the only difference among these methods to be p tar .
Thus we can conclude the improvement we achieved comes purely from the reﬁned supervision.
See Appendix D for other temperatures.
[Table]
Input: Dataset { ( x n , y n ) } Nn =1 , where I n = { 1 , 2 , ..., N } is the index for each pair
# Train the teacher
Initialize a network model, initialize an N × K matrix called q smooth
Go through the entire dataset, calculate q smooth [ n ] = Softmax(model( x n )) for epoch
= 1 , 2 , ..., T do for n ∈ { 1 , 2 , . . .
, N } in random order do ˆ p =
Softmax(model( x n ))
q smooth [ n ] =
(1 − α ) · q smooth [ n ] + α · ˆ p
Update parameters loss ( ˆ p, y n ), end for
Check the early stopping criterion; stop training if satisﬁed end for # Train the student
For each input x n , set p tar = q smooth [ n ] ; train the network under the supervision of p tar
The ﬁrst task we consider is noisy-label classiﬁcation, where we train and validate the model on a dataset with some labels randomly ﬂipped.
After training, all the models are evaluated on a clean held-out test set.
The experiments are conducted on CIFAR (Figure 7) and TinyImageNet (Table 1), under different noise ratios σ : σ = 0 .
1 means 10% of the labels are ﬂipped.
In Figure 7, an interesting trend can be observed: the enhancement brought by Filter-KD is not signiﬁcant when σ is too small or too large.
That is reasonable because for small σ , few samples have bad labels, thus the possible enhancement might not be large.
When σ is too high, the labels of similar points become less reliable, and the learning path will no longer head as reliably towards p ∗ .
Thus, for very high noise ratios, the performance of Filter-KD decays back towards that of OHT.
Filter-KD also outperforms other methods in both accuracy and calibration on the clean dataset, as illustrated in Table 3.
This remains true in the more common KD case when the teacher network is bigger than the student; see results in Table 2.
Note that in Table 2, we no longer keep τ = 1 , as the baselines are more sensitive to τ when the teacher is large (see Appendix D for more discussion); instead we optimize τ for each setting.
Here “Eff” is short for EfﬁcientNet-B1 (Tan & Le, 2019), “Res” is short for ResNet18, “Mobi” is short for MobileNetV2 (Sandler et al., 2018), “VGG” is short for VGG-11 (Simonyan & Zisserman, 2015).
In summary, the explanation in Section 4 and experimental results in this section demonstrate that better supervisory signal, which can be obtained by Filter-KD, can enhance the prediction performance in both clean and noisy-label case. (It is also possible that the algorithm improves perfor- mance for other reasons as well, especially in the clean label case; the inﬂuence of temperature may be particularly relevant.)
R ELATED WORK
Human label reﬁnement Peterson et al.
(2019) use a distribution of labels obtained from multiple annotators to replace the one-hot label in training, and ﬁnd that doing so enhances both the generalization performance of trained models and their robustness under adversarial attacks.
However, this approach is relatively expensive, as it requires more annotation effort than typical dataset creation techniques; several experienced annotators are required to get a good estimate of p ∗ .
Label smoothing Another popular method is label smoothing (Szegedy et al., 2016), discussed in the previous sections.
We believe that suitable smoothing can decrease (cid:107) p tar − p ∗ (cid:107) 2 to some extent, but the lower bound of this gap should be large, because label smoothing treats each class and each sample in the same way.
Much prior research (e.g. M¨uller et al., 2019) shows that KD usually outperforms label smoothing as well.
KD and ESKD Knowledge distillation, by contrast, can provide a different p tar for each training sample.
KD was ﬁrst proposed by Hinton et al.
(2015) for model compression.
Later, Furlanello et al.
; Cho & Hariharan (2019); Zhang et al.
; Yuan et al.
(2020) found that distillation can help even in “self-distillation,” when the teacher and student have identical structure.
Our work applies to both self-distillation and larger-teacher cases.
Another active direction in research about KD is ﬁnding explanations for its success, which is often still considered somewhat mysterious.
Tang et al.
(2020) decompose the “dark knowledge” provided by the teacher into three parts: uniform, intra-class and inter-class.
This explanation also overlaps with ours: if we observe the samples with different difﬁculty in the same class, we are discussing intra-class knowledge; if we observe samples with two semantically similar classes, we then have inter-class knowledge.
Among previous work, the most relevant study is that of Menon et al.
Our Hypothesis 1 is suggested by their main claim, which focuses on the question of what is a good p tar .
Our work builds off of theirs by helping explain why this good p tar emerges, and how to ﬁnd a better one .
Besides, by looking deeper into the learning path of samples with different difﬁculty, our work might shed more light on KD, anti-noisy learning, supervised learning, overﬁtting, etc.
Noisy label task Learning useful information from noisy labels is a long-standing task (Angluin & Laird, 1988; Natarajan et al., 2013).
There are various ways to cope with it; for instance, Menon et al.
(2019) use gradient clipping, Patrini et al.
(2017) use loss correction, Huang et al.
(2020) change the supervision during training, and Zhang et al.
(2020) employ extra information.
It is possible to combine KD-based methods with traditional ones to further enhance performance; our perspective of learning paths also provides further insight into why KD can help in this setting.
D ISCUSSION
In this paper, we ﬁrst claim that better supervision signal, i.e., p tar that is closer to p ∗ , leads to better generalization performance; this is supported by results of Menon et al.
(2021) and further empirical suggestions given here.
Based on this hypothesis, we explain why LS and KD outperform one-hot training using experiments on a toy Gaussian dataset.
To further understand how such better supervision emerges, we directly observe the behavior of samples with different difﬁculty by projecting them on a 2-D plane.
The observed zig-zag pattern is well-explained by considering the tradeoff between two forces, one pushing the prediction to be near that of similar inputs, the other pushing the prediction towards its training label; we give an intuitive account based on Propositionof why this leads to the observed zig-zag pattern.
To apply these ﬁndings on real tasks, in which the data and network are more complex, we conduct low-pass ﬁlter on the learning path, and propose Filter-KD to further enhance p tar .
Experimental results on various settings (datasets, network structures, and label noise) not only show the advantage of the proposed method as a method for a teacher in knowledge distillation methods, but also help verify our analysis of how generalization and supervision work in training neural network classiﬁers. A CKNOWLEDGEMENTS This research was enabled in part by support provided by the Canada CIFAR AI Chairs program, WestGrid and Compute Canada.
A M ISCELLANEOUS BACKGROUND
Calculation of ECE Expected calibration error is a measurement about how well the predicted conﬁdence represent the true correctness likelihood (Guo et al., 2017).
For example, if our model gives 100 predictions, each with conﬁdence, say, q ( y = k | x ) ∈ [0 . 7 , 0 . 8] ).
Then we might expect there are 70 ∼ 80 correct predictions among those 100 ones.
To calculate this, we ﬁrst uniformly divide [0 , 1] into M bins, with each bin represented by I m ∈ (cid:0) m − 1 M , mM (cid:3) and B m is all the x samples whose conﬁdence falls into I m .
Then, ECE is calculated as:
[Equation]
where acc ( B m ) = 1 | B m | (cid:80) i ∈ B m 1 (ˆ y [Equation] =
[Equation]
[Equation]
B [Equation] = 1 | B m | (cid:80) i
[Equation]
B m q ( y = k i | x i ) , k i is the true label and ˆ y i = argmax [ q ( y | x i )] is the model’s prediction.
All the ECE mentioned in this paper is calculated by setting M = 10 .
Barycentric coordinate system When visualizing the learning path, it is problematic to directly choose two dimensions and draw them in the Cartesian coordinate system, as illustrated in the ﬁrst panel in Figure 8.
In geometry, a suitable way to project a 3-simplex vector onto a 2-D plane is converting it to a point in a barycentric coordinate system.
Speciﬁcally, we have three basis vectors: v 0 =
[0 , 0] T , v 1 =
[1 , 0] T , and v 2 = (cid:104) 12 , √ 32 (cid:105) T , which are the corner and two edges of an equilateral triangle respectively.
Then the 2D-coordinate is calculated as ( x, y ) =
[ v 0 ; v 1 ; v 2 ] · q .
So every points in the left corner of a Cartesian system plane can be converted to the triangle in a barycentric system, as illustrated in the last panel in Figure 8.
B R ISK ESTIMATE VARIANCE BOUND
The following result is a generalization of Proposition 3 of Menon et al.
(2021), whose proof we replicate and extend here:
Proposition 2.
Let L be any bounded loss, L ( y, ˆ y ) ≤ (cid:96) < ∞ , and consider R tar of (3) .
For any predictor f : X → ∆ K , we have that
[Equation]
where ξ can be any of the following seven quantities:
[Equation]
Proof.
To begin,
[Equation]
For the variance term, since R ( f ) is a constant and R tar is an average of N i.i.d.
terms, we get
Var D
[Equation]
[ R tar
[Equation]
f, D (cid:48) )
− R ( f )]
= 1 N Var x [ p tar ( x ) T L ( f ( x ))] .
The other term, as R tar is an average of i.i.d.
terms and R ( f ) =
E x p ∗ ( x ) (cid:62) L ( f ( x )) , is
[Equation]
For the ﬁrst bound, we apply the Cauchy-Schwarz inequality,
[Equation]
since the elements of L ( f ( x )) are each at most (cid:96) , the term (cid:107) L ( f ( x ))
(cid:107) 2 is at most √ K(cid:96) .
For the other bounds, we instead apply H ¨ older’s inequality, yielding
[Equation]
The KL bounds follow by Pinsker’s inequality and then Jensen’s inequality.
The last bound, for the
Jeffreys divergence, combines the two KL bounds.
C T OY G AUSSIAN DATASET
In Sections 2 and 3, we apply a toy Gaussian dataset to verify two facts derived from Hypothesis 1, so as the learning path of speciﬁc samples.
Here we provide more details about this dataset. [Table] τ [Table]
τ
[Table] τ [Table] τ
[Table]
τ
[Table]
τ
[Table]
τ
[Table]
τ
[Table]
τ
[Table]
τ
[Table]
10 OHT
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Generate the dataset Here, we have N samples, each sample a 3-tuple ( x , y, p ∗ ) .
To get one sample, we ﬁrst select the label y
=
k following an uniform distribution over all K classes.
After that, we sample the input signal x | y =
k ∼ N ( µ k , σ 2 I ) , where σ is the noisy level for all the samples.
µ k is the mean vector for all the samples in class k .
Each µ k is a 30-dim vector, in which each dimension is randomly selected from {− δ µ , 0 , δ µ } .
Such a process is similar to selecting 30 different features for each class.
Finally, we calculate the true Bayesian probability of this sample, i.e., p ∗ ( y | x ) .
Calculate the ground truth probability We use the fact that p ∗ ( y | x ) ∝ p ( x | y ) p ( y ) .
As y follows an uniform distribution, we have p ∗ ( y | x )
= p ( x | y = k ) (cid:80) j (cid:54) =
k p ( x | y = j ) .
Following p ( x | y = k ) ∼ N ( µ k , σ 2 I ) , we ﬁnd p ∗ ( y | x ) should have a Softmax form, i.e., p = [Equation] sk
[Equation]
j
[Equation]
= k e sj .
Speciﬁcally, we have:
[Equation]
Setup of experiments in Figure 1 in this experiment, we generate a toy Gaussian dataset with
K = 3 , σ = 2 and N = 10 5 .
To reduce the variance of test error, we make a train/valid/test split with ratio [0.05 0.05, 0.9].
We train an MLP with 3 hidden layers, each with 128 hidden units and ReLU activations.
We ﬁrst conduct experiments on some baseline settings, i.e., learning from one-hot supervision (OHT for short), from smoothed label supervision (LS for short), from a converged teacher’s prediction (KD for short), and from an early-stopped teacher’s prediction (ESKD for short).
In OHT case, an NN is trained under the supervision of e y .
If we train this
NN until the convergence of training accuracy, we obtain the p tar for the KD case.
If we select the snapshot of that NN based on the best validation accuracy, we obtain the p tar for the ESKD case.
If we directly set p tar = 0 .
9 e y +0 .
1 u , where u is an uniform distribution over K classes, we obtain the supervision for LS case.
With different supervisions, we train a new network with identical structure until the validation accuracy no longer increase.
Furthermore, to see a trend between generalization ability and (cid:107) p tar − p ∗ (cid:107) 2 , we run the experiment 200 times under different noisy supervisions.
D T EMPERATURE IN DIFFERENT KD METHODS
Temperature is an important hyper-parameter in different kinds of KD methods, which might inﬂuence the performance a lot.
In this appendix, we will discuss why we prefer τ = 1 .
The role played by τ
In general, the loss function in KD has the form:
[Equation]
) where β ∈ [0 , 1] is another hyper-parameter to trade-off the importance between one-hot label and teacher’s predictions.
Furthermore, for this loss, the gradient of L to logits z is:
[Equation]
Why we use τ = {+
+}
The most important reason for us to choose τ = 1 (as well as β = 1 ), even with the risk of providing sub-optimal performance, is that we want to observe how much enhancement is brought by reﬁning the label.
In our mind, KD’s success comes from the following two aspects:
(a) better label (i.e., p tar is better than e y ); (b) better learning dynamics (soften q to q τ make the training easier).
The ﬁrst one provides the student with more useful knowledge, and the second one helps the student to extract it.
The focus of our paper is the improvement in labels.
When the temperature is not 1, both (a) and (b) are inﬂuenced, as we are using q τ (i.e., the smoothed student output) to match p τ tar (i.e., the smoothed target) during training and using q to inference during testing.
If we ﬁx τ = 1 , the only difference between Filter-KD, ESKD, KD, LS, and OHT training is p tar .
Under such a condition, we believe the fact that Filter-KD/ESKD outperforms OHT (strictly better in each run) is enough to conclude KD methods can provide better labels.
Furthermore, as Filter-KD is proposed to mitigate the high-variance issue in ESKD, and we can directly observe the zig-zag learning path of samples with bad labels, we believe it is reasonable to make the conclusion in the paper. [Figure] s [Figure]
u r
[Figure] c [Figure] Furthermore, using τ (cid:54) = 1 doesn’t substantially change our analysis.
As illustrated in Equation (8), the gradient shows that q τ moves towards p τ tar in each update.
Hence our analysis in Proposition 1 still holds, which means the trade-off between the two forces still exists.
τ = 1 is reasonably good in our settings
In fact, the optimal τ depends on many settings, e.g., teacher’s size, optimizer, learning rate (scheduler), and etc: there is still no consensus on what is the best choice of τ for all settings.
So for the experiments in the main context, we ﬁx τ and ﬁne-tune other hyperparameters to achieve good results.
However, to verify this choice of τ is not terrible, we performed a coarse grid search on both CIFAR and toy examples.
As illustrated in Table 4 and Figure 11, choosing τ ∈ [1 , 2] seems to be good choice: the performance doesn’t signiﬁcantly decay until τ is too large.
Some readers might also curious about why our results don’t match the common notion that the optimal τ should be 4 , which is ﬁrst provided in Hinton et al.
and followed by much later [Figure] work (e.g. Gao et al., 2018; Zagoruyko & Komodakis, 2017; Cho & Hariharan, 2019).
Some works also use a grid search to conclude that a temperature as high as 20 should be the best choice (Yuan et al., 2020;
Kim et al., 2019).
We ﬁnd this mismatch comes from the relative difference between the network size of teacher and student.
In short, when distilling from a large teacher to a small student (as most of the cases in aforementioned works), high τ is preferred.
When conducting selfdistillation, τ need not be that high.
For example, Zhang et al.
and Roth et al.
(2021) claim τ = 1 is the best choice, while Zhang & Sabuncu (2020) claim τ ≈ 2 is the best.
To further verify this, we compare the temperature trend between two cases on the toy dataset.
In Figure 12, the ﬁrst row is distilling a 10-layer, 256-width MLP to a 3-layer, 32-width MLP; the second row is self-distillation between 3-layer, 32-width MLPs.
At the same time, we also run a grid search the smoothing factor α of our Filter-KD in Table 5.
[Table]
E H OW REPRESENTATIVE IS THE ZIG - ZAG PATTERN ?
In the main paper, we only visualize the zig-zag learning path of a few samples in Figures 3 and 5.
The readers might then wonder whether this pattern is representative across the whole dataset.
To verify this, we deﬁne a quantitative metric called zig-zag score .
Speciﬁcally, we ﬁrst calculate the integral of each dimension of the prediction:
[Equation]
We then use the highest Q i among i ∈ { 1 , ..., K } \ { y } , where y is the label’s class, as the zig-zag score.
In other words, we focus on the behavior of q on those dimensions that are not the training label.
If this score is large, we might expect the neighbouring samples exert high inﬂuence on the path, and vice versa.
Note that as we have (cid:80) i Q i = constant for any sample (as q t is a K-simplex), this zig-zag score might correlated with C-score of Jiang et al.
However, their focuses are different: C-score is similar to − Q i =
y and focuses more on how fast the prediction converge to the label; zig-zag score, on the other hand, is more about how much the path deviates towards a different class, possibly the label close to p ∗ .
With this score, we test the correlation between base difﬁculty and zig-zag score on toy datasets under different settings.
From Figure 14, it is safe to conclude that samples with higher base difﬁculty will have a more zig-zagging path during training.
We also compare the expected zig-zag score of the 1000 samples with ﬂipped label in CIFAR10 (recall the experiment in Figure 6).
In
Figure 13, it is clear that the zig-zagness of these wrong label samples (which we are sure have high base difﬁculty) is signiﬁcantly higher than the average.
Finally, we also randomly select some data samples in each class of CIFAR10 and visualize their learning paths.
Figure 15 shows random samples for the noisy label experiment.
It is clear that the learning path of easy samples with correct labels converge fast while that of samples with wrong labels is zig-zag.
In Figure 16, which shows the learning path with high zig-zag score when the training set is clean, we can still observe some samples with zig-zag path.
These samples might be ambiguous, with a quite ﬂat p ∗ .
However, as we do not know the true p ∗ of these samples, it is impossible to provide a result like Figure 13.
F D ISTANCE GAP UNDER DIFFERENT SUPERVISIONS
Figure 2 provides the distance gap between q and corresponding p ∗ for each training sample in different training stages, under the supervision of one-hot label e y .
From the results in this ﬁgure, we notice that the behavior of hard samples contributes more to the success of ESKD.
Here we further visualize how these gaps changes when the model is trained under different types of supervision, i.e., ground truth, LS and ESKD.
In the ﬁrst row, which is the result of label smoothing, we see the (cid:107) p tar − p ∗ (cid:107) 2 (i.e., red dots) has a “V”-shape.
The vertex location depends on the smoothing parameter we choose.
However, as discussed previously, label smoothing has only one parameter to control p tar , which is like using a linear model to ﬁt a high-order function.
So, although a proper smoothing value can bring better supervision than one-hot label, its upper bound might be limited.
Regarding the training dynamics, we can see a similar trend as the results shown in the one-hot case, i.e., all the samples ﬁrst move down and then converge to p tar .
From the middle panel, we might expect label smoothing to outperform the one-hot case, because the scatters are closer to the x-axis, which represents the ground truth p ∗ .
The second and the third rows demonstrate the KD and ESKD case.
Results in the KD case are quite similar to the one-hot case in Figure 2, because the converged p tar is close to e y .
However, we can still expect KD to outperform one-hot training, because the p tar is closer to p ∗ than e y is.
The last panel in this row also demonstrates that (cid:107) q − p ∗ (cid:107) 2 might be smaller than (cid:107) p tar − p ∗ (cid:107) 2 , which can be considered as an explanation for why iterated self-distillation, e.g., Born Again Networks (Furlanello et al., 2018), can improve the performance.
For the ESKD case, we see the overﬁtting almost disappear: the distribution of the blue points do not change too much after the early stopping criterion is satisﬁed.
In the last row, which illustrates the training under the supervision of p ∗ , we ﬁnd all the blue points move toward the x-axis, i.e., their p tar = p ∗ , and ﬁnally converge to it.
There is also no overﬁtting in this case.
From the last two panels, we see the disperse of blue points is signiﬁcantly smaller than all other settings, which means the network’s prediction is quite close to p ∗ .
Hence the performance of this case is the best.
[Figure]
G M ORE ABOUT THE DECOMPOSITION AND NTK MODEL
Proof of Proposition 1.
Recall z ( x ) =
f ( w , x ) is the vector of output logits, and q = Softmax ( z ) is the output probability. We are taking a step of SGD observing x u , and observing the change in predictions on x o . We begin with a Taylor expansion,
[Equation]
To evaluate the leading term
[Equation]
we plug in the deﬁnition of SGD and repeatedly use the chain rule: ∇ w [Equation]
x
[Equation]
For the higher-order term, using as above that [Equation] and noting that, since the vectors are probabilities, (cid:107) p tar ( x u ) − q t ( x u ) (cid:107) is bounded, we have that
[Equation]
In the last line,
[Equation]
which is a symmetric positive semi-deﬁnite (PSD) matrix 8 with trace tr( A t ( x o ))
= 1 − (cid:80)
Ki =
1 q 2 i .
As we have (cid:80) i q i = 1 , it is easy to check the trace of this matrix is larger at the beginning of training (when q tends to be ﬂat) than that at the end of training ( q tends to be peaky), as illustrated by most panels in Figure 18.
Given that the trace of a matrix is the sum of its eigenvalues and A t ( x o ) is PSD, smaller tr( A t ( x o ))
= 1 − (cid:80)
Ki =
1 q 2 i means this matrix will tend to shrink its inputs more.
Hence the change in predictions tends to decrease when q t becomes more peaky.
The second term in that expression, K t ( x o , x u ) , is the outer product of gradients at x o and x u .
Intuitively, if their gradients have similar directions, this matrix is large, and vice versa.
This matrix is known as the empirical neural tangent kernel, and it can change through the course of training as the network’s notion of “similarity” evolves.
For appropriately initialized very wide networks trained with very small learning rates, K t remains almost constant during the course of training, and is almost independent of the initialization of the network parameters; the kernel it converges to is known as the neural tangent kernel (Jacot et al., 2018; Arora et al., 2019).
Learning Dynamics of q ( x o ) In Proposition 1, we decompose q t +1 ( x o ) − q t ( x o ) into three parts; we use this to analyze what the learning path of a training sample might be like in Section 3.3.
Here we will provide more detailed illustration of the two groups of force imposed on q ( x o ) .
Speciﬁcally, q t +1 ( x o ) − q t ( x o ) will be inﬂuenced by two variables, i.e., updating sample x u and time t .
We discuss their effects separately.
For x u , only the last two terms, i.e., K t ( x o , x u ) and ( p tar ( x u ) − q t ( x u )) depends on it.
In Section 3.3, we claim that if x o and x u are similar, the norm of K 0 ( x o , x u ) might be large, and vice versa.
Here we empirically demonstrate this using a toy Gaussian dataset, as illustrated in Figure 18.
The ﬁgure shows how the similarity between x o and x u correlates with tr( K 0 ( x o , x u )) .
Each panel represents one x o , which is claimed in the title of the subﬁgures.
The x-axis represents the rank of the cosine similarity between observed x o and all N training samples (including itself).
The y-axis is the trace of K 0 ( x o , x u ) .
The color of each scatter point is the true label of x u .
From the ﬁgure, we can observe a clear decreasing trend, which means smaller similarity leads to larger tr( K 0 ( x o , x u )) .
Additionally, the term ( p tar ( x u ) − q t ( x u )) provides a direction that q t ( x o ) should move towards.
We claim in Section 3.3 that at any point in the input space, the labels of these input samples might follow the ground truth distribution, i.e., p ∗ ( y | x ) .
Hence most of the neighbouring x might pull q ( x o ) towards its ground truth p ∗ ( y | x o ) .
We will discuss the norm of this term when discussing the inﬂuence of t .
In short, at any time, the neighboring x u will impose stronger force on x o and the direction of the force roughly points to the ground truth p ∗ ( y | x o ) .
As discussed, K t is roughly constant over t in the very-wide limit.
For ﬁnite width networks, however, it adapts to reﬂect the network’s new “understanding” of similarities.
For instance, it might learn that certain types of images are more semantically similar than the randomly-initialized network thought.
This does not fundamentally change our intuitions as long as it doesn’t happen too often, but could potentially lead to more complicated zig-zag patterns as the network’s estimate of p ∗ from neighboring points perhaps improves over time.
Over the course of training, A t ( x o ) and ( p tar ( x u ) − q t ( x u )) will also change.
In practical regimes, none of these terms have an easy analytical expression w.r.t. t : q t is quite complicated.
Thus, we provide some intuition, with experimental veriﬁcation.
In Figure 18, we show how tr( A t ( x o )) depends on q t : ﬂat q t leads to larger trace.
A similar trend also exists in the norm of ( p tar ( x u ) − q t ( x u )) .
As the initialized q tend to be ﬂat, updates of any samples will inﬂuence network’s parameters a lot.
When the training progresses, those easy samples converge fast, so their (cid:107) p tar ( x u ) − q t ( x u ) (cid:107)and tr( A t ) become small.
However, as the q t for the hard sample is still far away from its e y , the large (cid:107) p tar ( x u ) − q t ( x u ) (cid:107) 2 and tr( A t ) will ﬁnally drag q t back towards the one-hot distribution, as illustrated in Figure 3. H C OMPARISON TO L IU ET AL . (2020) AND H UANG ET AL . (2020)
The main claim of this paper is that better supervisory signals can enhance the generalization ability of the trained model. Inspired by the success of KD, we ﬁnd that the neural network can spontaneously reﬁne those “bad” labels during training by observing their learning path. The learning path of those hard samples will ﬁrst move towards their true p ( y | x ) and then converge to their label p tar or e y . We explain why this phenomenon occurs by expanding the gradients of each training sample.
This phenomenon is also explained in Proposition 1, and formally proved for a particular softmax regression model by Liu et al.
As a complement, we propose an explanation using an NTK model, and experimentally verify it by observing the learning path and distance gap during training.
Another difference between these two works is that we consider the problem of reﬁning supervisory signals, while Liu et al.
(2020) consider correcting wrong labels (a special case of “bad” supervision).
Our work provides additional emphasis and empirical study for the clean-label case.
Regarding the algorithm, Liu et al.
(2020) design an effective regularization term inspired by early stopping regularization.
They apply exponential moving average (EMA) on the model’s output when calculating this regularization term to further enhance the performance.
This method is similar to that proposed by Huang et al.
(2020), who switch between optimizing the training loss and an objective based on the EMA of the model over the course of training.
Although it bears signiﬁcant similarity to these methods, Filter-KD does not change the course of training the teacher model.
Rather, we propose (based on the high variance of the zig-zag learning paths) to simply use the the EMA of that model’s outputs as a teacher for later distillation.
We suspect that these three algorithms work because of essentially the same underlying principle, whether we think of this as being based on the zig-zag learning path or as early-stopping regularization.
We expect that this principle will be helpful moving forward in the ﬁeld’s understanding of the learning dynamics of SGD methods for neural networks.
I L OW PASS FILTER ON NETWORK PARAMETERS
In Section 4, we point out the high variance issue of the traditional KD methods after observing the learning path of those hard samples.
We then propose a Filter-KD algorithm to smooth the output of each training sample during training.
Such a low pass ﬁltering method is quite similar to the momentum mechanism used in self-supervised learning, e.g., from the classic method of
Ruppert (1988) to MOCO (He et al., 2020), which conduct low pass ﬁltering on each parameter of the network.
As mentioned before, the proposed Filter-KD algorithm might require more memory when the dataset becomes larger, because q smooth would record every training sample’s prediction.
Conducting low pass ﬁltering on network parameters might be a good way to solve this.
To verify whether this method works, we train a ResNet18 on CIFAR100, using one-hot supervision.
At the beginning of training, we copy the training model and call it tracking model.
At the end of each update (i.e., each batch), the parameters of training model is updated as usual while the tracking model’s parameters are updated with momentum.
Speciﬁcally, for the training model, w t +1 train =
w t train + η ∇ L , for the tracking model, w t +1 track =
(1 − α ) w t track + α w t +1 train .
We train the model for 150 epochs, and show the learning curve of test accuracy in the left panel of Figure 19.
We see the performance of the tracking model converges faster than the training model, which is reasonable because ﬁltering parameters is regularizing the training process.
However, the converging accuracy of this two models are the same.
At the same time, we ﬁnd this tracking model is not as good as q smooth when teaching a new model, as illustrated in the right panel of Figure 19.
As this paper mainly focuses on the behavior of the model’s prediction, the discussion and experiments of ﬁltering in parameter space is limited.
However, as many papers demonstrates the effectiveness of this momentum mechanism, we think it is important to explore the relationship further.
[Table]
[Table]
FilterKD 95.
OHT 78.
[Table]
[Table]
[Table]
KD [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
FilterKD 79.
[Table]