B ETTER S UPERVISORY S IGNALS BY O BSERVING L EARNING P ATHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Better supervision might lead to better performance.
In this paper, we ﬁrst clarify what makes supervision good for a classiﬁcation problem, and then explain two existing label reﬁning methods: label smoothing and knowledge distillation, in terms of our proposed criterion.
To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the network’s predicted distribution, for each training sample during training.
We surprisingly ﬁnd that the model can spontaneously reﬁne “bad” labels with through a “zig-zag” learning path, which occurs on both toy and real datasets.
Observing the learning path not only provides a new perspective for understanding knowledge distillation, overﬁtting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks.
Inspired by this, we propose a new algorithm, Filter-KD, which further enhances classiﬁcation performance of knowledge distillation, as veriﬁed by experiments in various settings.
I NTRODUCTION
In multi-class classiﬁcation problems, we usually supervise our model with “one-hot” labels: label vectors y which have y i = 1 for one i , and 0 for all other dimensions.
Over time, however, it has gradually become clear that this “default” setup is not always the best choice in practice, as measured by the generalization performance of the resulting model.
1 One alternative is to summarize a distribution of human annotations, as Peterson et al.
(2019) did on CIFAR-10.
An alternative approach is label smoothing (e.g. Szegedy et al., 2016), mixing between a one-hot label and the uniform distribution.
Knowledge distillation (KD), ﬁrst training a teacher network on the training set and then a student network on the teacher’s output probabilities, was originally proposed for model compression (Hinton et al., 2015) but can also be thought of as reﬁning the supervision signal: it provides “soft” teacher outputs rather than hard labels to the student.
Knowledge distillation is promising because it requires no additional annotation effort, but, unlike label smoothing, can still provide sample-speciﬁc reﬁnement.
Perhaps surprisingly, knowledge distillation can improve student performance even when the teacher is of exactly the same form as the student ; this is known as self-distillation (Furlanello et al., 2018; Zhang et al., 2019).
There have been many recent attempts to explain the success of knowledge distillation and speciﬁcally self-distillation (e.g. Menon et al., 2021; Allen-Zhu & Li, 2020; Tang et al., 2020), from both optimization and supervision perspectives.
We focus on the latter area, where it is usually claimed that the teacher provides useful “dark knowledge” to the student through its labels.
Inspired by this line of work, we further explore why and how this improved supervisory signal emerges during the teacher’s one-hot training.
Speciﬁcally, we ﬁrst clarify that given any input sample x , good supervision signals should be close (in L 2 distance) to the ground truth categorical distribution, i.e., p ∗ ( y | x ) .
We then show that a neural network (NN) can automatically reﬁne “bad labels”, those where p ∗ ( y | x ) is far from the one-hot vector from the training set.
During one-hot training, the model prediction on such a sample ﬁrst moves towards p ∗ ( y | x ) , and then slowly converges to its supervisory label, following a “zig-zag” pattern.
A well-trained teacher, one that does not overﬁt to particular training labels, can thus provide supervisory signals closer to p ∗ ( y | x ) .
We ﬁnd this phenomenon is quite common in gradient descent training, and provide some analytical justiﬁcation (Section 3.3).
Based on our explanations, we point out that this signal can be better achieved if we take a moving average of the teacher’s prediction, an algorithm we term Filter-KD.
This approach provides better supervision and hence better ﬁnal performance, especially when there are more bad labels.
S UPERVISION I NFLUENCES G ENERALIZATION
We will begin by establishing the classiﬁcation setting, and making clearer how the choice of supervisory signal affects the learned model.
2. C HOICES OF S UPERVISION S IGNAL
In K -way classiﬁcation, our goal is to learn a mapping f : X → ∆ K that can minimize the risk
[Equation]
where L ( y, f ( x )) is the loss function measuring the gap between model’s prediction and the true label (e.g. cross-entropy or square loss).
The true label y is an integer ranging from 1 to K , and the input signal x is usually high dimensional, e.g., an image or a sequence of word embeddings.
The joint distribution of ( x , y ) is P , whose density 3 can be written as p ( x ) p
( y | x ) .
In practice, as P is unknown, we instead (approximately) minimize the empirical risk
[Equation]
where 1 ( y n = k ) is an indicator function which equals 1 if y n = k or 0 otherwise, and e y n ∈ { 0 , 1 } K is its one-hot vector form.
L ( f ( x n ))
=
( L (1 , f ( x n )) , . . .
, L ( K, f ( x n )))
∈ R K is the loss for each possible label.
In R emp , the N training pairs D (cid:44) { ( x n , y n ) }
Nn =1 are sampled i.i.d.
from P .
Comparing (2) to (1), we ﬁnd that p ( x ) is approximated by an uniform distribution over the samples, which is reasonable.
However, using an indicator function (i.e., one-hot distribution) to approximate p ( y | x ) bears more consideration.
For example, if a data point x is quite vague and its true p ( y | x ) is ﬂat or multimodal, we might hope to see x multiple times with different label y during training.
But actually, most datasets have only one copy of each x , so we only ever see one corresponding e y .
Although R emp is an unbiased estimator for R , if we used a better (e.g. lower-variance) estimate of p ( y | x ) , we could get a better estimate for R and thus, hopefully, better generalization.
Speciﬁcally, suppose we were provided a “target” distribution p tar ( y | x ) (written in vector form as p tar ( x ) ) for each training point x , as D (cid:48)
= { ( x n , p tar ( x n ))
} Nn =1 .
Then we could use
R tar ( f, D (cid:48) )
(cid:44)
[Equation]
Standard training with R emp is a special case of R tar , using p tar ( x n ) =
e y n .
The CIFAR-10H dataset (Peterson et al., 2019) is one attempt at a different p tar , using multiple human annotators to estimate p tar .
Label smoothing (e.g. Szegedy et al., 2016) sets p tar to a convex combination of e y and the constant vector 1 K 1 .
In knowledge distillation (KD; Hinton et al., 2015), a teacher is ﬁrst trained on
D , then a student learns from D (cid:48) with p tar based on the teacher’s outputs.
All three approaches yield improvements over standard training with R emp .
[Figure]
M EASURING THE Q UALITY OF S UPERVISION
Choosing a different p tar , then, can lead to a better ﬁnal model.
Can we characterize which p tar will do well?
We propose the following, as a general trend.
Hypothesis 1.
Suppose we train a model supervised by p tar , that is, we minimize R tar ( f, D (cid:48) ) .
Then, smaller average L 2 distance between p tar and the ground truth p ∗ on these samples, i.e. small E x [ (cid:107) p tar ( x ) − p ∗ ( x ) (cid:107) 2 ] , will in general lead to better generalization performance.
This hypothesis is suggested by Proposition 3 of Menon et al.
(2021), which shows that for any predictor f and loss bounded as L ( y, ˆ y ) ≤ (cid:96) ,
[Equation] [Equation]
When N is large, the second term will dominate the right-hand side, implying smaller average (cid:107) p tar − p ∗ (cid:107) will lead to R tar being a better approximation of the true risk R .
Minimizing a better
R tar should then lead to a better model, suggesting that the quality of the supervision signal can be roughly measured by its L 2 distance to the ground truth p ∗ .
To further support this hypothesis, we conduct experiments on a synthetic Gaussian problem (Figure 1 (a); details in Appendix B), where we can easily calculate p ∗ ( y | x ) for each sample.
We ﬁrst generate several different p tar by adding noise 4 to the ground truth p ∗ , then train simple 3-layer
NNs under that supervision.
We also show ﬁve baselines: one-hot training (OHT), label smoothing (LS), KD, early-stopped KD (ESKD), and ground truth (GT) supervision (using p ∗ ).
KD refers to a teacher trained to convergence, while ESKD uses a teacher stopped early based on validation accuracy.
We early-stop the student’s training in all settings.
From Figure 1 (b-c), it is clear that smaller (cid:107) p tar − p ∗ (cid:107) 2 leads to better generalization performance, as measured either by accuracy (ACC) or expected calibration error (ECE) 5 on a held-out test set.
Appendix A has more detailed results.
I NSIGHTS F ROM THE L EARNING P ATH
In the toy example of Section 2, we see that ESKD outperforms other baselines in accuracy by a substantial margin (and all baselines are roughly tied in ECE).
We expect that supervision with smaller (cid:107) p tar − p ∗ (cid:107) 2 leads to better generalization performance, but it is not clear how better p tar emerges from when the teacher in ESKD is trained using one-hot labels.
This section will answer this, by observing the learning paths of training samples.
3. P AY M ORE A TTENTION TO H ARDER S AMPLES
For a ﬁner-grained understanding of early stopping the teacher, we would like to better understand how the teacher’s predictions evolve in training.
Assuming Hypothesis 1, the main factor is E x (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 , where q is the teacher’s output probability distribution.
We expect, though, that this term will vary for different x , in part because some samples are simply more difﬁcult to learn; as a proxy for this, we deﬁne base difﬁculty as (cid:107) e y − p ∗ ( x ) (cid:107) 2 .
This measure is high if:
• x is ambiguous: p ∗ has several large components, so
is
one-hot label near p ∗ .
• x is not very ambiguous (there is a one-hot label near p ∗ ), but the sample was “unlucky” and drew y from a low-probability class.
Figure 2 shows these two quantities at three points in training: initialization, the point where ESKD’s teacher stops, and convergence.
At initialization, most points 6 have large (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 .
By the point of early stopping, most q ( x ) values are roughly near p ∗ .
At convergence, however, q ( x ) ≈ e y , as the classiﬁer has nearly memorized the training inputs, leading to a diagonal line in the plot.
It is clear the biggest contributors to E x (cid:107) q ( x ) − p ∗ ( x ) (cid:107) 2 when training a model to convergence are the points with high base difﬁculty.
Per Hypothesis 1, these are the points we should most focus on.
3. L EARNING P ATH OF D IFFERENT S AMPLES
To better understand how q changes for each sample, we track the model’s outputs of all training samples at each training step.
We plot the vectors of three probabilities as points on the simplex (details in Appendix A), and visualize four samples with different base difﬁculty in Figure 3.
The easy samples very quickly move to the correct location near e y (as indicated by the light color until reaching the corner).
The medium sample takes a less direct route, drifting off slightly towards p ∗ , but still directly approaches e y .
The hard sample, however, does something very different: it ﬁrst approaches p ∗ , but then veers off towards e y , giving a “zig-zag” path.
In both the medium and hard cases, there seems to be some “unknown force” dragging the learning path towards p ∗ ; in both cases, the early stopping point is not quite perfect, but is noticeably closer to p ∗ than the ﬁnal converged point near e y .
In other words, during one-hot training, the NNs can spontaneously reﬁne the “bad labels.”
Under Hypothesis 1, this partly explains the superior performance of ESKD to
KD with a converged teacher.
We only show four points, because they are quite representative of all samples, under different toydataset settings.
We also provide a “zig-zagness score” to further illustrate the correlation between base difﬁculty and the learning path patterns (refer to Appendix D for more details).
3. E XPLANATION OF PATTERNS IN THE LEARNING PATH
Where does this “unknown force” pushing q towards p ∗ come from?
The following decomposition, proved in Appendix G, will help elucidate its source:
Proposition 1.
Let z t ( x ) (cid:44) f
( w t , x ) denote the NN output logits with parameters w t , and q t ( x ) =
Softmax( z t ( x ))
the corresponding probabilities.
Let w t +1 (cid:44) w t − η ∇ w (cid:0) p tar ( x u ) T L ( q t ( x u )) (cid:1) be the result of applying one step of SGD to w t on the data point ( x u , p tar ( x u )) with learning rate η .
Then, the change in predictions for a particular observed sample, x o , is
[Equation]
where A t ( x o )
= ∇ z q t ( x o ) and K t ( x o , x u ) =
( ∇ w z ( x o ) | w t ) ( ∇ w z ( x u ) | w t ) T are K × K matrices.
The matrix K t is known as the empirical neural tangent kernel , which we can think of roughly as a notion of “similarity” between x o and x u based on the network’s representation at time t .
In general, K t changes during training in potentially complex ways, but in very wide networks with appropriate random initialization, it is nearly invariant throughout training and nearly independent of the initialization (Jacot et al., 2018; Arora et al., 2019).
For more details, see Appendix G.
Figure 4 shows the learning path of a hard sample during one-hot training, say x o , where the label e y o is far from p ∗ ( x o ) .
In each epoch, w will receive N − 1 updates based on the loss of x u (cid:54) =
x o (the small blue path), and one update based on x o (the big red path).
At any time t , “dissimilar” samples will have small K t ( x o , x u ) (as measured, e.g., by its trace), and hence only slightly affect the predicted label for x o .
Similar samples will have large K t ( x o , x u ) , and hence affect its updates much more; because p ∗ is hopefully similar for similar x values, it is reasonable to expect that the mean of e y for data points with similar x will be close to p ∗ ( x o ) .
Thus the net effect of updates for x u (cid:54) =
x o should be to drag q ( x o ) towards p ∗ ( x o ) .
This is the “unknown force” we observed earlier.
The other force affecting q ( x o ) during training is, of course, the update based on x o , driving q ( x o ) towards e y o .
How do these two forces balance over the course training?
Early on, (cid:107) e y u
−
q
t
( x u ) (cid:107) 2 is relatively large for nearly any x u , because near initialization q t ( x u ) will be relatively ﬂat.
Hence the size of the updates for “similar” x u and the update for x o should be comparable, meaning that, if there are at least a few “similar” training points, q t ( x o ) will move towards p ∗ ( x o ) .
Throughout training, some of these similar samples will become well-classiﬁed, so that (cid:107) e y u − q t ( x u ) (cid:107)becomes small, and their updates will no longer exert much force on q ( x o ) .
Thus, the x o updates begin to dominate, causing the zig-zag pattern as the learning path turns towards e y o .
For easy samples, where p ∗ and e y o are in the same direction, these forces agree and lead to fast convergence.
On samples like the “medium” point in Figure 3, the two forces broadly agree early on, but the learning path deviates slightly towards p ∗ en route to e y o .
L EARNING P ATHS O N R EAL T ASKS
Our analysis in Sections 2 and 3 demonstrate that the model can spontaneously reﬁne the bad labels during one-hot training: the zig-zag learning path ﬁrst moves towards the unknown p ∗ .
But is this also true on real data, for more complex network architectures?
We will now show that they do, although seeing the patterns requires a little more subtlety.
We then propose an algorithm called
Filter-KD to take further advantage of these observations for improved generalization.
To verify this explanation on real tasks – we use CIFAR10 and CIFAR100 as examples – the ﬁrst conundrum is that we can no longer calculate the base difﬁculty (cid:107) e y − p ∗ ( x ) (cid:107) 2 of a sample, because p ∗ is unknown.
Furthermore, as the labels of CIFAR dataset are relatively clean, we cannot reliably ﬁnd many samples with bad labels to see whether the pattern is representative.
Recall that there are two cases for a label to be bad: ambiguous sample with ﬂat p ∗ , and unambiguous samples with “wrong” labels.
It might be more persuasive to see that the model can spontaneously “correct” those wrong labels by early stopping at the inﬂection of the zig-zagging path.
Thus, we choose a random 1,000 of the 50,000 training samples, and ﬂip their labels to a random y (cid:48) (cid:54) =
y .
With these changes, we visualize the learning path of data points while training a ResNet-18 (He et al., 2016) on CIFAR10 for 200 epochs as an example.
The ﬁrst panel of Figure 5 shows the learning path of an easy sample with good label.
We can see that q t converges quickly towards the left corner, the one-hot distribution for the correct class, because the color of the scattered points in that ﬁgure are quite light.
At the early stopping epoch, q t has already converged to e y .
However, for a sample with a bad label, it is very difﬁcult to observe any patterns from the raw path (blue points): there are points almost everywhere in this plot.
This is likely caused by the more complex network and dataset, as well as high learning rate in early training.
To ﬁnd the hidden pattern in this high-variance path, we treat q t as a time series signal with many high-frequency components.
Thus we can collect its low-frequency components via a low-pass ﬁlter and then draw it, i.e., taking a moving average of the q t (red points).
This makes it clear that, overall, the pattern zig-zags, ﬁrst moving towards the unknown true label before eventually turning to memorize the wrong label.
[Figure]
This smoothing method not only helps us observe the zig-zag pattern, but also improves the predictions.
Figure 6 tracks how often the most likely prediction of the network, argmax q t , recovers the true original label, rather than the provided random label, for the corrupted data points.
At the beginning of the model’s training, the initialized model randomly guesses labels, getting about 10% of the 1,000 ﬂipped labels correct.
During training, the model spontaneously corrects some predictions, as the learning path ﬁrst moves towards p ∗ .
Eventually, though, the model memorizes all its training labels.
Training with the predictions from the 400th epoch corresponds to standard KD (no corrected points).
Early stopping as in ESKD would choose a point with around 700 corrected labels.
The smoothed path, though, performs best, with over 800 corrections.
Appendix E.
S MOOTHING AS A M ETHOD FOR K NOWLEDGE D ISTILLATION
Figure 6 gives a clear motivation for a new knowledge distillation method, which we call Filter-KD (Algorithm 1): train the student from the smoothed predictions of the teacher network.
Speciﬁcally, we maintain a look-up table q smooth ∈ R N × K to store a moving average of q t for each training sample.
Note that in one epoch, each q smooth ( x n ) will be updated only once.
We check the early stopping criterion with the help of a validation set.
Afterwards, the teaching supervision q smooth is ready, and we can train a student network under its supervision.
This corresponds to using a moving average of the teacher model “in function space,” i.e. averaging the outputs of the function over time.
Compared to ESKD, Filter-KD can avoid the extremely high variance of q t during training.
Note that unlike Figure 5 which plots q t after every iteration of training, most practical implementations only consider early-stopping at the end of each epoch.
This is equivalent to down-sampling the noisy learning path (as in Figure 5 (c)), further exacerbating the variance of q t .
Thus ESKD will likely select a bad p tar for many data points.
Filter-KD, by contrast, has much more stable predictions.
We will show the effectiveness of this algorithm shortly, but ﬁrst we discuss its limitations.
Compared to ESKD, the running time of Filter-KD might be slightly increased (but not too much), because we only do a few extra multiplications and summations for each batch.
More importantly, compared to the teaching model in ESKD, the Filter-KD requires a teaching table q smooth , which will require more memory when the dataset is large.
One alternative avoiding the need for this table would be to instead take an average “in parameter space,” like e.g. the momentum parameters updating as in Ruppert (1988); Szegedy et al.
; He et al.
We empirically ﬁnd that, although this helps the model converge faster, it does not lead to a better teacher network; see Appendix H.
Thus, although Filter-KD has clear drawbacks, we hope that our explanations here may lead to better practical algorithms for large datasets in the future.
Input: Dataset { ( x n , y n ) } Nn =1 , where I n = { 1 , 2 , ..., N } is the index for each pair
# Train the teacher
Initialize a network model, initialize an N × K matrix called q smooth
Go through the entire dataset, calculate q smooth [ n ] = Softmax(model( x n )) for epoch
= 1 , 2 , ..., T do for n ∈ { 1 , 2 , . . .
, N } in random order do ˆ p =
Softmax(model( x n ))
q smooth [ n ] =
(1 − α ) · q smooth [ n ] + α · ˆ p loss =
CrossEntropy( ˆ p, y n )
Backpropagate and update the parameters end for
Check the early stopping criterion; stop training if satisﬁed end for # Train the student
For each input x n , set p tar = q smooth [ n ] ; train the network under the supervision of p tar
Algorithm 1: Filter-KD.
α controls the cut-off frequency of low-pass ﬁlter (0.05 here).
5. Q UANTITATIVE R ESULTS OF F ILTER -KD
We now compare the performance of Filter-KD and other baselines on a real dataset.
We focus on self-distillation and a ﬁxed temperature τ = 1 (except Table 2), as we want the only difference among these methods to be p tar .
Thus we can conclude the improvement we achieved purely comes from the reﬁned supervision.
Appendix C for discussion and results related to the choice of temperature; τ = 1 is reasonably good for our current setting.
The ﬁrst task we consider is noisy-label classiﬁcation, where we train and validate the model on a dataset with some labels randomly ﬂipped.
After training, all the models are evaluated on a clean held-out test set.
The experiments are conducted on CIFAR (Figure 7) and TinyImageNet (Table 1), under different noise ratios σ : σ = 0 .
1 means 10% of the labels are ﬂipped.
In Figure 7, an interesting trend can be observed: the enhancement brought by Filter-KD is not signiﬁcant when σ is too small or too large.
That is reasonable because for small σ , the few samples have bad labels,
0.8 Noisy ratio 0.
T e s t a cc u r a c y
CIFAR
OHTESKDFilter 0.
0.8 Noisy ratio 0.
T e s t a cc u r a c y
CIFAR
OHTESKDFilter
[Table] [Table]
thus the possible enhancement might not be large.
When σ is too high, the labels of similar points become less reliable, and the learning path will no longer head as reliably towards p ∗ .
Thus, for very high noise ratios, the performance of Filter-KD decays back towards that of OHT.
Filter-KD also outperforms other methods in both accuracy and calibration on the clean dataset, as illustrated in Table 3.
This remains true in the more common KD case when the teacher network is bigger than the student; see results in Table 2.
Note that in Table 2, we no longer keep τ = 1 , as the baselines are more sensitive to τ when teacher is large (see Appendix C for more discussion); instead we optimize τ for each setting.
Here “Efﬁ” is short for EfﬁcientNet-B1 (Tan & Le, 2019), “Res” is short for ResNet18, “Mobi” is short for MobileNetV2 (Sandler et al., 2018), “VGG” is short for VGG-11 (Simonyan & Zisserman, 2015).
[Table]
R ELATED W ORK
Human label reﬁnement.
Peterson et al.
(2019) use the distribution of annotations from multiple annotators to replace the one-hot label in training, and ﬁnd that doing so enhances both the generalization performance of trained models and their robustness under adversarial attacks.
However, this approach is relatively expensive, as it requires more annotation effort than typical dataset creation techniques; several experienced annotators are required to get a good estimate of p ∗ .
It is also unclear in general how well the distribution of decisions by human annotators corresponds with the true variability under the data distribution.
Appendix E has experiments based on this dataset.
Label smoothing.
Another popular method is label smoothing (Szegedy et al., 2016), discussed in the previous sections.
We believe that suitable smoothing can decrease (cid:107) p tar − p ∗ (cid:107) 2 to some extent, but the lower bound of this gap should be large, because label smoothing treats each class and each sample in the same way.
Much prior research (e.g. M¨uller et al., 2019) shows that KD usually outperforms label smoothing as well.
KD and ESKD.
Knowledge distillation, by contrast, can provide a different p tar for each training sample.
KD was ﬁrst proposed by Hinton et al.
(2015) for model compression, under the assumption that it would help to have a more complex teacher extract more information from the dataset.
Later,
Furlanello et al.
; Cho & Hariharan (2019); Zhang et al.
; Yuan et al.
(2020) found that distillation can bring enhancement even when teacher and student have identical structure.
These surprising ﬁndings sparked studies of self-distillation; our work applies to both self-distillation and larger-teacher cases.
Another active direction in research about KD is ﬁnding explanations for its success, which is often still considered somewhat mysterious.
Tang et al.
(2020) decompose the “dark knowledge” provided by the teacher into three parts: uniform, intra-class and inter-class.
This explanation also overlaps with ours: if we observe the samples with different difﬁculty in the same class, we are discussing intra-class knowledge; if we observe samples with two semantically similar classes, we then have inter-class knowledge.
Among previous work, the most relevant study is that of Menon et al.
(2021), who proposes a statistical explanation of KD.
Our Hypothesis 1 is suggested by the main claim of that paper, which focuses on the question of what is a good p tar .
Our work builds off of theirs by helping explain why this good p tar emerges, and how to ﬁnd a better one .
Besides, by looking deeper into the learning path of samples with different difﬁculty, our work might shed more light on KD, anti-noisy learning, supervised learning, overﬁtting, etc.
Noisy label task.
Learning useful information from noisy labels is a long-standing task (Angluin & Laird, 1988; Natarajan et al., 2013).
There are various ways to cope with it; for instance, Menon et al.
(2019) use gradient clipping, Patrini et al.
(2017) use loss correction, Huang et al.
(2020) change the supervision during training, and Zhang et al.
(2020) take advantage of extra information to achieve good performance.
It is possible to combine KD-based methods with traditional ones to further enhance performance; our perspective of learning paths also provides further insight into why KD can help in this setting.
D ISCUSSION
In this paper, we ﬁrst claim that better supervision signal, i.e., p tar that is closer to p ∗ , leads to better generalization performance; this is supported by results of Menon et al.
(2021) and further empirical suggestions given here.
Based on this hypothesis, we explain why LS and KD outperform one-hot training using experiments on a toy Gaussian dataset.
To further understand how such better supervision emerges, we directly observe the behavior of samples with different difﬁculty by projecting them on a 2-D plane.
The observed zig-zag pattern is well-explained by considering the tradeoff between two forces, one pushing the prediction to be near that of similar inputs, the other pushing the prediction towards its training label; we give an intuitive account based on Propositionof why this leads to the observed zig-zag pattern.
To apply these ﬁndings on real tasks, in which the data and network are more complex, we conduct low-pass ﬁlter on the learning path, and propose Filter-KD to further enhance p tar .
Experimental results on various settings (datasets, network structures, and label noise) not only show the advantage of the proposed method as a method for a teacher in knowledge distillation methods, but also help verify our analysis of how generalization and supervision work in training neural network classiﬁers.
A M ISCELLANEOUS B ACKGROUND
Calculation of ECE: expected calibration error is a measurement about how well the predicted conﬁdence represent the true correctness likelihood (Guo et al., 2017).
For example, if our model gives 100 predictions, each with conﬁdence, say, q ( y = k | x ) ∈ [0 . 7 , 0 . 8] ).
Then we might expect there are 70 ∼ 80 correct predictions among those 100 ones.
To calculate this, we ﬁrst uniformly divide [0 , 1] into M bins, with each bin represented by I m ∈ (cid:0) m − 1 M , mM (cid:3) and B m is all the x samples whose conﬁdence falls into I m . Then, ECE is calculated as: [Equation] where acc ( B m ) =
[Equation]
| B m | (cid:80)
[Equation]
, k i is the true label and ˆ y i = argmax [ q ( y | x i )] is the model’s prediction.
All the ECE mentioned in this paper is calculated by setting M = 10 .
Barycentric coordinate system: when visualizing the learning path, it is problematic to directly choose two dimensions and draw them in the Cartesian coordinate system, as illustrated in the ﬁrst panel in Figure 8.
In geometry, a suitable way to project a 3-simplex vector onto a 2-D plane is converting it to a point in a barycentric coordinate system.
Speciﬁcally, we have three basis vectors: v 0 =
[0 , 0] T , v 1 =
[1 , 0] T , and v 2 = (cid:104) 12 , √ 32 (cid:105) T , which are the corner and two edges of an equilateral triangle respectively.
Then the 2D-coordinate is calculated as ( x, y ) =
[ v 0 ; v 1 ; v 2 ] · q .
So every points in the left corner of a Cartesian system plane can be converted to the triangle in a barycentric system, as illustrated in the last panel in Figure 8.
B T OY G AUSSIAN D ATASET
In Sections 2 and 3, we apply a toy Gaussian dataset to verify two facts derived from Hypothesis 1, so as the learning path of speciﬁc samples.
Here we provide more details about this dataset.
Generate the dataset: here, we have N samples, each sample a 3-tuple ( x , y, p ∗ ) .
To get one sample, we ﬁrst select the label y
=
k following an uniform distribution over all K classes.
After that, we sample the input signal x | y =
k ∼ N ( µ k , σ 2 I ) , where σ is the noisy level for all the samples.
µ k is the mean vector for all the samples in class k .
Each µ k is a 30-dim vector, in which each dimension is randomly selected from {− δ µ , 0 , δ µ } .
Such a process is similar to selecting 30 different features for each class.
Finally, we calculate the true Bayesian probability of this sample, i.e., p ∗ ( y | x ) .
Calculate the ground truth probability: we use the fact that p ∗ ( y | x ) ∝ p ( x | y ) p ( y ) .
As y follows an uniform distribution, we have p ∗ ( y | x ) =
[Equation]
= k p ( x | y = j ) .
Following p ( x | y = k ) ∼ N ( µ k , σ 2 I ) , we ﬁnd p ∗ ( y | x ) should have a Softmax form, i.e., p =
[Equation]
Speciﬁcally, we have:
[Equation]
Setup of experiments in Figure 1 : in this experiment, we generate a toy Gaussian dataset with
K = 3 , σ = 2 and N = 10 5 .
To reduce the variance of test error, we make a train/valid/test split with ratio [0.05 0.05, 0.9].
We train an MLP with 3 hidden layers, each with 128 hidden units and ReLU activations.
We ﬁrst conduct experiments on some baseline settings, i.e., learning from one-hot supervision (OHT for short), from smoothed label supervision (LS for short), from a converged teacher’s prediction (KD for short), and from an early-stopped teacher’s prediction (ESKD for short).
In OHT case, an NN is trained under the supervision of e y .
If we train this
NN until the convergence of training accuracy, we obtain the p tar for the KD case.
If we select the snapshot of that NN based on the best validation accuracy, we obtain the p tar for the ESKD case.
If we directly set p tar = 0 .
9 e y +0 .
1 u , where u is an uniform distribution over K classes, we obtain the supervision for LS case.
With different supervisions, we train a new network with identical structure until the validation accuracy no longer increase.
Furthermore, to see a trend between generalization ability and (cid:107) p tar − p ∗ (cid:107) 2 , we run the experiment 200 times under different noisy supervisions.
C T EMPERATURE IN DIFFERENT KD METHODS
Temperature is an important hyper-parameter in different kinds of KD methods, which might inﬂuence the performance a lot.
In this appendix, we will discuss why we prefer τ = 1 , and why this is a reasonably good empirical choice in our settings.
The role played by τ : In general, the loss function in KD has the form: [Equation] [Equation]
where β ∈ [0 , 1] is another hyper-parameter to trade-off the importance between one-hot label and teacher’s predictions. Furthermore, for this loss, the gradient of L to logits z is: [Equation]
Why we use τ = 1 :
The most important reason for us to choose τ = 1 (as well as β = 1 ), even with the risk of providing sub-optimal performance, is that we want to observe how much enhancement is brought by reﬁning the label. In our mind, KD’s success comes from the following two aspects: Run Run
[Table] τ [Table] τ
[Table]
τ = τ = τ =
[Table]
τ
[Table]
τ = τ [Table] τ = [Table]
τ
[Table]
τ
[Table]
τ =10 OHT 94.
[Table]
- - - ESKD
[Table]
FilterKD
[Table]
OHT
[Table]
[Table] [Table]
[Table]
FilterKD
[Table]
(a) better label (i.e., p tar is better than e y ); (b) better learning dynamics (soften q to q τ make the training easier)
The ﬁrst one provides the student with more useful knowledge, and the second one helps the student to extract it.
The focus of our paper is the improvement in labels.
When the temperature is not 1, both (a) and (b) are inﬂuenced, as we are using q τ (i.e., the smoothed student output) to match p τ tar (i.e., the smoothed target) during training and using q to inference during testing.
If we ﬁx τ = 1 , the only difference between Filter-KD, ESKD, KD, LS, and OHT training is p tar .
Under such a condition, we believe the fact that Filter-KD/ESKD outperforms OHT (strictly better in each run) is enough to conclude KD methods can provide better labels.
Furthermore, as Filter-KD is proposed to mitigate the high-variance issue in ESKD, and we can directly observe the zig-zag learning path of samples with bad labels, we believe it is reasonable to make the conclusion in the paper.
Furthermore, using τ (cid:54) = 1 doesn’t substantially change our analysis.
As illustrated in Equation (8), the gradient shows that q τ moves towards p τ tar in each update.
Hence our analysis in propositionstill holds, which means the trade-off between the two forces still exists.
But considering τ (cid:54) =makes the explanation awkward because the prediction of the model is q , not q τ : readers might feel confused why we visualize q and p tar in the learning path but in fact, the gradient comes from q τ and p τ tar .
If we directly visualize q τ and p τ tar , how could we read the value of q from the ﬁgure?
So, to make the paper easier to follow, we prefer τ = 1 .
τ = 1 is reasonably good in our settings:
In fact, the optimal τ depends on many settings, e.g., teacher’s size, optimizer, learning rate (scheduler), and etc: there is still no consensus on what is the best choice of τ for all settings.
So for the experiments in the main context, we ﬁx τ and ﬁne-tune other hyper-parameters to achieve good results.
However, to verify this choice of τ is not terrible, we performed a coarse grid search on both
CIFAR and toy examples.
As illustrated in Table 4 and Figure 11, choosing τ ∈ [1 , 2] seems to be good choice: the performance doesn’t signiﬁcantly decay until τ is too large.
[Figure]
Some readers might also curious about why our results don’t match the common notion that the optimal τ should be 4 , which is ﬁrst provided in Hinton et al.
and followed by much later work (Gao et al., 2018; Zagoruyko & Komodakis, 2017; Cho & Hariharan, 2019, e.g.).
Some works also use a grid search to conclude that a temperature as high as 20 should be the best choice (Yuan et al., 2020;
Kim et al., 2019).
We ﬁnd this mismatch comes from the relative difference between the network size of teacher and student.
In short, when distilling from a large teacher to a small student (as most of the cases in aforementioned works), high τ is preferred.
When conducting self-distillation, τ need not be that high.
For example, Zhang et al.
and Roth et al.
(2021) claim τ = 1 is the best choice, while Zhang & Sabuncu (2020) claim τ ≈ 2 is the best.
To further verify this, we compare the temperature trend between two cases on the toy dataset.
In Figure 12, the ﬁrst row is distilling a 10-layer, 256-width MLP to a 3-layer, 32-width MLP; the second row is self-distillation between 3-layer, 32-width MLPs.
At the same time, we also run a grid search the smoothing factor α of our Filter-KD in Table 5.
[Table]
D H OW R EPRESENTATIVE IS THE Z IG -Z AG P ATTERN ?
In the main paper, we only visualize the zig-zag learning path of a few samples in Figures 3 and 5.
The readers might then wonder whether this pattern is representative across the whole dataset.
To verify this, we deﬁne a quantitative metric called zig-zag score .
Speciﬁcally, we ﬁrst calculate the integral of each dimension of the prediction:
[Equation]
We then use the highest Q i among i ∈ { 1 , ..., K } \ { y } , where y is the label’s class, as the zig-zag score.
In other words, we focus on the behavior of q on those dimensions that are not the training label.
If this score is large, we might expect the neighbouring samples exert high inﬂuence on the path, and vice versa.
Note that as we have (cid:80) i Q i = constant for any sample (as q t is a K-simplex), this zig-zag score might correlated with C-score of Jiang et al.
However, their focuses are different: C-score is similar to − Q i =
y and focuses more on how fast the prediction converge to the label; zig-zag score, on the other hand, is more about how much the path deviates towards a different class, possibly the label close to p ∗ .
With this score, we test the correlation between base difﬁculty and zig-zag score on toy datasets under different settings.
From Figure 14, it is safe to conclude that samples with higher base difﬁculty will have a more zig-zagging path during training.
We also compare the expected zig-zag score of the 1000 samples with ﬂipped label in CIFAR10 (recall the experiment in Figure 6).
In Figure 13, it is clear that the zig-zagness of these wrong label samples (which we are quite have high base difﬁculty) is signiﬁcantly higher than the average.
Finally, we also randomly select some data samples in each class of CIFAR10 and visualize their learning paths.
Figure 15 shows random samples for the noisy label experiment.
It is clear that the learning path of easy samples with correct labels converge fast while that of samples with wrong labels is zig-zag.
In Figure 16, which shows the learning path with high zig-zag score when the [Figure] training set is clean, we can still observe some samples with zig-zag path.
These samples might be ambiguous, with a quite ﬂat p ∗ .
However, as we do not know the true p ∗ of these samples, it is impossible to provide a result like Figure 13.
E CIFAR10H
As previously mentioned, Peterson et al.
(2019) propose a CIFAR10H dataset to reﬁne the one-hot label by human annotators.
However, we decided not to use this baseline in the main body of the paper, for the following reasons:
• CIFAR10H only annotates the test set of CIFAR10, making it hard to compare with other standard baselines; • We cannot control the difﬁculty of the task like we can for the toy dataset; • We believe the labels provided by human annotators is not a perfect estimate of p ∗ .
The ﬁrst two reasons are straightforward.
For the third: recall that from our deﬁnition, p ∗ (cid:44) p ∗ ( y | x ) is related to the generating process of the image.
Although Peterson et al.
(2019) show that training using this label can bring enhancement and hence make the prediction more robust, we still believe that human annotators do not correspond to the true labeling process in this case; labels from CIFAR10H are far better than one-hot, but not perfect.
To verify this, we conduct the following two experiments.
First, we add noise to the labels of CIFAR10H and use them to train a student network.
We then evaluate these models on the original CIFAR10 training set; Peterson et al.
(2019) also use this strategy to evaluate.
Results are provided in the ﬁrst two panels of Figure 17, similar to Figure 9 for the toy dataset.
For the ﬁrst panel, when the added noise are high enough to ﬂip some labels, we see a clear negative correlation between (cid:107) p tar − p cifarh (cid:107) 2 and test accuracy.
However, when we add small amount of noise to p cifarh , as in the second panel in the ﬁgure, the test accuracy can increase substantially.
This means p cifarh might be a little conservative when estimating p ∗ : adding noise and hence making p tar smoother can enhance the generalization ability.
In the toy dataset, where p ∗ is precisely the generating probability, learning from p ∗ always performs the best.
Furthermore, we also compare the supervisions provided by Fitler-KD and p cifarh .
As illustrated in the last panel in Figure 17, Filter-KD can achieve notably higher accuracy than training on CIFAR10H labels, which we believe is more evidence for our concern.
[Figure]
[Figure]
F DISTANCE GAP UNDER DIFFERENT SUPERVISIONS
Figure 2 provides the distance gap between q and corresponding p ∗ for each training sample in different training stages, under the supervision of one-hot label e y .
From the results in this ﬁgure, we notice that the behavior of hard samples contributes more to the success of ESKD.
Here we further visualize how these gaps changes when the model is trained under different types of supervision, i [Figure] e., ground truth, LS and ESKD.
In the ﬁrst row, which is the result of label smoothing [Figure] we see the (cid:107) p tar − p ∗ (cid:107) 2 (i.e., red dots) has a ‘V’-shape.
The vertex location depends on the smoothing parameter we choose.
However, as discussed previously [Figure] label smoothing has only one parameter to control p tar , which is like using a linear model to ﬁt a high-order function.
So, although a proper smoothing value can bring better supervision than one-hot label, its upper bound might be limited.
Regarding the training dynamics, we can see a similar trend as the results shown in the one-hot case, i.e., all the samples ﬁrst move down and then converge to p tar .
From the middle panel, we might expect label smoothing to outperform the one-hot case, because the scatters are closer to the x-axis, which represents the ground truth p ∗ .
The second and the third rows demonstrate the KD and ESKD case.
Results in the KD case are quite similar to the one-hot case in Figure 2, because the converged p tar is close to e y .
However, we can still expect KD to outperform one-hot training, because the p tar is closer to p ∗ than e y is.
The last panel in this row also demonstrates that (cid:107) q − p ∗ (cid:107) 2 might be smaller than (cid:107) p tar − p ∗ (cid:107) 2 , which can be considered as an explanation for why iterated self-distillation, e.g., Born Again Networks (Furlanello et al., 2018), can improve the performance.
For the ESKD case, we see the overﬁtting almost disappear: the distribution of the blue points do not change too much after the early stopping criterion is satisﬁed.
In the last row, which illustrates the training under the supervision of p ∗ , we ﬁnd all the blue points move toward the x-axis, i.e., their p tar = p ∗ , and ﬁnally converge to it.
There is also no overﬁtting in this case.
From the last two panels, we see the disperse of blue points is signiﬁcantly smaller than all other settings, which means the network’s prediction is quite close to p ∗ .
Hence the performance of this case is the best.
G M ORE A BOUT THE D ECOMPOSITION AND NTK M ODEL
Proof of Proposition 1: to observe how q t ( x o ) changes when the network’s parameters update after observing x u , we can directly write down ∆ t ( x o ) =
q t +1 ( x o ) − q t ( x o ) and simplify it. Recall z ( x ) =
f ( w , x ) is the vector of output logits, and q = Softmax ( z ) is the output probability); the derivation is explained line-by-line afterwards.
[Equation]
[Equation]
For the right-hand term, we used (cid:107)∇ w L (cid:107) 2 ≤ (cid:107)∇ z L (cid:107) 2 · (cid:107)∇ w z (cid:107) 2 ; note that (cid:107)∇ z L (cid:107) =
(cid:107) p tar − q t (cid:107) is bounded because they are probabilities.
The term (cid:107)∇ w z (cid:107) is also bounded if we use a bounded loss, such as gradient-clipped softmax, and will in general stay bounded as long as the optimization doesn’t diverge.
Thus the last term is O ( η 2 ) , which can be ignored when the learning rate η is small.
The ﬁrst line is a Taylor expansion, where ∆ t ( x o ) is a K × 1 vector, ∇ w q t ( x o ) | w t is a K × d matrix, and w is a d × 1 vector ( d is the number of parameters).
The second line follows the chain rule, where ∇ z q t ( x o ) | w t is a K × K matrix and ∇ w z ( x o ) | w t is a K × d matrix.
The third line applies the deﬁnition of SGD, where ∇ w L ( x u ) | w t is a 1 × d vector.
The fourth line also follows the chain rule, where ∇ z L ( x u ) | w t is a 1 × K vector and ∇ w z ( x u ) | w t is a K × d matrix.
The ﬁfth and the sixth lines are rearrangements of different terms, where both A t and K t are K × K matrices.
In this expression,
[Equation]
which is a symmetric positive semi-deﬁnite (PSD) matrix 9 with trace tr ( A t ( x o ))
= 1 − (cid:80)
Ki =
1 q 2 i .
As we have (cid:80) i q i = 1 , it is easy to check the trace of this matrix is larger at the beginning of training (when q tend to be ﬂat) than that at the end of training ( q tend to be peaky), as illustrated by most panels in Figure 19.
Given that the trace of a matrix is the sum of its eigenvalues and A t ( x o ) is PSD, smaller tr ( A
[Equation]
x [Equation]
means this matrix will tend to shrink the vector it multiplies more.
Hence the step size of ∆ t ( x o ) tend to decrease when q t becomes more peaky.
The second term in that expression, K t ( x o , x u ) , is the inner product of gradients at x o and x u .
Intuitively, if their gradients have similar directions, this matrix is large, and vice versa.
This matrix is known as the empirical neural tangent kernel, and it can change through the course of training as the network’s notion of “similarity” evolves.
For appropriately initialized very wide networks trained with very small learning rates, however, K t remains almost constant during the course of training, and is almost independent of the initialization of the network parameters; the kernel it converges to is known as the neural tangent kernel (Jacot et al., 2018; Arora et al., 2019).
Learning Dynamics of q ( x o ) : in proposition 1, we decompose ∆ t ( x o ) into three parts and analyze what the learning path of a training sample might be like in Section 3.3.
Here we will provide more detailed illustration of the two groups of force imposed on q ( x o ) .
Speciﬁcally, ∆ t ( x o ) will be inﬂuenced by two variables, i.e., updating sample x u and time t .
We discuss their effects separately.
For x u , only the last two terms, i.e., K t ( x o , x u ) and ( p tar ( x u ) − q t ( x u )) depends on it.
In Section 3.3, we claim that if x o and x u are similar, the norm of K 0 ( x o , x u ) might be large, and vice versa.
Here we empirically verify this using a toy Gaussian dataset, as illustrated in Figure 19.
The ﬁgure shows how the similarity between x o and x u correlates with tr ( K 0 ( x o , x u )) .
Each panel represents one x o , which is claimed in the title of the subﬁgures.
The x-axis represents the rank of the cosine similarity between observed x o and all N training samples (including itself).
The y-axis is the trace of K 0 ( x o , x u ) .
The color of each scatter point is the true label of x u .
From the ﬁgure, we can observe a clear decreasing trend, which means smaller similarity leads to larger tr ( K 0 ( x o , x u )) .
Additionally, the term ( p tar ( x u ) − q t ( x u )) provides a direction that q t ( x o ) should move towards.
We claim in Section 3.3 that at any point in the input space, the labels of these input samples might follow the ground truth distribution, i.e., p ∗ ( y | x ) .
Hence most of the neighbouring x might pull q ( x o ) towards its ground truth p ∗ ( y | x o ) .
We will discuss the norm of this term when discussing the inﬂuence of t .
In short, at any time, the neighboring x u will impose stronger force on x o and the direction of the force roughly points to the ground truth p ∗ ( y | x o ) .
Another factor to consider is t , which inﬂuences ∆ t ( x o ) through the term A t ( x o ) and term ( p tar ( x u ) − q t ( x u )) .
However, these two terms do not have an analytical expression w.r.t t , as q t is quite complicated.
So we can only provide some intuitions through the behavior of q t and then experimentally verify them.
In Figure 19, we show how tr ( A t ( x o )) depends on q t : ﬂat q t leads to larger trace.
A similar trend also exists in the norm of ( p tar ( x u ) − q t ( x u )) .
As the initialized q tend to be ﬂat, updates of any samples will inﬂuence network’s parameters a lot.
When the training progress, those easy samples converge fast, so their (cid:107) p tar ( x u ) − q t ( x u ) (cid:107) 2 and tr ( A t ) become small.
However, as the q t for the hard sample is still far away from its e y , the large (cid:107) p tar ( x u ) − q t ( x u ) (cid:107)and tr ( A t ) will ﬁnally drag q t back towards the one-hot distribution, as illustrated in Figure 3.
H LOW PASS FILTER ON NETWORK PARAMETERS
In Section 4, we point out the high variance issue of the traditional KD methods after observing the learning path of those hard samples.
We then propose a Filter-KD algorithm to smooth the output of each training sample during training.
Such a low pass ﬁltering method is quite similar to the momentum mechanism used in self-supervised learning, e.g., from the classic method of
Ruppert (1988) to MOCO (He et al., 2020), which conduct low pass ﬁltering on each parameter of the network.
As mentioned before, the proposed Filter-KD algorithm might require more memory when the dataset becomes larger, because q smooth would record every training sample’s prediction.
Conducting low pass ﬁltering on network parameters might be a good way to solve this.
To verify whether this method works, we train a ResNet18 on CIFAR100, using one-hot supervision.
At the beginning of training, we copy the training model and call it tracking model.
At the end of each update (i.e., each batch), the parameters of training model is updated as usual while the tracking model’s parameters are updated with momentum.
Speciﬁcally, for the training model, w t +1 train =
w t train + η ∇ L , for the tracking model, w t +1 track =
(1 − α ) w t track + α w t +1 train .
We train the model for 150 epochs, and show the learning curve of test accuracy in the left panel of Figure 20.
We see the performance of the tracking model converges faster than the training model, which is reasonable because ﬁltering parameters is regularizing the training process.
However, the converging accuracy of this two models are the same.
At the same time, we ﬁnd this tracking model is not as good as q smooth when teaching a new model, as illustrated in the right panel of Figure 20.
As this paper mainly focuses on the behavior of the model’s prediction, the discussion and experiments of ﬁltering in parameter space is limited.
However, as many papers demonstrates the effectiveness of this momentum mechanism, we think it is important to explore the relationship further.
Test ACC
Test ECE
Run
[Table]
OHT 95.
KD 95.
ESKD 95.
FilterKD 95.
OHT 78.
KD 78.
ESKD 78.
FilterKD 79.