O N STUDENT - TEACHER DEVIATIONS IN DISTILLATION : DOES IT PAY TO DISOBEY ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowledge distillation has been widely-used to improve the performance of a “student” network by hoping to mimic soft probabilities of a “teacher” network.
Yet, for self-distillation to work, the student must deviate from the teacher in some manner (Stanton et al., 2021).
What is the nature of these deviations, and how do they relate to the generalization gains of distillation?
To investigate these questions, we ﬁrst conduct a variety of experiments across image and language classiﬁcation datasets.
One of our key observations is that in a majority of our settings, the student underﬁts points that the teacher ﬁnds hard.
We also ﬁnd that studentteacher deviations during the initial phase of training are not crucial to see the beneﬁts of distillation — simply switching to distillation in the middle of training can recover much of its gains.
We then provide two parallel theoretical perspectives of these deviations, one casting distillation as a regularizer in eigenspace, and another as a denoiser of gradients.
In both views, we argue how student-teacher deviations emerge, and how they relate to generalization in the context of our experiments.
Our analysis also bridges fundamental gaps between existing theory and practice by focusing on gradient descent and avoiding label noise assumptions.
I NTRODUCTION
Distillation (Bucil ˇ a et al., 2006; Hinton et al., 2015) has emerged as a highly effective model compression technique, wherein one trains a small “student” model to match the predicted soft label distribution of a large “teacher” model, rather than match one-hot labels.
An actively developing literature has sought to explore applications of this technique to various settings (Radosavovic et al., 2018; Furlanello et al., 2018; Xie et al., 2019), design more effective variants of the above recipe (Romero et al., 2015; Anil et al., 2018;
Park et al., 2019; Beyer et al., 2022), and better understand theoretically when and why distillation is effective (Lopez-Paz et al., 2016; Phuong &
Lampert, 2019; Mobahi et al., 2020; Allen-Zhu & Li, 2020; Menon et al., 2021; Dao et al., 2021).
On paper, distillation intends to transfer the teacher’s soft probabilities over to the student.
However, Stanton et al.
(2021) challenge this premise: they show there is often a mismatch of student and teacher probabilities, and in fact, that a greater mismatch is correlated with better student performance.
Indeed, in the self-distillation setting (Furlanello et al., 2018; Zhang et al., 2019) — where the student and teacher architectures are identical — some form of deviation (in the representation, if not in the probabilities) is necessary for the student’s generalization to supercede the teacher.
In this work, we are interested in better characterizing these deviations in probabilities, and in understanding how they play a role in the student outperforming the teacher.
In the ﬁrst half of the paper, we conduct experiments characterizing what kind of deviations exist between the teacher and the student, and which deviations are relevant for better generalization.
In the second half, we provide two complementary theoretical perspectives on how distillation can induce such deviations, and why that can subsequently aid generalization.
More concretely, our key contributions are as follows:
(i) What deviations exist? Across various architectures (ResNet56, ResNet20, MobileNet, and RoBERTa), and image/language classiﬁcation data (CIFAR100, TinyImageNet, CIFAR10, GLUE).
we empirically demonstrate (§3.1) that the the student tends to underﬁt on “hard” points for the teacher (Fig 1a) in terms of the ﬁnal probabilities learned by both models. (ii) Which deviations matter? We ﬁnd (§3.2) that it is possible to switch from one-hot loss in the middle of training to distillation loss and (a) still recover a considerable fraction of distillation’s
[Figure]
gains (Fig 1b), ( b) and also recover the ﬁnal-epoch underﬁtting behavior on TinyImageNet and CIFAR100.
Thus, we conclude that any student-teacher deviations unique to the early phase of training – such as those proposed in Allen-Zhu & Li ; Jha
et al are not by themselves adequate to explaining the success of distillation, but the underﬁtting may be. Next, we ask how deviations arise and why they help
(iii) Eigenspace view: We provide a counterpart of the seminal result of Mobahi et al.
(2020) — which demonstrates distillation as a regularizer in a non-gradient-descent setting — for the gradient descent setting for linear regression (Theorem 4.1).
We propose this view as a way to understand the empirically observed underﬁtting in distillation.
Besides providing a much simpler proof and a more practically relevant version of Mobahi et al.
(2020), our result also formalizes existing empirical intuition about the importance of early-stopping in distillation (Dong et al., 2019; Cho & Hariharan, 2019; Ji & Zhu, 2020; Wang et al., 2022).
(iv) Gradient space view. As a complementary viewpoint, we formalize distillation as a gradient denoiser in the presence of class similarities (Theorem 4.2).
We propose this view as a way to understand our empirical observations on loss-switching.
Importantly, unlike prior work (Menon et al. (2021)), we show how denoising can occur even when the data is perfectly classiﬁable and has no inherent label noise.
(v) A uniﬁed view. We informally unify these two views, thus painting a more coherent picture of two disjoint lines of existing theories (Mobahi et al.
vs. Menon et al.
Overall, we hope that our discussion helps bridge the gap between existing theoretical understanding and empirics in distillation by (a) making more practical assumptions than existing theories, and (b) making connections to various empirical observations.
Our ﬁndings also suggest that not matching the teacher probabilities exactly can be a good thing, which future empirical work on distillation may want to be mindful of.
B ACKGROUND AND N OTATION
Our interest in this paper is multiclass classiﬁcation problems.
This involves learning a classiﬁer h : X → Y which, for input x ∈ X , predicts the most likely label h ( x ) ∈ Y =
[ K ] .
= { 1 , 2 , . .
. , K } .
Such a classiﬁer is typically implemented by computing logits f : X → R K that score the plausibility of each label, and then computing h ( x ) =
argmax
y ∈ Y f y ( x ) .
In neural models, these logits are parameterised as f ( x ) =
W (cid:62) Z
( x ) for learned weights W ∈ R D × K and embeddings Z ( x ) ∈ R D .
One may learn such logits by minimising the empirical loss on a training sample S .
= { ( x n , y n ) } Nn =1 :
[Equation] [Equation]
where e ( y ) ∈ { 0 , 1 } K denotes the one-hot encoding of y , (cid:96) ( · ) .
=
[ (cid:96) (1 , · ) , . . .
, (cid:96) ( K, · )]
∈ R K denotes the loss vector of the predicted logits, and each (cid:96) ( y, f ( x )) is the loss of predicting logits f ( x ) ∈ R K when the true label is y ∈
[ K ] .
Typically, we set (cid:96) to be the softmax cross-entropy (cid:96) ( y, f ( x ))
= − log p y ( x ) , where p ( x ) ∝ exp( f ( x )) is the softmax transformation of the logits.
Equation 1 guides the learner via one-hot targets e ( y n ) for each input.
Distillation (Bucilˇa et al., 2006; Hinton et al., 2015) instead guides the learner via a target label distribution p te ( x n ) provided by a teacher , which are the softmax probabilities from a distinct model trained on the same dataset.
In this context, the learned model is referred to as a student , and the training objective is
[Equation]
One may also consider a weighted combination of R emp and R dist for algorithmic reasons, but we focus on the above objective in this paper since we are interested in understanding each objective individually.
Compared to training on the one-hot labels, distillation often results in improved performance for the student (Hinton et al., 2015).
Typically, the teacher model is of higher capacity than the student model; the performance gains of the student may thus informally be attributed to the teacher transferring rich information about the problem to the student.
In such settings, distillation may be seen as a form of model compression.
Intriguingly, however, even when the teacher and student are of the same capacity (a setting known as self-distillation ), one may see gains from distillation (Furlanello et al., 2018; Zhang et al., 2019).
The questions we explore in this paper are motivated by the self-distillation setting; however, for a well-rounded analysis, we will empirically study both the self- and cross-architecture-distillation settings.
A FINE - GRAINED LOOK AT TEACHER - STUDENT DEVIATIONS
Stanton et al.
(2021) found that, contrary to the premise of distillation, more accurate students are poorer at matching the teacher probabilities.
They quantiﬁed the student-teacher deviation by measuring the disagreement or the KL divergence between the student and teacher probabilities, in expectation over all points.
Could the average deviation in probabilities be attributed merely to an arbitrary lack of precision in matching the probabilities during training?
In the following section, through a closer study of per-sample relationship between the teacher and student predictions, we do not ﬁnd this to be the case; rather, there are discernible patterns in student-teacher deviations.
3. W HAT DEVIATIONS EXIST : A PER - SAMPLE VIEW
Suppose we have teacher and distilled student models f te , f st : X → R K respectively.
We seek to visualize the deviations in the corresponding predicted probability vectors p te ( x ) and p st ( x ) for each ( x, y ) in the train and test set, rather than in the aggregated sense as in Stanton et al.
To visualize the deviations, we need a scalar summary of these vectors.
An obvious candidate is the probabilities ( p te y (cid:63) ( x ) , p st y (cid:63) ( x )) assigned to the ground truth label y (cid:63) .
However, since the student does not have access to the ground truth label, and is only trying to mimic the teacher, we examine deviations of probabilities of the teacher’s predicted label , i.e., ( p te y te ( x ) , p st y te ( x )) where y te .
= argmax y (cid:48) ∈
[ K ] p te y (cid:48)
( x ) .
To make patterns easier to detect, we further perform a monotonic logit transformation φ ( u ) =
log [ u/ (1 − u )] that produces real-values in ( −∞ , + ∞ ) .
Thus, we compare φ ( p te y te ( x )) and φ ( p st y te ( x )) for each train and test sample ( x, y ) .
We report a scatter plot of
( p te te ( vs p y te x )
) on the training set for both various self-distillation settings in Figures 1a and cross-architecture distillation settings in Fig 2.
In all plots, the indicates the line.
All values are computed
The tasks considered include image classiﬁcation benchmarks, namely CIFAR10, CIFAR-100, TinyImageNet, and text classiﬁcation tasks from the GLUE benchmark (e.g., MNLI (Williams et al., 2018),
AGNews (Zhang et al., 2015).
See Appendix C.1 for details on the experimental hyperparameters.
Many additional plots are presented in §C.2.1.
Distilled students tend to underﬁt hard samples .
Our main ﬁnding is that in most of our cases, the student underﬁts a subset of points (i.e., Y ≤ X in the scatter plot) that are typically hard for the teacher (i.e., for small values of X ).
We note that this pattern is particularly strong on test data, and appears in the test data even in certain exceptions where it does not appear in the training data (see §C.2.1, e.g., CIFAR-10 plots Fig 7).
We also note that in the language datasets, typically there is high deviation for harder points, involving both underﬁtting and overﬁtting, along with low deviation for easier points (i.e., large X ), typically involving only overﬁtting (as can be seen in AGNews Figand in §C.2.1 Fig 8, Fig 9).
These patterns however break down for cross-architecture distillation in language datasets.
See §C.2.1 for a discussion of the exceptions.
Why is underﬁtting good?
To better understand this, consider the setting where we perform distillation when a portion of CIFAR100 one-hot labels are mislabeled. In Fig 1a and Fig 2, we observe that the mislabeled points the student underﬁts the teacher’s probabilities on all mislabeled points, implying that the student somehow denoises the data.
This forms the core of our intuition for why underﬁtting can be helpful . As an aside, we note that even though the student denoises a portion of the data, this may not be reﬂected in the overall accuracy numbers when the student is smaller (Table 2). This brings out the subtlety that smaller student models can go beyond a larger teacher in ways that are not apparent through mere accuracy on the whole dataset .
We provide a more formal view of the underﬁtting in §4.1 where we present distillation as a form of regularization.
Having said that, we must note that underﬁtting may not always correspond to improved generalization (cf.
§C.2.1, Table 2), just like regularization of any form may not always be helpful.
Evolution of teacher-student deviations.
While we have looked at deviations between the ﬁnal probabilities learned by the models, it is also instructive to look at how these deviations evolve as student training progresses.
Concretely, in Fig 3a (and §C.2.3 Fig 13) we present a series of scatter plots for CIFAR100 (and Tiny-Imagenet) settings where the X axis is always ﬁxed to be the ﬁnal (logit-transformed) teacher probabilities while the Y axis corresponds to snapshots of the distillation-trained and the one-hot trained model at various instances.
Here, we ﬁnd signiﬁcant differences between the scatter plots early on during training.
In particular, the distilled model has converged much faster to its ﬁnal values than the one-hot model.
We also see from the left-most snapshot that the student has prioritized its easier points (i.e., points with large X are close to the
X = Y line), while for the one-hot model, the easiest points are not as prioritized (i.e., points with large X are farther from X
=
Y compared to other points).
3. W HICH DEVIATIONS MATTER : T HE EFFECT OF LOSS SWITCHING
[Figure]
Motivation. Prior works suggest that student-teacher deviations in the very initial part of training are critical for why distillation helps: it conditions the network into a more favorable representation Allen-Zhu & Li (2020) or basin early on (Jha et al., 2020), or induces faster convergence (Phuong & Lampert, 2019) or perhaps, induces a favorable learning order (as is suggested by our own experiment in Fig 3a).
However, in this section, we question whether initial phase deviations are critical at all.
Experiment design.
To test the relevance of early-phase deviations, we conduct late distillation experiments where we train a network with one-hot loss for a fair part (say, half) of training, and then switch to distillation loss (gradually).
If distillation relied crucially on early phase deviations, late distillation on a one-hot-pretrained network should not result in substantial improvements. Observations.
We make three key observations here. First, from Figures 4 (and Sec C.2.3 Fig 14), we observe that replacing the ﬁrst 1 / 4 th of training with one-hot is able to recover all of distillation’s gains in our CIFAR100 and TinyImagenet settings; replacing the ﬁrst 1 / 2 of training, recovers a signifcant fraction of the gains. This happens despite the fact that when we switch to distillation, we have a much smaller learning rate (due to our chosen schedule, see Sec C.1).
This implies that early phase deviations are not a necessary requirement for distillation to help .
Conversely, we also switch from distillation to one-hot in Fig 1b and Fig 16.
Here we ﬁnd that switching to one-hot undoes the gains that distillation has achieved thus far, given a sufﬁciently long one-hot training. This suggests an inherent destructive effect in one-hot training that we formalize in
Sec 4.2 .
We provide a more nuanced discussion of these experiments in Sec C.2.3 (and Fig 15).
Our ﬁnal observation concerns the underﬁtting behavior in Section 3.1.
In particular, even though early-phase one-hot training has already ﬁt all examples, we ﬁnd that subsequent distillation “unlearns” the harder examples to force the typical underﬁtting behavior (See Fig 3b, top).
Conversely, switching to one-hot ﬁxes the underﬁtting already induced by an initial distillation phase.
We argue that this reinforces the relevance of the underﬁtting of the ﬁnal probabilities to the working of distillation .
F ORMALIZING STUDENT - TEACHER DEVIATIONS : T WO PERSPECTIVES
To understand the above empirical observations regarding underﬁtting and loss-switching, and their connections to the beneﬁts of distillation, we provide two complementary perspectives of distillation: an eigenspace perspective and a gradient-space view.
These perspectives also aim to provide key clariﬁcations about existing intuition in literature, and paint a more coherent picture of disjoint threads of theoretical research.
4. T HE EIGENSPACE VIEW : DISTILLATION AS A REGULARIZER
The underﬁtting of hard points possibly suggests that distillation induces a form of regularization that deprioritizes certain complex features.
Indeed, Mobahi et al.
(2020) have proven that self-distillation acts as a form of sparsiﬁer that focuses on simpler basis functions while ignoring more complex ones.
However, their result is agnostic to the biases of gradient descent.
We bridge the gap between this intuition and practice, by showing that a similar sparsiﬁcation effect emerges under gradient descent training of linear models, thanks to the inherent bias of gradient descent in converging at varied rates along various eigendirections.
Concretely, we analyze a continuous-ﬂow gradient descent model on a linear regression setting with early-stopping, a typical design choice in distillation practice (Dong et al., 2019; Cho & Hariharan, 2019; Ji & Zhu, 2020).
Consider an n × p dataset X (where n is the number of samples, p the number of parameters) with target labels y .
Assume that the Gram matrix XX (cid:62) is invertible.
Note that this setting includes overparameterized scenarios ( p > n ) such as when X corresponds to the linearized (NTK) features of neural networks (Jacot et al., 2018; Lee et al., 2019).
Then, a standard calculation reveals that the weights learned at time t under gradient descent on the loss (1 / 2) · (cid:107) X − y (cid:107) 2 can be written as:
[Equation]
where A ( t ) := [Equation] Intuitively, A is a “sparsifying” matrix that skews down the weight assigned to an eigendirection of eigenvalue λ by the value 1 − e − λt .
As t → ∞ this factor goes to 1 for all directions, thus becoming irrelevant; but for any ﬁnite t > 0 , the topmost direction would have a larger factor than the rest, thus producing some sort of sparsiﬁcation.
Our argument is that, while standard gradient-descent already has an implicit sparsiﬁcation effect, distillation further ampliﬁes the sparsiﬁcation effect .
Consider a setting where the teacher is trained to time T te .
Through a simple calculation, the weights of the student model can be written as:
[Equation]
Akin to Mobahi et al.
(2020), one can argue that the sparsiﬁer ˜
A corresponding to the student is more skewed towards the top eigenvectors than the teacher.
More formally:
Theorem 4.1.
Let α k ( t ) , ˜ α k ( t ) be the eigenvalues of the k ’th top eigendirection in A ( t ) and ˜ A ( t ) respectively.
Let λ k be the corresponding eigenvalue of the Gram matrix XX (cid:62) .
For any two indices k 1 < k 2 such that λ k 1 > λ k 2 (assuming such a pair exists), and for any α (cid:63)k 1 ∈ (0 , 1) (a required “convergence lower bound” on the higher eigenvector), there exists a choice of distillation hyperparameters T te , ˜ t such that (a) ˜ α k 1 (˜ t ) ≥ α (cid:63)k 1 , and (b) for any choice of time t for the standalone early-stopped model satisfying α k 1 ( t ) ≥ α (cid:63)k 1 , the components along the lower eigenvectors satisfy: [Equation] [Table] [Equation]
[Table]
[Equation]
The result says that the student relies less on the bottom eigendirections than the teacher if we compare them at any instant when they have both converged equally well along the top eigendirections (i.e., they have sufﬁciently ﬁt the “nice” directions). Crucially, this implies that the student traverses a different part of the parameter space, thus capturing a different bias than the teacher. In Sec C.4, we empirically verify that the student indeed ampliﬁes the sparsiﬁcation bias of the teacher, in a more general non-linear setting involving cross-entropy loss and a neural network.
Connection to underﬁtting.
The connection between how distillation ampliﬁes regularization and the underﬁtting in §3.1 is best veriﬁed through our experiments on noisy CIFAR100 experiments from Fig 1a and Fig 2.
First, it is well-known (Li et al., 2020; Dong et al., 2019; Arpit et al., 2017; Kalimeris et al., 2019) that the inherent bias of early-stopped gradient-descent ﬁts noisy subset of the data more slowly than the clean data because noisy labels correspond to bottom eigendirections.
This is indeed the case since mislabeled points come lowest in terms of teacher probabilities (small
X values). But our insights is that the student exaggerates this effect by assigning further lower probabilities to the mislabeled data ( Y < X ).
This exaggeration must be the manifestation of how distillation ampliﬁes the sparsiﬁcation in the eigenspace.
4. T HE GRADIENT - SPACE VIEW : DISTILLATION AS A GRADIENT DENOISER
While the previous perspective gives us an abstract theory of how distillation can induce regularization, it may not be obvious when this regularization can be helpful besides in settings with label noise.
In fact, if ignoring the bottom eigenvectors led to ignoring non-noisy “tail” datapoints, this can even be hurtful (Feldman, 2019).
Below, we formalize how distillation can help by extending a line of work analyzing distillation on noisy datasets with class similarities (Menon et al., 2021; Dao et al., 2021;
Ren et al., 2022).
But crucially, we show how distillation can help by denoising even in a perfectly classiﬁable dataset with class similarities, and no explicit label noise .
A concrete example.
To make our discussion easier, consider a concrete K -class classiﬁcation dataset, where the i th datapoint’s features can be written as a K -“channel” input x i
=
( x ( i ) 1 , . . .
, x ( i ) K ) where each channel x ( i ) k
∈
R
D is D -dimensional.
Assuming a uniform distribution over K classes, given a label y , we generate x y from a Gaussian N (0 , I /D ) truncated to the support x y · µ (cid:63)y i = τ for a pre-deﬁned τ > 0 and a “ground truth” class vector µ (cid:63)k .
We also assume that for point x i , we pick a non-target “similar” class
z i for which we pick x z i from a Gaussian N (0 , I /D ) truncated to the support x
z i · µ (cid:63)z i = τ/ 2 .
All other co-ordinates are zero for x i .
We consider a simpliﬁed linear architecture whose K -dimensional output is of the form f ( x ) =
( w 1 · x 1 , . . .
, w K · x K ) where the k ’th node acts only on the k ’th channel.
Observe that this is a perfectly classiﬁable dataset with deterministic labels since if we set the model weights to be w k = µ (cid:63)k , we obtain that for the corresponding logits f (cid:63) , arg max k f (cid:63)k ( x ) =
y (cid:63) .
In other words, the ground truth probabilities are one-hot, which can be recovered by computing softmax ( { α · f k ( x ) } Kk =1 ) as α → ∞ .
Thus, it may seem that training on the one-hot loss is the wisest choice in this setting.
However, we show that distillation’s gradients (a) are more optimal than one-hot gradients and also (b) more optimal than the teacher’s weights themselves.
To formalize this, let cos-sim ( · , · ) denote the cosine-similarity between two vectors.
Then:
Theorem 4.2.
(informal; see Thm B.4) Consider a student whose weights satisfy w k = α µ (cid:63)k .
Consider an imperfect teacher with weights such that (cid:107) w te k (cid:107) =
α te k and cos-sim ( w te k , µ (cid:63)k ) · ≥ 1 − (cid:15) .
Let v k be the one-hot update on node k of the student, and ˜ v k the distillation update, both under cross-entropy loss.
Then, for α (cid:28) α te and α te ≤ O ( 1 τ log K ) , and sufﬁciently large τ, D :
[Equation]
[Figure]
[Table]
[Equation]
There are three salient aspects to this result.
First, this demonstrates that one-hot gradients can be inherently sub-optimal at any timestep in training even if the model is initialized at an optimal solution .
Conversely, distillation can help improve these gradients even if the teacher is imperfect .
Finally, distillation gradients can themselves be “more perfect” than the imperfect teacher, thus formalizing how the student can go beyond the teacher.
To understand the intuition behind this result, imagine that at some point of training, we have recovered a good f such that f ≈ α · f (cid:63) (for some ﬁnite scaling factor α ).
Now as we train longer, we ideally hope that (a) α continues increasing towards ∞ to recover the one-hot ground truth probabilities, and (b) the approximation of the ground truth logits continues to hold throughout.
This is however not the case with the one-hot updates.
Consider any point ( x , y ) where f (cid:63)k ( x ) is high for some k (cid:54) =
y i.e., the point has similarities to a non-target class.
On such a point, the ideal dynamics of increasing α requires that we increase the logit on the non-target node f k ( x ) .
But the one-hot update would attempt to suppress this logit, since the target probability for this node is 0 .
We argue that this is a destructive gradient since it directs the model in the opposite direction of the ideal dynamics.
Notably, this manifests despite the lack of inherent label noise in the training set.
Distillation on the other hand, can ﬂip the sign of these gradients (assuming the teacher is earlystopped), thus denoising the gradients — even if there is no explicit noise in the data.
Notably, this sign-ﬂipping provides a “fresh example” for the non-target node k , which is a privilege unavailable to the teacher.
This is crucial to prove that the student is more denoised than the the teacher.
Connection to loss-switching.
The gradient-space view gives us a way to make sense of the ﬁndings in §3.2.
In particular, this tells us why each gradient in itself may be constructive (in the case of distillation) or destructive (in the case of one-hot).
Thus, switching to distillation (or one-hot) even in the middle of training can still be helpful (or hurtful), given sufﬁcient time after the switch.
U NIFYING THE TWO PERSPECTIVES
We now discuss how various themes in existing distillation theory can be understood within both of the above perspectives.
Through this, we hope to piece together a more coherent and uniﬁed picture of the two lines of work we build upon: Mobahi et al.
(2020) and Menon et al.
Dark knowledge and denoising.
Distillation is said to beneﬁt by implicitly introducing “dark knowledge” in the form of new information absent in the observed labels of the original datasets (Hinton et al., 2015).
In both views, we can view dark knowledge as a form of recovering denoised features:
(i)
In the gradient-space view, the dark knowledge comes from the non-target logits, which allows us to denoise the gradients during distillation.
(ii)
In the eigenspace view, we posit that the new information corresponds to knowledge of the top eigenvector(s) of the data.
Crucially, these eigenvectors are independent of labels and thereby independent of any label noise; hence these eigenvectors correspond to “denoised” features in the data, which distillation more readily relies on.
On the gap between early-stopping and distillation.
In both our views, we see a subtle distinction between early-stopping and distillation:
(i)
In the eigenspace perspective, although both induce a similar sparsiﬁcation effect, distillation ampliﬁes this effect.
(ii)
In the gradient-space view, early-stopping would reduce the overall magnitude of destructive gradients accumulated by the teacher.
But distillation ampliﬁes this effect further by altogether ﬂipping the sign of these gradients.
Lack of ﬁdelity in probabilities.
We can understand the underﬁtting phenomenon from both views:
(i) Recall that in the eigenspace view, the student’s underﬁtting of teacher’s probabilities can be seen as a consequence of ignoring the lower eigenvectors.
(ii)
In the gradient-space view, the harder points of the teacher would have high similarity to other classes.
Subsequently, under the one-hot loss, these points would experience heavy destructive gradients on the non-target logits.
Since distillation would no longer suppress the non-target probabilities, those probabilities would increase, while the target probability would be underﬁt.
R ELATION TO E XISTING W ORK
Distillation as a probability matching process.
Distillation has been touted to be a process that beneﬁts from matching the teacher’s probabilities (Hinton et al., 2015).
Indeed, many distillation algorithms have been designed in a way to more aggressively match the student and teacher functions (Czarnecki et al., 2017; Beyer et al., 2022).
Theoretical analyses too rely on explaining the beneﬁts of distillation based on a student that obediently matches the teacher’s probabilities (Menon et al., 2021).
However, as Stanton et al.
(2021) demonstrate, an exact matching of probabilities can be anti-correlated with performance improvements.
Our work builds on this nuance as to why we may desire that the student deviate from the teacher.
Understanding distillation via class similarities.
A long-standing intuitive explanation for why distillation helps is that the teacher’s probabilities contain “dark knowledge” about class similarities (Hinton et al., 2015; Müller et al., 2019), which is the idea we formalize in §4.2.
Several works (Menon et al., 2021; Dao et al., 2021;
Ren et al., 2022; Zhou et al., 2021) have formalized this by assuming that the ground truth class memberships are inherently soft, and so the observed one-hot labels are “noisy”.
They argue that distillation reduces the student model’s variance (in an abstract non-gradient-descent setting), assuming the student exactly matches the teacher.
We demonstrate a novel form of denoising even in the absence of noisy labels (via a completely different technical argument). We also show how the student can outperform the teacher under gradient descent.
Alternative theories of distillation.
Some works (Furlanello et al., 2018; Yuan et al., 2020; Tang et al., 2020) have argued that the “class similarity” hypothesis cannot be the sole explanation, because distillation can help even if the student is only taught information about the target probabilities (e.g., by smoothing out all non-target probabilities).
This has resulted in various alternative hypothesis such as faster convergence (Phuong & Lampert, 2019; Rahbar et al., 2020; Ji & Zhu, 2020) (which we question in §3.2), feature learning (Allen-Zhu & Li, 2020) (which would not apply in our linear settings), and regularization, either in the sense of Mobahi et al.
(2020) or in the sense of instance-speciﬁc label smoothing (Zhang & Sabuncu, 2020; Yuan et al., 2020; Tang et al., 2020).
Our eigenspace perspective in §4.1 falls under this umbrella.
Finally, we also refer the reader to Lopez-Paz et al.
; Kaplun et al.
(2022) who theoretically study distillation in orthogonal settings.
Early-stopping and knowledge distillation.
Early-stopping has received much attention in the context of distillation (Liu et al., 2020;
Ren et al., 2022; Dong et al., 2019; Cho & Hariharan, 2019;
Ji & Zhu, 2020).
We closely build on Dong et al.
(2019), who argue how early-stopping a gradientdescent-trained teacher can automatically denoise the labels due to regularization in the eigenspace.
However, neither Dong et al.
(2019) nor any of the other works provide a formal argument for why distillation can outperform the teacher.
Note that Mobahi et al.
(2020) claim early-stopping has a densiﬁcation rather than a sparsiﬁcation effect; but this holds only in their non-gradient-descent setting where the function is picked from a Hilbert space with (cid:96) 2 regularization.
Empirical studies of distillation.
Our study crucially builds on observations from (Stanton et al., 2021; Lukasik et al., 2021) demonstrating student-teacher deviations in an aggregated sense than a sample-wise sense sense.
Other studies (Abnar et al., 2020; Ojha et al., 2022) investigate in what ways the student is similar to the teacher, beyond in terms of class probabilities e.g., out-of-distribution behavior, calibration, and so on.
Deng & Zhang (2021) show how a smaller student can outperform the teacher if it was allowed better match the teacher on more data, which is orthogonal to our setting.
C ONCLUSION
We empirically ﬁnd that a distilled student tends to underﬁt points that the teacher ﬁnds hard.
We also ﬁnd that student-teacher deviations that occur in the initial phase of training are not critical for the success of distillation.
We then formalized how deviations arise and can help the student.
Speciﬁcally, in the context of gradient descent, distillation can act as a regularizer in eigenspace and can denoise gradients even in the absence of explicit label noise.
Overall, our work provides a concrete argument for why certain deviations between the student and teacher are good, contrary to conventional understanding of distillation. We also bridge key gaps between theory and practice, and unify disjoint lines of work following Mobahi et al.
(2020) and Menon et al.
There are several directions for future work.
One question is whether we can study more practical distillation setups, such as in semi-supervised setups (Cotter et al., 2021).
It would also be insightful to extend our eigenspace view in to multi-layered models where the eigenspace regularization effect may “compound” across layers.
Another interesting direction is whether we can build on the gradient space view to design teacher-free training techniques that suppress “destructive” non-target gradients in one-hot training.
Arguably, the most important implication is on current attempts to bridge the gap between student and teacher performance.
Perhaps, these attempts would beneﬁt from encouraging some carefully-designed gaps in the teacher and student probabilities.
A L IMITATIONS
We highlight a few key limitations to our results that may be relevant for future work to look at: 1.
While we show underﬁtting across many datasets, we do not establish when this form of underﬁtting can help in situations besides label noise.
Indeed, we ﬁnd that, in our language datasets, underﬁtting can occur even in the absence of any improvement in generalization.
Future work may explore more sophisticated measures of student-teacher deviation that correlates with the gains of distillation.
We highlight further caveats of our empirical results in Sec C.2.1.
Both our formalizations are based on linear models.
On the one hand, this implies that our results are general and fundamental in that they are not unique to neural network training.
On the other, this means our result may not capture other interesting effects in the context of distillation (such as that of feature-learing as in Allen-Zhu & Li (2020)).
Indeed, we hypothesize that the eigenspace regularization effect may compound across multiple layers of a neural network.
This is an open direction for future work.
While our gradient-space view suggests why switching to one-hot can hurt, it does not explain why switching to one-hot for a very short amount of time increases the performance — even beyond distillation — in the case of ResNet50 self-distillation (cf.
Sec C.2.
Fig 16 and Fig 15).
Analyzing this nuanced effect of distillation-to-one-hot switching is an interesting question for future work.
B E IGENSPACE VIEW
Below, we provide the proof for the sparsity levels of an early-stopped teacher and early-stopped student.
Proof.
(of Theorem 4.1)
For the early-stopped model, we need 1 − e − λ k 1 t ≥ α
(cid:63)k 1 .
For any value of t that satisﬁes this, we can lower bound α k 2 ( t ) as:
[Equation]
Now, recall that we need to show that there exists a set of distillation hyperparameters ˜ t, T te that satisfy our required bounds.
We will show that such a set of hyperparameters exist under the simpliﬁcation
that T te =
˜
t .
Thus, for the student model, we need (1 − e − λ k 1 ˜ t )(1 − e − λ k 1 t ) ≥ α (cid:63)k 1 .
To achieve the largest skew, we should choose the smallest ˜ t , which is achieved when (1 − e − λ k 1 ˜ t )
= α (cid:63)k 1 .
Plugging this into the expression for ˜ α k 2 (˜ t ) , we get:
[Equation]
For simplicity, let us deﬁne the fraction κ :=
λ k 2 λ k 1 .
Putting the above two sets of equation together, we have:
[Equation]
The last step follows from the power means inequality: we have that since f < 1 ,
[Equation]
is at most
[Equation]
B. G RADIENT - SPACE VIEW
In this section
[Equation]
we prove Theorem 4.2 where we analyzed the quality of one-hot and distillation gradients in a setting with class similarity and zero label noise.
We ﬁrst recall the setting below.
Setting.
We consider a K -class classiﬁcation dataset, where the i th datapoint’s features can be written as a K -“channel” input x i =
( x ( i ) 1 , . . .
, x ( i ) K ) where each channel x ( i ) k
∈
R
D is D -dimensional.
Assuming a uniform distribution over K classes, given a label y , we generate x y from a Gaussian
N (0 , I /D ) truncated to the support x y · µ (cid:63)y i = τ for τ > 0 and for a pre-deﬁned class vector µ (cid:63)k .
We also assume that for point x i , we pick a non-target “similar” class
z i for which we pick x z i from a Gaussian N (0 , I /D ) truncated to the support x
z i · µ (cid:63)z i = τ/ 2 .
All other co-ordinates are set to zero.
We consider a simpliﬁed linear architecture whose K -dimensional output is of the form f ( x ) =
( w 1 · x 1 , . . .
, w K · x K ) where the k ’th node acts only on the k ’th channel.
Notations : Throughout the rest of the discussion, we ﬁx some node k , and deﬁne S + = { ( x , y ) ∈
S | y
= k } and S − = { ( x , y ) ∈ S | y (cid:54)
=
k, z = k } .
Intuitively, S + is the subset of samples that provide “constructive” gradients for node k under the one-hot loss, while S − is the subset that provides “destructive” gradients under the one-hot loss (i.e., gradients in the opposite direction of optimality).
With an abuse of notation, we will write x ∈ S to denote an x such that ( x , y ) ∈ S .
We will use p te to denote a perfect teacher’s probabilities, and q te to denote an imperfect teacher’s probabilities.
For any channel k , and for any vector w ∈ R D of weights/gradients corresponding to that channel, we will use w (cid:107) to denote the projection w (cid:107) =
( w · µ (cid:63)k ) µ k , and w ⊥ to denote the orthogonal vector, w (cid:107) =
w − ( w · µ (cid:63)k ) µ k . Also recall that α te denotes the “scale” of the teacher’s logits — think of this as a proxy for how long the teacher has been run to minimize one-hot loss.
Similarly, α denotes the scale of the student model’s logits.
We now state some simplifying assumptions.
Assumption B.1.
We make the following simplifying assumptions:
We assume that we have a training dataset S such that S = mK .
The dataset is balanced in that for every k , | S + | = | S − | = m .
We assume (cid:107) µ (cid:63)k (cid:107)
= 1 . 3.
We assume (cid:15) te < 1 and τ > 1 . 4.
We assume 4 α te τ(cid:15) te ≤ 1 .
In Lemma B.1, we show that as long as both the teacher and the student are not trained for too long (and thus their scales α te and α are low), the (perfect) teacher’s supervision is “constructive” on node k for all points in S − and S + .
This is crucial for us to show that distillation is able to denoise the one-hot gradients (which are destructive on S − ).
In the subsequent Lemma B.2, we bound the difference in supervision of a perfect and imperfect teacher.
Using this, we can show that despite imperfection, the supervision of the teacher can be constructive for node k on both S − and S + .
Lemma B.1.
Assuming α te τ ≤ log( K − 2) and α te − α ≥ 2 τ log 31 − κ , then for any x ∈ S + ∪ S − , p te k
( x ) − p k ( x
[Equation] p te k ( x ) ≥ κ .
Proof.
If the point belongs to S − , then:
[Equation]
Then, the ratio is :
[Equation]
Since α te τ ≤ log( K − 2) , we can upper-bound the ﬁrst two terms in the numerator by the last term.
Hence,
[Equation]
If the point belongs to S + , then:
[Equation]
The ratio is
[Equation]
Again, we can upper-bound each term in the denominator by the last term.
[Equation]
Lemma B.2.
Consider a perfect teacher model with weights α te µ (cid:63)k and corresponding probabilities p te ( · ) .
Consider an (cid:15) te -imperfect teacher model with weights w k such that (cid:107) w k (cid:107)
=
α te and (cid:107)
[Equation]
⊥ k (cid:107)
[Equation]
(cid:107) k [Equation] 2 ≤ (cid:15) te .
Let the corresponding probabilities be given by q te ( · ) .
Assuming 4 α te τ(cid:15) te ≤ 1 , on any x ∈ S + ∪ S − ,
[Equation]
Proof.
We have (cid:107) w ⊥ (cid:107) 2 ≤ ( α te ) 2 (cid:15) te 1+ (cid:15) te and (cid:107) w (cid:107) (cid:107) 2 ≥
( α te ) 2 1 1+ (cid:15) te .
We have
[Equation]
We have (cid:107) w (cid:107) (cid:107) ≤ α te and (cid:107) w (cid:107) (cid:107) ≥ α te 1 √ 1+ (cid:15) te ≥ α te (1 − (cid:15) te ) .
We also have | x k · µ (cid:63) k | ≤ τ .
Next, (cid:107) w ⊥ (cid:107) ≤ α te (cid:113) (cid:15) te 1+ (cid:15) te ≤ α te √ (cid:15) te .
Finally, lim D →∞ (cid:107) x ⊥ k (cid:107) ≤ 1 .
Putting these together, we get:
[Equation]
Here, we’ve assumed (cid:15) te < 1 and τ > 1 .
The above gives us a bound on the differences in the logits.
But to bound the probabilities themselves, note that due to the structure of the softmax, both the numerator and the denominator can either grow or diminish by a factor of e 2 α te τ √ (cid:15) te .
Thus, e 4 α te τ √ (cid:15) te ≥ q te k ( x ) p te k ( x ) ≥ e − [Equation] te τ √ (cid:15) te .
We have e
[Equation] Also, given that 4 α te τ(cid:15) te ≤ 1 , e 4 α te τ(cid:15) te ≤ 1 + e · 4 α te τ(cid:15) te .
Rearranging this gives us our ﬁnal inequality.
First, we prove a simpler statement assuming that the teacher is perfect. We will build on this proof for an imperfect teacher.
Lemma B.3.
Consider a student model with weights w k = α µ (cid:63)k such that α ≤ 1 τ log( K − 2) .
Let u k denote the one-hot gradients on this model under the cross-entropy loss on the dataset S .
Consider an perfect teacher model with weights w k such that w k = α te µ (cid:63)k .
Assume the teacher model is early-stopped in that α te ≤ 1 τ log( K − 2) .
Assuming α te ≥ 1 τ log 3 , let κ > 0 denote the level of early-stopping in the student, such that α te − α ≥ 2 τ log 31 − κ .
Let ˜ u
k denote the gradient under distillation loss with this teacher on the above student model.
Assume that the simplifying assumptions of Assumption B.1 hold.
Then,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proof.
We have assumed | S + | = | S − | = m .
Note that for all x in S + , p k ( x ) =
p + for some constant p + > 0 .
Similarly, for all x in S − , p k ( x ) = p − for some constant p + > 0 .
Recall that the gradient descent update on node k be written as u k = u (cid:107) k
+ u ⊥ k , where u ⊥ k · µ (cid:63)k = 0 .
The one-hot gradient at node k can be written as:
[Equation]
We project the sum total of all these gradients along µ (cid:63)k to get (cid:107)
[Equation]
=
[Equation] Next, we want to compute the total magnitude of all the gradients orthogonal to µ (cid:63)k , namely (cid:107) u ⊥ k (cid:107) .
Note that the projection of any sample x orthogonal to µ (cid:63)k is essentially a D − 1 multivariate zeromean Gaussian with variance 1 /D along each direction.
Thus, the projection of u ⊥ k along each of these D − 1 directions is a zero-mean Gaussian with variance equal to mD (cid:0) (1 − p + ) 2 + p 2 − (cid:1) .
Thus (cid:107) u ⊥ k (cid:107) 2 is the sum of squared of D − 1 of these random variables.
Therefore,
[Equation]
Thus, the ratio can be written as:
[Equation]
We can lower bound p − /p
+ using Eq 17 and Eq 20 as:
[Equation]
where we use ατ ≤ log( K − 2) to upper bound each of the denominator terms by ( K − 2)
e − ατ/ 2 .
Plugging this back
[Equation]
we get the following bound on the quality of one-hot gradients:
[Equation]
Now let p te +
and p te − denote the probabilities of a perfect teacher.
We can apply an identical argument for distillation to bound the ratio of the gradients as:
[Equation]
where in the second line we’ve used Lemma B.1 to lower-bound the teacher-student probability gap, subsequently followed by the AM-GM inequality.
Here, we have used the fact that p te − − p − > 0 , or in other words, that the gradients from S − are constructive. We are now ready to state and prove our main theorem:
Theorem B.4. Consider a student model with weights w k = α µ (cid:63) k such that α ≤ 1 τ log( K − 2) . Let u k denote the one-hot gradients on this model under the cross-entropy loss on the dataset S . Consider an imperfect teacher model with weights w k such that (cid:107) w k (cid:107)
=
α te and (cid:107) w ⊥ k (cid:107) 2 (cid:107) w (cid:107) k (cid:107) 2 ≤ (cid:15) te where (cid:15) te > 1 mτ 2 .
Assume the teacher model is early-stopped in that α te ≤ 1 τ log( K − 2) .
Assuming α te ≥ 1 τ log 3 , let κ > 0 denote the level of early-stopping in the student, such that α te − α ≥ 2 τ log 31 − κ .
Let ˜ u
k denote the gradient under distillation loss with this teacher on the above student model.
Assume that the simplifying assumptions of Assumption B.1 hold.
Then, for a sufﬁciently large τ , there exists values of α, κ such that α ∈ [0 , α te ] such that κ 3 ≥ 192 α te τ √ m(cid:15) te
and so we have for any class k that:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
[Equation]
Furthermore, the quality of the teacher weights can be upper bounded by the quality of distillation gradients as :
[Equation] Proof. We will now extend the proof of Lemma B.3 to an imperfect teacher with probabilities q te ( x ) . For convenience, let us assume that for x ∈ S + | p te k ( x ) − q te k ( x ) | ≤ γ + and x ∈ S − , | p te k
( x ) − q te k ( x ) | ≤ γ − for some constants γ + and γ − .
We can say that:
[Equation]
If we have γ − ≤ (cid:15) (cid:48) 2 κp te − for some (cid:15) (cid:48) , from Lemma B [Equation]
[Equation] we have γ − ≤ (cid:15) (cid:48) 2 κ ( p te − − p − ) .
We can make a similar claim for γ + .
Thus,
[Equation]
As for the orthogonal gradient term [Equation] recall that (cid:107) u ⊥ k (cid:107) 2 is essentially the summation of D − 1 squared random variables.
With a perfect teacher [Equation] we could write each of the D − 1 variables as a sum of m Gaussians each scaled by a constant probability ( p te + − p + ) and ( p te − − p − ) independent of the sample x .
However, with the imperfect teacher, the probabilities are themselves random variables dependent on the draw of x .
So we ﬁrst bound the gap between this norm and the norm under the perfect teacher.
Without loss of generality [Equation] let us denote the D − 1 dimensions of x k that is orthogonal to µ k(cid:63) as x k, 1 , x k, 2 , . . .
, x k,D − 1 .
Then,
[Equation] [Equation]
For each j , A j is the squared of the sum of the absolute value m Gaussians with variance γ 2+ D and another m with variance γ 2 − D .
The means of these variables are γ + (cid:113) 1 D 2 π and γ − (cid:113) D 2 π respectively.
The expectation of A j is the squared sum of the means and the sum of the variances of all these variables, equalling m ( γ 2+ + γ 2 − ) D
+ m 2 (cid:16)
γ + (cid:113) D 2 π + γ − (cid:113) 1 D 2 π (cid:17) 2 .
The second term can be upper bounded by m 2 (cid:16)
γ 2+ D + γ 2 − D (cid:17) .
Therefore, we can conclude that lim D →∞ (cid:80) D − 1 j {+=
+}
A j ≤ 3 m 2 ( γ 2+ + γ 2 − ) .
For each j , B j can be expanded into (2 m ) 2 terms.
The expected value of each of these terms can be bounded in the form of ( p te k
( x ) − p k
( x )) (cid:0) γ
+ D (cid:1) or ( p te k
( x ) − p k
( x )) (cid:0) γ − D (cid:1) .
Thus, the expected value of
B j is m (cid:0) γ − D + γ + D (cid:1) (cid:80) x
( p te k ( x ) − p k
( x )) .
Let lim D →∞ B j = m ( γ − + [Equation]
x [Equation]
x [Equation] p [Equation]
x ))
Now, if we have that √ mγ − ≤ (cid:15) (cid:48) 4 κp te − and √ mγ + ≤ (cid:15) (cid:48) 4 κp te + , for some (cid:15) (cid:48) we have:
[Equation]
In the second term, we can upper bound ( κp te + ) 2 + ( κp te − ) 2 by (( p te + − p + ) 2 + ( p te − − p − ) 2 ) using
Lemma B.1.
In the third term, we can similarly upper bound κp te + and κp te − followed by an AM-GM inequality to upper-bound by 2(( p te + − p + )
+ ( p te − − p − ) 2 ) .
Therefore,
[Equation]
Then, we have that for our imperfect teacher:
[Equation]
[Table] The last step follows if we have (cid:15) (cid:48) ≤ κ 2 8 .
Plugging this back in our required inequality of the form √ mγ − ≤ (cid:15) (cid:48) 4 κp te − , we need √ mγ − ≤ κ 3 16 p te − and similarly for γ + .
From Lemma B.2, this means we need κ 3 ≥ 192 α te τ √ m(cid:15) te .
Now, we need to make sure that there are possible values of α for which the above lower-bound on κ can be realized.
Note that κ can only attain values of 1 − 3 e − ( α te − α ) τ 2 for various values of α .
This follows from the statement of Lemma B.1.
Thus, we want 1 − 3 e − ( α te − α ) τ 2 > (cid:16) 192 α te τ √ m(cid:15) te (cid:17) for there to be any values of α to achieve our lower-bound on κ .
Furthermore, we have that (cid:15) = cmτ 2 for some c > 1 .
Thus, we need 1 − 3 e − ( α te − α ) τ 2 > (cid:0) 192 α te √ c (cid:1) to have feasible solutions for α .
This is possible as long as τ is sufﬁciently large. [Table]
C E XPERIMENTS AND PLOTS C. D ETAILS OF EXPERIMENTAL SETUP
We present details on relevant hyper-parameters for our experiments.
Model architectures .
For all image datasets (CIFAR10, CIFAR100, Tiny-ImageNet), we use ResNetv2 (
He et al., 2016a) and MobileNet-v2 (Sandler et al., 2018), models.
Speciﬁcally, for CIFAR, we consider the CIFAR ResNet- { 56 , 20 } family and MobileNet-v2 architectures; for Tiny-ImageNet, we consider the ResNet- { 50 , 18 } family and MobileNet-v2 architectures; For all ResNet models, we employ standard augmentations as per He et al.
(2016b).
For all text datasets (MNLI, AGNews, QQP, IMDB), we ﬁne-tune a pre-trained RoBERTa
(Liu et al., 2019) model.
We consider a combinations of cross-architecture- and self-distillation with RoBERTa -Base, -Medium and -Small architectures.
Training settings .
We train using minibatch SGD applied to the softmax cross-entropy loss.
For all image datasets, we follow the settings in Table 1.
For all text datasets, we use a batch size of 64 , and train for 25000 steps.
We use a peak learning rate of 10 − 5 , with 1000 warmup steps, decayed linearly.
For the distillation experiments, we use a distillation weight of 1 . 0 .
We use temperature τ = 2 . 0 for MNLI, τ = 16 . 0 for IMDB, τ = 1 . 0 for
QQP, and τ = 1 . 0 for AGNews.
C. A DDITIONAL RESULTS C.2. S CATTER PLOTS OF PROBABILITIES
In this section, we present additional scatter plots of the teacher-student logit-transformed probabilities for the class corresponding to the teacher’s top prediction:
Fig 5 (for CIFAR100), Fig 6 (for
TinyImagenet), Fig 7 (for CIFAR10), Fig 8 (for MNLI and AGNews), Fig 9 (for self-distillation on
QQP, IMDB and AGNews) and Fig 10 (for cross-architecture distillation on language datasets).
We ﬁnd underﬁtting of the hard points in a majority of the cases, with a few notable exceptions and caveats :
For MobileNet self-distillation on CIFAR100, and for a majority of the CIFAR10 experiments, we ﬁnd no underﬁtting of the harder points on the training dataset .
However, we do ﬁnd underﬁtting of the harder points on the test dataset .
Interestingly, we also ﬁnd an underﬁtting of easier points in the training dataset.
In the language datasets, we generally ﬁnd the plots to be different in pattern from the image datasets.
In particular we ﬁnd that, for harder points, there is both signiﬁcant underﬁtting [Table]
and overﬁtting. For easier points, there is less deviation, and if any, the deviation is from overﬁtting.
We interpret this as the regularization from distillation inducing less precision on the harder points, rather than an underﬁtting per se.
Our patterns generally break down in the cross-architecture settings of language datasets (with the exception of a couple of settings).
We suspect that this may be because certain cross-architecture effects dominate over the underﬁtting, which is a much more subtle effect.
The underﬁtting we observe in the language datasets are not always associated with an improvement in the student’s generalization.
e.g., we ﬁnd no improvement in AGNews, and a decrease in generalization in QQP (see Table 2).
(Note that we didn’t ﬁnetune hyperparameters in these settings to make distillation work.)
C.2. T EACHER ’ S PREDICTED CLASS VS . GROUND TRUTH CLASS
Recall that in all our scatter plots we have looked at the probabilities of the teacher and the student on the teacher’s predicted class i.e., ( p te y te ( x ) , p st y te ( x )) where y te .
= argmax y (cid:48) ∈
[ K ] p te y (cid:48)
( x ) .
Another natural alternative would have been to look at the probabilities for the ground truth class, ( p te y (cid:63) ( x ) , p st y (cid:63) ( x )) where y (cid:63) is the ground truth label.
We chose to look at y te however, because we are interested in the “shortcomings” of the distillation procedure where the student only has access to teacher probabilities and not ground truth labels.
Nevertheless, one may still be curious as to what the probabilities for the ground truth class look like.
First, we note that the plots look almost identical for the training dataset owing to the fact that the teacher model typically ﬁts the data to zero training error (we skip these plots to avoid redundancy).
However, we ﬁnd stark differences in the test dataset as shown in Fig 11.
In particular, we see that the underﬁtting phenomenon is no longer prominent, and almost non-existent in many of our settings. [Figure]
This is surprising as this suggests that the student somehow matches the probabilities on the ground truth class despite not knowing what the ground truth class is.
This also tells us the mechanism by which the student manages to correct some of the teacher’s labels: by reducing the probability of non-ground-truth classes rather than increasing the probability of the ground truth class.
Indeed, in
Fig we verify this to be the case.
We dissect the test data plots into four parts depending on which of the teacher and student model gets the label right.
We consistently ﬁnd that the points that the student gets right but the teacher not, fall in the underﬁt set of points (AGNews, QQP, IMDB, MNLI):
We report plots for various cross-architecture distillation settings involving RoBERTa models.
While we ﬁnd signiﬁcant student-teacher deviations in these settings, our typical patterns do not apply here.
We believe that effects due to “cross-architecture gaps” may have likely drowned out the underﬁtting patterns, which is a more subtle phenomenon that shines in self-distillation settings.
Interestingly, the underﬁt set of points is roughly equivalent to the set of all points where at least one of the models is incorrect This suggests that in its attempt to underﬁt some of the points, the student can get some incorrect, potentially points which are inherently fuzzy (e.g., they are similar to multiple classes).
Theorem 4.2 suggests that the student would use these points to improve its features, thereby increasing accuracy on other points that are not inherently as fuzzy.
Finally, we note that previous work (Lukasik et al., 2021) has examined deviations on ground truth class probabilities albeit in an aggregated sense (at a class-level rather than at a sample-level).
While they ﬁnd that the student tends to have lower ground truth probability than the teacher on problems with label imbalance, they do not ﬁnd
any such difference on standard datasets without imbalance.
This is in alignment with what we ﬁnd above.
C.2. T HE EFFECT OF LOSS SWITCHING
We provide additional results on loss-switching in this section.
Fig 14 presents the accuracy values under a one-hot-to-distillation switch.
Fig 15 presents the accuracy values under a distillation-to-onehot switch.
Fig 13 presents student-teacher scatter plots over the course of training under all the four methods (with and without loss switches).
Fig 16 presents the trajectory of accuracies for ResNet on TinyImagenet under all the four settings. It is worth noting a nuance in this story.
First, switching to one-hot typically preserves the gains of distillation if it is not run for too long — and in the rare case of ResNet50 self-distillation on
TinyImagenet even outperforms distillation — which supports earlier ﬁndings (Cho & Hariharan, 2019; Zhou et al., 2020; Jafari et al., 2021) that advocate a soft ﬁnal switch to one-hot training. Overall, this seems to suggest that, the early phase of distillation may be a sufﬁcient requirement to gain all the beneﬁts of distillation (even if our earlier experiments suggest that they are not necessary ). However , we consistently ﬁnd that switching to and training with one-hot for a sufﬁciently long time deteriorates the gains made by distillation (Fig 1b, (Fig 16).
C. A BLATIONS
Here, we conduct two experiments showing that the underﬁtting phenomenon holds under other conditions, speciﬁcally (a) longer training of the student and (b) smaller batch sizes and learning rate.
Longer training: In Fig 17 (left two images), we conduct experiments where we run knowledge distillation with the ResNet-56 student on CIFAR100 for 2 .
3 × longer ( 50 k steps instead of 21 .
6 k steps overall) and with the ResNet-50 student on TinyImagenet for about 2 × longer ( 300 k steps over instead of roughly 150 k steps).
We ﬁnd the resulting plots to continue to have the same underﬁtting as the earlier plots.
It is worth noting that in contrast, in a linear setting, it is reasonable to expect the underﬁtting to disappear after sufﬁciently long training.
Therefore, the persistent underﬁtting in the non-linear setting is remarkable and suggests one of two possibilities:
• The underﬁtting is persistent simply because the student is not trained sufﬁciently long enough i.e., perhaps, when trained 10 × longer, the network might end up ﬁtting the teacher probabilities perfectly.
• The network has reached a local optimum of the knowledge distillation loss and can never ﬁt the teacher perfectly.
This may suggest an added regularization effect in distillation, besides the eigenspace regularization that introduces the hard-point-underﬁtting in the ﬁrst place.
This unknown regularization effect may perhaps disappear if the learning rate was even smaller or if the student network was larger than the teacher network, allowing to reach higher precision.
Smaller batch size/learning rate: Finally, in Fig 17 (right image), we also verify that in the
CIFAR100 setting if we set peak learning rate to 0 .
1 (rather than 1 . 0 ) and batch size to 128 (rather than 1024 ), our observations still hold.
C.3. S CATTER PLOT FOR OTHER METRICS
In the main paper, recall that we look at student-teacher deviations via scatter plots of the probabilities of either models on the teacher’s top class, after applying a logit transformation .
It is natural to ask what these plots would look like under other variations.
We explore this in Fig 18 for the CIFAR
ResNet-56 self-distillation setting.
[Figure]
[Figure]
For quick reference in the top row of Fig we ﬁrst show the standard logit-transformed probabilities plot where we ﬁnd the underﬁtting phenomenon.
In the second ﬁgure, we then directly plot the probabilities instead of applying the logit transformation on top of it.
We ﬁnd that the underﬁtting
30000 Step 0.
T e s t a cc u r a c y
G r a d u a l l o ss s w i
t c h
Res56 to Res56 CIFAR
Method KD
OneHot KD to OneHot
OneHot to KD
However, we note that for the Tiny-Imagenet, there is a small window of time for which there is a signiﬁcant increase in accuracy under a switch to one-hot – this is later destroyed with longer training.
Nevertheless, the overall ﬁnding here reinforces the intuition that the one-hot loss results in destructive gradients as discussed in Section 4.2.
phenomenon does not prominently stand out here (although visible upon scrutiny, if we examine below the X
= Y line for X ≈ 0 ).
This illegibility is because small probability values tend to concentrate around 0 ; the logit transform however helps magnify the behavior of small probability values.
For the third plot, we provide a scatter plot of entropy values of the teacher and student probability values to determine if the student distinctively deviates in terms of entropy from the teacher.
It is not clear what characteristic behavior appears in this plot.
In the bottom plots, on the Y axis we plot the KL-divergence of the student’s probability from the teacher’s probability.
Along the X axis we plot the same quantities as in the top row’s three plots.
Here, we observe interesting behavior across the board: the KL-divergence of the student tends to be higher on teacher’s harder points, where “hard points” can be interpreted as either points where its top probability is low, or points where the teacher is “confused” enough to have high entropy.
C. V ERIFYING THE EIGENSPACE VIEW EMPIRICALLY
In this section, we demonstrate the eigenspace view
Sec 4.1 in practice even in situations where our theoretical assumptions do not hold good.
We go beyond our theoretical assumptions in the following ways:
We consider two self-distillation settings: ﬁrst, a linear random features model trained on a noisy version of the MNIST dataset and next a 2-layer multi-layer perceptron (MLP) model trained on a subset of the MNIST dataset.
Both are trained with the cross-entropy loss (and not the squared error loss as used in our theory).
We consider a multi-class problem and so the output is not a single scalar value.
We use a ﬁnite learning rate with minibatches and Adam.
We provide exact details of these two settings at the end of the section
Observations.
In short, we ﬁrst observe in Fig 19 that in both these settings, the harder points of the teacher are underﬁt as usual.
At the same time, we observe in Fig 20 and Fig 21 that the convergence rate of the student is much faster along the top eigendirections, when compared to the teacher (explained shortly).
We show train-test accuracy plots in Fig 22.
To verify our eigenspace view theory, we show 2D projections of the trajectory of the teacher and student along two eigendirections picked at random (with the higher eigendirection one plotted along the X axis).
The ﬁnal solution found is marked by a ‘ (cid:63) ’, (typically at the top-right of the plot).
Here, we observe that the teacher already has an implicit bias towards converging faster along the top eigendirection (as is well-known).
This can be inferred from the fact that the trajectories move quickly along the X axis towards its ﬁnal X axis value, before making progress along the Y axis.
But more interestingly, we ﬁnd that for the student, the bias in this trajectory is more exaggerated; the student converges faster towards the ﬁnal X value of the teacher than the rate at which teacher gets there.
In doing so, the student covers a completely different part of parameter space never traversed
[Figure]
by the teacher .
In this sense, the bias of distillation is an exaggerated but non-identical version of the bias of standard gradient descent.
In the linear setting, we ﬁnd this bias to hold in all of the 15 different random pairs of eigendirections, while in the MLP setting this holds in all but two of the 15 different random pairs of eigendirections.
Note that these eigendirections are picked at random from the set of all directions and not cherry picked.
Speciﬁcally, the top direction is sampled at random from the top 15 directions (without replacement), and the bottom from the directions with indices in [20 , 60] (without replacement).
To compute the eigendirections, in the case of the random features setting, we compute the directions of the random-features-transformed data.
We then project the weight matrix W along the eigendirection v , and then take the (cid:96) 2 norm (cid:107) W (cid:62) v (cid:107) 2 to compute the projection.
Note that in the theory, we dealt with a model with a scalar-valued output, and so W (cid:62) v would have been a scalar.
In the case of the 2-layer MLP, we directly compute the eigendirections of the input data.
We then take the ﬁrst layer matrix W and compute the projection similarly.
Takeaway.
We ﬁnd that distillation leads to an exaggerated bias in terms of the rate of convergence along various eigendirections.
This happens even in a setting trained with cross-entropy loss, and with a non-linear neural network, going beyond our theoretical assumptions.
Thus, our insights from the linear regression setting in Sec 4.1 apply to a wider range of settings.
We also ﬁnd that underﬁtting happens in these settings, reinforcing the connection between the eigenspace regularization effect and underﬁtting.
Other details.
For the random features setting, we train on a subset of 128 datapoints with
ReLU random features.
The training data has 0 .
25 probability of a mislabeling.
We use a batch size of 16 and learning rate of 0 . 001 .
Both teacher and student are trained for 40 epochs, and the student with a temperature of 5 .
In the MNIST setting, we use a 2-layer MLP with 1000 hidden units trained on a subset ofdatapoints with no label noise.
We use a batch size of 16 and learning rate of 0 . 0001 .
Both teacher and student are trained for 20 epochs; the distillation loss uses a temperature of 4 .
All other details are identical to the previous setting.
C. V ERIFYING THE GRADIENT SPACE VIEW IN PRACTICE
We now demonstrate that the conclusions of Theorem 4.2 indeed hold good empirically (even in a settings where our theoretical assumptions don’t exactly hold e.g., we will use mini-batch instead of full-batch).
To do this, we build a synthetic perfectly classiﬁable dataset (very similar to that used in
Theorem 4.2) with class similarities encoded at the logit level.
We then show that distillation helps in better aligning the weights of that dataset, thus verifying that it has experienced “denoised” gradients.
[Table] [Figure]
setting with multi-class cross-entropy loss, ﬁnite learning rate etc.,
Dataset details.
We consider a K -class classiﬁcation dataset ( K = 10) where the i th datapoint’s features can be written as a K -“channel” input x i =
( x ( i ) 1 , . . .
, x ( i ) K ) where each channel x ( i ) k
∈
R
D is D -dimensional
( D = 100 ).
For each k , we pick a global “ground truth” class vector µ (cid:63)k sampled from the D -dimensional normal distribution.
Assuming a uniform distribution over K classes, given a label y , we generate y th channel input x y as x y = α µ (cid:63)y + (1 − α )
z y , where α = 0 .
01 and z y is uniformly sampled from the D -dimensional unit hypersphere truncated to x y · µ (cid:63)y > 0 .
Here, the α µ (cid:63)y is added so as to provide sufﬁcient non-zero margin for the points from the decision boundary.
Next, we also set the co-ordinates corresponding to a few other classes.
We ﬁrst randomly pickother classes meant to be “similar non-target” classes.
Then for each such class k , we set x k = β
( α µ (cid:63)k + (1 − α )
{+z k ) where α and z k are sampled as before, and β is a random value rescaled Beta distribution with parameters a = 4 and b = 1 rescaled by a multiplicative factor equal to 0 . 
+}
· ( x y · µ (cid:63)y ) .
This scaling factor ensures that x k · µ (cid:63)y is smaller than the ground truth margin x y · µ (cid:63)y ) , and so the dataset remains linearly separable.
Besides the target class and the 3 non-target similar classes, all other co-ordinates are set to be zero for x .
Training details.
For training, we consider a linear classiﬁer that has K output nodes, each drawing input from the corresponding D dimensions of its class.
We use Adam with a learning rate of 0 . 1 , with batch size 128 , and 512 training
datapoints.
We train both teacher and student for 10 epochs, and the student with temperature
Observations.
In this setting, although both the teacher and student have high accuracy, the selfdistilled student outperforms the teacher by about 0 .
7% (see Fig 23 left).
Next, we compute the number of ( x , k ) pairs in the training set whose gradients have the correct sign under either loss.
For each point x and each node k , we check whether x k · µ (cid:63) k > 0 if and only
[Figure]
We note that in the linear case (left), there is little difference in the accuracies of the student and the teacher, likely because this is a very simple setting where ignoring the lower eigendirections (as seen in Fig 20 to distillation) has little effect.
But for the MLP setting (right), we ﬁnd that the student achieves higher test accuracies than the teacher for a given training accuracy.
This is evidence that the student uses “better” directions (i.e., top eigendirections) to ﬁt the data.
if the target probability under the given loss is non-zero.
This can verify our key proof idea that distillation denoises the gradients even in the absence of explicit noise in the dataset.
According to this computation, we ﬁnd that for one-hot loss, only 73 .
22% of such pairs have the right sign of gradient, while for distillation this is 97 .
77% , thus verifying that distillation is indeed able to denoise.
Inspired by the formulation in Theorem 4.2, we then analyze the cos-similarity between the weights learned for each class, and its corresponding class vector.
We ﬁnd in Fig 23 middle that the student has a better alignment than the teacher, despite the fact that the student is trained with the teacher’s logits (and no extra information is given).
This proves the main result of Theorem 4.2.
Thus, even in a perfectly classiﬁable dataset, we are able to make the student outperform the teacher because of class similarities at the logit level.
Notably, we also observe the underﬁtting phenomenon in this setting (see Fig 23 right).
This suggests that the underﬁtting phenomenon is indeed connected to how distillation denoises gradients in the presence of class similarities.