S CAFFOLDING A S TUDENT TO I NSTILL K NOWLEDGE
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel knowledge distillation (KD) method to selectively instill teacher knowledge into a student model motivated by situations where the student’s capacity is significantly smaller than that of the teachers.
In vanilla KD, the teacher primarily sets a predictive target for the student to follow, and we posit that this target is overly optimistic due to the student’s lack of capacity.
We develop a novel scaffolding scheme where the teacher, in addition to setting a predictive target, also scaffolds the student’s prediction by censoring hard-to-learn examples.
The student model utilizes the same information as the teacher’s soft-max predictions as inputs, and in this sense, our proposal can be viewed as a natural variant of vanilla KD.
We show on synthetic examples that censoring hard-examples leads to smoothening the student’s loss landscape so that the student encounters fewer local minima.
As a result, it has good generalization properties.
Against vanilla KD, we achieve improved performance and are comparable to more intrusive techniques that leverage feature matching on benchmark datasets.
I NTRODUCTION
A fundamental problem in machine learning is to design efficient and compact models with near state-of-the-art (SOTA) performance.
Knowledge Distillation (KD) (Zeng & Martinez, 2000; Bucila et al., 2006; Ba & Caruana, 2014; Hinton et al., 2015) is a widely used strategy for solving this problem wherein the knowledge from a large pre-trained teacher model with SOTA performance is distilled onto a small student network.
Vanilla KD.
Hinton et al.
(2015) proposed the popular variant of KD by matching the student soft predictions, s ( x ) with that of the pre-trained teacher, t ( x ) on inputs x .
Informally, during student training, an additional loss term, D KL ( t ( x ) , s ( x )) is introduced that penalizes the difference between student and teacher predictive distributions using Kullback-Leibler (KL) divergence.
This promotes inter-class knowledge learned by the teacher.
We will henceforth refer to Vanilla KD as KD.
Capacity mismatch between student and teacher.
One of the primary issues in Vanilla KD is that the loss function is somewhat blind to the student’s capacity to interpolate.
In particular, when the student’s capacity is significantly lower than the teacher’s, we expect the student to follow the teacher only on those inputs realizable by the student.
We are led to the following question: What can the teacher provide by way of predictive hints for each input, so that the student can leverage this information to learn to its full capacity?
Our Proposal: Scaffolding a Student to Distill Knowledge (DiSK).
To address this question, we propose that the teacher, during training, not only set a predictive target, t ( x ) , but also provide hints on hard to learn inputs.
Specifically, the teacher utilizes its model to output a guide function, g ( x ) , such that the student can selectively focus only on those examples that it can learn
• if g ( x ) ≈ 1 , teacher discounts loss incurred by the student on the input x . • if g ( x ) ≈ 0 , teacher signals the input x as learnable by student.
With this in mind we modify the KL distance in the Vanilla KD objective and consider,
D KL ( t ( x ) , ϕ ( s ( x ) , g ( x ))) , where ϕ ( s , g ) , which will be defined later, is such that, ϕ ( s , 0) =
s if g offers no scaffolding.
However, we must impose constraints on the guide function g to ensure that only hard-to-learn examples are scaffolded - in the absence of such constraints, the guide can declare all examples to be hard, and the student would no longer learn.
We propose to do so by means of a budget constraint B ( s , g ) ≤ δ to ensure that the guide can only help on a small fraction of examples.
While more details are described in Sec. 3, we note that, in summary, our proposed problem is to take the empirical linear combination of the aforementioned KL distance and a cross-entropy term as the objective, and minimize it under the empirical budget constraint.
We emphasize that g ( x ) is used only during training .
The inference logic for the student remains the same as there is no need for g ( x ) during inference.
The guide function supported student training has three principal benefits.
The benefits are explored in Sec. 2.
• Censoring Mechanism.
Our guide function censors examples that are hard to learn for the student.
In particular, when there is a large capacity gap, it is obvious that the student cannot fully follow the teacher.
For this reason, the teacher must not only set an expectation for the student to predict, but also selectively gather examples that the student has the ability to predict.
• Smoothen the Loss landscape.
We also notice in our synthetic experiments that whenever scaffolding is powerful, and can correct student’s mistakes, the loss landscape undergoes a dramatic transformation.
In particular, we notice fewer local minima in the loss viewed by the guided student.
• Good Generalization.
The solution to our constrained optimization problem in cases where the guide function is powerful can ensure good student generalization.
Specifically, we can bound the statistical error essentially in terms of student complexity, and as such do not suffer additional complexity due to the teacher. Contributions.
We summarize our main results.
• We develop a novel approach to KD that exploits teacher representations to adjust the predictive target of the student by scaffolding hard-to-learn points.
This novel scaffolding principle has wider applicability across other KD variants, and is of independent interest.
• We design a novel response-matching KD method (Gou et al., 2021) which is particularly relevant in the challenging regime of large student-teacher capacity mismatch. We propose an efficient constrained optimization approach that produces powerful training scaffolds to learn guide functions. • Using synthetic experiments, we explicitly illustrate the structural benefits of scaffolding. In particular, we show that under our approach, guides learn to censor difficult input points, thus smoothening the student’s loss-landscape and often eliminating suboptimal local minima in it.
• Through extensive empirical evaluation, we demonstrate that the proposed DiSK method – yields large and consistent accuracy gains over vanilla KD under large student-teacher capacity mismatch (upto 5% and 2% on CIFAR-100 and Tiny-Imagenet).
– produces student models that can get near-teacher accuracy with significantly smaller model complexity (e.g. 8 × computation reduction with ∼ 2% accuracy loss on CIFAR-100).
– improves upon vanilla KD even under small student-teacher capacity mismatch, and is even competitive with modern feature matching approaches.  I LLUSTRATIVE E XAMPLES We present two synthetic examples to illustrate the structural phenomena of the censoring mechanism and smoothening of student’s loss landscape enabled by the scaffolding approach DiSK, which lead to globally optimal test errors. We defer exact specification of the algorithm to Sec.3.
Example 1 (1D Intervals) .
Consider a toy dataset with one dimensional features x ∈ [0 , 9] and binary class labels
y ∈
{ Red , Blue } as shown in Figure 1.
There are two Blue labelled clusters as in [2 , 3] and in [5 , 7] .
The remaining points are labelled as Red .
We sample 1000 i.i.d.
data points as the training set and 100 data points as the test set with balanced data from both classes.
We describe the details of the experiment setup such as models and training details in Appendix A.1.
Teacher T belongs to the 2-interval function class, and the capacity-constrained student S belongs to the 1-interval function class.
Since teacher capacity is sufficient to separate the two classes without error, it learns the correct classifier (see Figure 1).
In contrast, the best possible student hypothesis cannot correctly separate the two classes.
Hence, the student will have to settle onto one of the many local minima.
We show these minima and the contour plot for the student in Figure 1.
We present the results of training student models with different initializations in Table 1.
KD suffers from bad local minima.
KD loss landscape contains many local minima (see Figure 1).
Due to a big gap between student and teacher capacity, it is unable to help the student discern between these minima.
Hence, KD fails to distinguish between the different minima (see Table 1). [Figure] [Table]
DiSK censors interval [2 , 3] and in addition focuses training on learnable datapoints.
If we analyze the guide function at the end of the training, we see that it covers (censors) the first Blue cluster.
Indeed, both clusters are not simultaneously l earnable with available student capacity.
However, if we were to censor the interval [2 , 3] , then the problem becomes realizable for the student model.
The guide function thus captures the excess capacity of the data.
DiSK smoothens loss landscape.
The guide function and the budget constraint enable our method to have a smooth loss landscape thanks to the guide-function censoring points, which eliminate the local minima.
Hence, DiSK solution lands in the global minimum with high probability.
Example 2 (2D Gaussians).
Consider another toy dataset with two dimensional features x ∈ Rand three class labels y ∈
{ Red , Green , Blue } .
Here we wish to show that DiSK can allow for globally optimal solutions reaching 100% accuracy, which appears unachievable with cross-entropy minimization regardless of data size.
Figure 2a shows the labelled data.
There are six cluster centers, two with each class label.
Data points are drawn using Gaussian balls around the cluster centers with small radii.
We sample 1000 i.i.d.
data points as the training set and 1000 data points as the test set with equal representation from all three classes.
We provide details (hypothesis classes, learning procedure, etc.) in Appendix.
A.2.
The teacher is a 3-layer neural network with 8, 16, and 3 neurons.
The student is a 2-layer neural network with 2 and 3 neurons.
We point out that the teacher being an over-parameterized network in this feature space, easily learns the correct decision boundary.
While the student being severely constrained network suffers in learning the task.
Different training runs lead to different local minima.
We show teacher solution and student local minima in Figure 2a.
The contour plots for the student models under KD loss and DiSK loss are shown in Figure 2b-2c using Li et al.
The results are similar to the 1D example - KD converges to a poor local minimum with at least 43% of the initializations, while in contrast, DiSK escapes these by focusing on the learnable part of the input space (Fig. 2c), converging to the global minimum nearly always (Table 1).
To conceptualize our findings in these examples, let us attempt to intuitively infer the examplecensoring, landscape-smoothening, and good generalization, by utilizing the following conditions that appear to be satisfied for these synthetic examples.
Realizability.
Suppose we are in a situation where the guide function g ∈ G is sufficiently powerful that there is a student and guide function capable of interpolation,
i.e., predictions supported by the guide function, ϕ ( s , g ) , interpolates to mimic the labels.
[Figure] Example: For instance, consider a binary classification problem with the labels y ∈ {− 1 , 1 } .
Let ϕ ( s, g )
=
y ( s + g ) with s ( x ) ∈ [ − 1 , 1] .
Our realizability condition is that we always satisfy y ( s ( x ) + g ( x )) > 0 .
As such, this leads to the condition that if ys ( x ) ≤ 0 , then yg ( x ) > 0 .
Therefore, E [ 1 [ ys ( x ) < 0] ] ≤ E [ 1 [ ys ( x ) < 0 ,yg ( x ) > 0] ] ≤ E [ 1 [ yg ( x ) > 0] ] .
Small Guide Function Capacity.
In addition to realizability suppose the class of guide functions g ∈ G has a small capacity (for instance, small VC dimension).
For our case this condition is satisfied because our guidance function is obtained by using an MLP on teacher’s last layer features.
Example: Continuing with the example above, say we now have m training instances, ( x i , y i ) , i ∈ [ m ] , ˆ g ( x ) is guide function output of DiSK.
We can infer by standard statistical learning results (Shalev-Shwartz & Ben-David, 2014) that, for the estimated function ˆ g ∈ G , it follows with probability greater than 1 − η that E [ 1 [ y {+ˆ g ( x ) > 0] ] ≤ 1 m (cid:80) mi =1 
+}
[ y i ˆ g ( x i ) > 0]
+ (cid:113) V
[Equation]
As a result, we can say that if there is a student, s ( x ) (not necessarily that output by DiSK), which complements ˆ g ( x ) and satisfies realizability, then with probability greater than 1 − η :
[Equation]
Remarks.
Note that the key point here is that the student capacity is considerably larger since we typically train an entire DNN, and student complexity-based bound can be vacuous.
While the guidance function does bound the student generalization error in terms of guide function complexity, there are strong caveats— we require the strong assumption of realizability on the entire domain, and additionally, while the guide function can witness student error, we are not in a position to precisely estimate it without additional training data.
Furthermore, the RHS is a relaxed bound on the student training error.
This motivates having a budget constraint to ensure that student learns with small training error, a point that will be clearer in the sequel.
D EFINITIONS AND F ORMULATIONS
Notation . Let X and Y = { 1 , . .
, C } be the feature and label spaces respectively, focusing on a Cclass classification task.
We assume that we have a training set of N i.i.d.
data points D
= { x i , y i } Ni =1 , where x i ∈ X
and y i ∈ Y .
We use symbols S and T to denote the student and teacher models respectively.
Let l S ( x ) ∈ R |Y| and l T ( x
) ∈ R |Y| be the scores (logits) vector predicted by S and T on input x .
We use τ as the temperature used to soften the probability distribution.
We write the resulting softened student and teacher probabilities as s ( x ) and t ( x ) , i.e.,
[Equation]
τ
[Equation]
The standard soft prediction probabilities correspond to s 1 ( x ) and t 1 ( x ) .
We will use symbols s τy ( x ) to denote the y th coordinate in s τ ( x ) , and similarly t τy ( x ) . The hard prediction of the student is p S ( x ) =
arg max y ∈Y s 1 ( x ) , and similarly p T ( x ) =
arg max y ∈Y t 1 ( x ) for the teacher.
We use g ( x ) ∈ [0 , 1] to denote the helper guide function for the student and teacher pair ( S, T ) .
Guide takes input x and any other feature processed by ( S, T ) pair and decides whether or not the student needs help on the input x .
Finally, we define ReLU activation as ( · ) + = max(0 , · ) .
3. V ANILLA K NOWLEDGE D ISTILLATION
Vanilla KD relaxes the 0 − 1 error between the student predictions and the true labels y using the cross-entropy loss L CE .
Similarly, KD denotes the distance between the student and teacher softened probability distributions using the KL divergence.
We summarize the corresponding losses as,
[Equation]
For hyperparameters α ∈ [0 , 1] , τ > 0 , KD minimizes a mixture of the above losses, as shown below
[Equation]
3. S ELECTIVE K NOWLEDGE D ISTILLATION .
KD attempts to transfer the knowledge from the teacher to the student on all training data points, which is a sub-optimal objective when there is a capacity mismatch between the student and the teacher.
Instead, we propose distilling selective knowledge (DiSK) to allow the student to selectively ignore some hard-to-learn data points during training, transferring the teacher’s knowledge only on easy-to-learn inputs, and matching the learning to student capacity.
Our objective is to minimize min s ,g,δ N N
[Equation]
Distance between T and S with help of g subject to.
[Equation]
Support budget constraint on g
[Equation]
where ϕ interpolates student predictions based on the guide’s help.
The divergence term helps in minimizing the distributional distance between the teacher and student probabilities after the guide g is included.
While the budget term in the optimization constrains the helper g to provide help only when necessary, the amount of help given to the student should be within the budget δ ∈ [0 , 1] .
Function g Construction.
As previously stated, we use the teacher’s last layer features and soft predictions as input to the guide g .
The guide is structured as a light-weight three-layer neural network with these inputs, with a sigmoid activation at the last layer.
We re-emphasise that g is not used at inference time, and only aids training.
More details are left to Appx. A.4.
Relaxed Losses, Lagrangian & Optimization Algorithm.
We relax Eq. 2 and construct a Lagrangian by integrating the constraint into the minimization.
Budget constraint relaxation.
We relax the indicator loss in the budget to a cross-entropy, and treat δ as a hyperparameter to get
[Equation]
We view the scaffold as a way to allow the student to interpolate the uncensored data. This suggests that a good initialization for the budget is the error of cross-entropy trained model, when the student does not have the benefit of teacher supervision. Using this insight, we scan the budget in a small interval around this initialization.
Distillation objective.
Motivated from KL loss, we construct a distance loss with guide function as, [Equation]
[Equation]
[Equation]
[Equation]
We point out two modifications in the distillation loss.
First, L dist explicitly adds guide value to softened student probabilities in selected class indices.
The class indices guide function adds value are picked as top K classes based on the teacher probabilities for any input
x i where K is a hyperparameter of our method.
The rest of the class indices do not get any value from g .
Second, we use different temperature parameters for teacher and student.
Temperature parameter for teacher, τ , is a hyperparameter.
The student temperature is found by minimizing the KL loss between teacher softened probabilities and the student softened probabilities over the training dataset, .i.e
[Equation]
Similar to KD, we incorporate standard cross entropy loss between student model predictions and the ground truth labels for stability.
We construct our Lagrangian by combining Eq.
3 and 4 as,
[Equation]
where α is a hyperparameter and λ is the dual parameter of DiSK.
We optimize Obj.
5 using a primal dual update scheme as explained in Algorithm 1.
Primal Parameter Updates ( s , g ).
We learn the student s and the guide function g using alternating minimization.
We approximate arg min with running SGD for a small number of epochs on the full dataset.
In each iteration, we first learn the guide function g to select the data partition from which the knowledge needs to be distilled.
Next, given the function g , we learn the student using the help g .
We empirically found that not optimizing the student model on budget loss gives more stable results.
Hence, we minimize the student model only on the distillation and cross-entropy losses.
Dual Parameter Update ( λ ) Intuition.
Although it is tempting to optimise the above via a dual ascent and primal descent scheme (wherein the dual parameter λ is increased by residual term in the constraint until constraint satisfaction), recent work (Sun & Sun, 2021) has proposed to decrease the λ in the non-convex regime.
Inspired by this, we update the dual parameter by a fixed schedule between [ λ min , λ max ] , and update the bounds λ min , λ max during training to enforce the constraint.
λ min encourages exploration and allows student model to distill knowledge from all points. On the other hand, λ max enforces the constraint and forces the student model to learn on uncensored inputs. We choose R ≈ 4 λ T , so that the algorithm is exposed to a few exploratory periods.
For the final period, we increase λ monotonically so that budget is more strictly enforced at termination. Computational Efficiency.
Algorithm 1 trains both student and guide networks.
The guide network being small (three-layer MLP) relative to the student (CNN model), the additional cost in training the guide is relatively insignificant, and as such DiSK efficiency is similar to vanilla KD.
E XPERIMENTS
We evaluate DiSK under various capacity mismatch scenarios on benchmark vision datasets.
Datasets.
We use publicly available CIFAR-100 (Krizhevsky, 2009), Tiny-Imagenet (Le & Yang, 2015) datasets.
CIFAR-100 contains 50 K training and 10 K test images from 100 classes with size 32 × 32 × 3 .
While Tiny-Imagenet contains 100 K training and 10 K test images from 200 classes with size 64 × 64 × 3 .
We provide the dataset setup and data augmentations used in detail in Appendix A.3.
Models.
We evaluate standard convolutional models on these datasets including ResNet(He et al., 2016), Wide-ResNet(Zagoruyko & Komodakis, 2016), MobileNet(Sandler et al., 2018), and Shuf- [Table] fleNet(Ma et al., 2018).
Table 2 shows the storage and computational requirements of all the models used in this work.
We provide explicit model configurations in Appendix A.4, including the tiny models we generate from the Resnet architectures [Table]
Methods.
We study performance against standard cross-entropy (CE) based learning and the vanilla
KD methods.
For each method, we train models for 200 epochs using SGD as the optimizer with 0 .momentum and 0 .
1 learning rate.
See Appendix A.5, for training details.
We have recorded the mean in our results as the variance of 3 trials in our experiments is not larger than 0 .
1 in most of the cases.
We perform evaluations in different settings.
Below, we explain individual setups.
Large Capacity Mismatch Setting.
We distill knowledge from a teacher model into a student model where the student has much less capacity compared to teacher model.
We use four large capacity Resnet teachers and five tiny Resnet students and train these students using CE, KD, and DiSK methods.
Performances, and the gains of DiSK are reported in Table 3.
Small Capacity Mismatch Setting.
While DiSK has been designed for the scenario when student capacity is very low, we further evaluate it in the setting where teacher and student capacities are similar, to probe how far the power of the method extends.
The model classes used are the standard choice for this scenario (Chen et al., 2022; Tung & Mori, 2019).
Performance is reported in Table 4. Table 4 further reports the results of the feature matching distillation methods: FitNets (Romero et al., 2015), SemCKD (Chen et al., 2021), and SimKD (Chen et al., 2022). Such methods can often outperform response matching KD on large students, due to student representations that are more aligned with the teacher, but typically at an increased training cost. While feature matching methods are not the main focus of our work (and in principle scaffolding idea can be extended to them), we observe that DiSK often improves upon their performance without any direct feature matching.
Experiment results .
Below, we highlight salient features of DiSK based on empirical data.
DiSK outperforms the baselines uniformly across all datasets and student sizes.
As shown by Table 3,
DiSK significantly improves the student performance in CIFAR-100 and the (more challenging) Tiny-Imagenet dataset, respectively showing accuracy gains of up to 5% and 2% compared to KD.
These gains are consistent across a wide range of student and teacher capacities.
DiSK achieves better performance with worse teachers than KD does with even the best teachers.
In Table 3, we point out that the student performance increases for KD as the teacher complexity is increased for a given student.
But note that for the same student, DiSK achieves much better performance with even worse teacher.
For instance, for the ‘Resnet10-m‘ student, KD accuracy increases from 66 .
96% to 68 .
09% by using high capacity teachers.
But ‘Resnet10-m‘ trained with even the worst teacher ( ‘Resnet10- ℓ ‘ ) achieves 70 .
03% accuracy.
This saves a lot of resources in any application as large teacher requires more training time, and larger compute resources.
DiSK is competitive even in small capacity difference setting.
As shown by Table 4, DiSK does not loose its competitive edge over the KD even when the student is relatively similar sized as teachers, and shows gains of up to 2 .
5% relative to KD.
We conjecture that the observed gains arise from the fact that DiSK provides scaffolding for hard points to the student in initial training stages, which promotes the student to learn easy examples first.
As training progresses, DiSK removes the discounted help from hard inputs.
As a result, the student evolves from simpler hypothesis to the ones [Table] [Table]
[Table] consistent with both easy and hard inputs. This justifies our dual parameter ( λ ) update in Algorithm 1, wherein we periodically increase and decrease λ to enforce and relax the budget constraint.
DiSK students achieve near teacher accuracy while saving up to 8 × MACs & 5 × Params.
As reported in Table 3, student (‘Resnet10-m‘) trained with the teacher (‘Resnet10- ℓ ‘) achieves close to the teacher accuracy of 71 .
In this process, it saves 4 × compute and requires 4 × less parameters.
Similarly, student (‘ShuffleNetV2‘) trained with the teacher (‘Resnet32x4‘) achieves close to the teacher accuracy of 81 .
In this process, it saves 24 × compute and requires 5 × less parameters.
DiSK cleverly selects a subset of datapoints and smoothens the loss landscape.
As illustrated in Figure 1 and 2, DiSK judiciously selects a subset of hard-to-learn data points for the students and provide discounted help to the student focus on easily learnable inputs.
As a result, it eliminates some bad local minima in the student loss-landscape, and smoothens tihs surface.
DiSK enables the student to reach saturation capacity.
In Table 3, the performance of KD often suffers as the teacher size is increased, e.g., student (‘Resnet10-s’) accuracy decreases substantially with the teacher ‘Resnet34’ versus the teacher ‘Resnet18’.
In contrast, DiSK saturates the student performance across different teachers.
For instance, student (‘Resnet10-m‘) accuracy is ≈ 70% for all the teachers.
Thus, we point out that DiSK enables the student to reach saturation.
This may be due to the fact that guide g identifies the same set of ‘easy’ points across different teachers.
R ELATED W ORK
We refer the reader to Gou et al.
(2021) for a comprehensive survey on knowledge distillation.
Response Matching.
Zeng & Martinez (2000); Bucila et al.
(2006) distill the response of an ensemble of classifiers into a single neural network by creating a pseudo-labeled dataset using the ensemble of classifiers.
Ba & Caruana (2014) extend this to the setting with the neural network as the teacher.
Hinton et al.
(2015) propose vanilla KD that distilled knowledge from an ensemble of neural networks into a single network by matching their output logits.
This work provided a simple recipe for aligning the teacher and student predictive distributions using the Kullback-Leibler (KL) divergence.
Recently,
Beyer et al.
(2021) modify the KD procedure to include patient and consistent teacher resulting in substantial gains.
Knowledge consistency is enforced by using the same aggressive data augmentation and image views in the student as in the teacher.
Patience is promoted using a very long training schedule.
This results in a computationally very expensive training process.
Stanton et al.
(2021) analyze response matching KD and suggests that difficulty in optimization leads to poor knowledge distillation.
Thus, the teacher and student predictions do not always match, even on the training data.
Cho & Hariharan (2019) study Vanilla KD through the lens of mismatched student and teacher capacities.
It shows that small students are unable to mimic complex teachers.
They proposed early stopping teacher training as to remedy to achieve a student-learnable teacher.
The above works fail miserably when the gap between student and teacher complexities is large.
Specifically, the student cannot learn the complex teacher decision boundaries primarily due to the small student capacity.
It becomes imperative to selectively choose only easy-to-learn data points and transfer the teacher knowledge from these points and ignore the hard-to-learn data points during distillation.
Thus, our proposal targets the problem of severe capacity gaps between student and teacher models.
Additionally, our experiments with standard student and teacher configurations show that DiSK is still competitive when the capacity difference is small.
Feature Matching.
In response matching, teacher supervision is limited to its logits.
We can enforce intermediate layer feature matching for refined teacher supervision.
FitNets (Romero et al., 2015) extend the KD by including the feature matching in the middle layers.
Zagoruyko & Komodakis (2017) used feature map attention as the teacher supervision.
Tung & Mori (2019) preserve pairwise similarity in feature maps amongst data points during the distillation.
Chen et al.
(2022) modify the student by projecting the student features onto the teacher feature space and by reusing the teacher classifier.
While our work focuses primarily on selective distillation in Vanilla KD for simplicity.
We can easily extend the proposed framework to incorporate it into feature-matching distillation.
Privileged Information.
Vapnik & Izmailov (2015) propose the ‘learning under privileged information‘ (LUPI) framework wherein a support vector machine is trained using privileged information unavailable during the inference stage.
Later, Lopez-Paz et al.
unified LUPI and Vanilla KD into generalized distillation, wherein the teacher is learned using the privileged information.
Next, the student is trained using the ground truth and the teacher labels.
These works rely on privileged information in the application domain and are shown to work on toy setups.
Since our guide function, g is available only during training, it can be thought of as privileged information from a teacher.
Curriculum Learning & Hard Instance Mining.
Curriculum Learning (CL) (Bengio et al., 2009;
Hacohen & Weinshall, 2019; Graves et al., 2017) sorts the data based on their hardness as measured by some scoring function ( ex., predictive entropy, softmax margin score, etc. ).
It presents the data points during training in the order of increasing hardness.
Similarly, Hard Instance Mining (HIM) (Zhou et al., 2020) reduces the weight of the easy example and increases the weight on hard inputs to promote hard-example learning.
We point out that our method is only conceptually related to these works via input hardness.
Our method helps the student with hard examples by providing explicitly discounted help g .
We learn the helper function g (through teacher representation) that decides whether the student needs help on a given input.
Thus, we do not prioritize learning hard examples keeping in mind the fact that student capacity is much smaller than the teacher.
C ONCLUSION
We develop a new knowledge distillation method that utilizes teacher predictions in novel ways combining predictive targets with scaffolding the student on hard-to-learn points by means of a guide function.
Our method is particularly relevant when there is large gap between student and teacher capacities.
We show that our method allows for convergence to better minima based on two key properties.
Our guide function allows for censoring hard-to-learn examples, and the predictive targets set by the teacher on remaining points allow for eliminating bad local minima and smoothening the resulting student loss landscape.
Against vanilla KD we achieve improved performance and are comparable to more intrusive techniques that leverage feature matching on benchmark datasets.