Knowledge Distillation from A Stronger Teacher
Anonymous Author(s)
AffiliationAddress email
Abstract
Unlike existing knowledge distillation methods focus on the baseline settings,where the teacher models and training strategies are not that strong and competingas state-of-the-art approaches, this paper presents a method dubbed DIST to distillbetter from a stronger teacher.
We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer.
As aresult, the exact match of predictions in KL divergence would disturb the trainingand make existing methods perform poorly.
In this paper, we show that simplypreserving the relations between the predictions of teacher and student wouldsuffice, and propose a correlation-based loss to capture the intrinsic inter-classrelations from the teacher explicitly.
Besides, considering that different instanceshave different semantic similarities to each class, we also extend this relationalmatch to the intra-class level.
Our method is simple yet practical, and extensiveexperiments demonstrate that it adapts well to various architectures, model sizesand training strategies, and can achieve state-of-the-art performance consistentlyon image classification, object detection, and semantic segmentation tasks.
Introduction
The advent of automatic feature engineering fuels deep neural networks to achieve remarkablesuccess in a plethora of computer vision tasks, such as image classification [17, 19, 38, 49, 54], objectdetection [2, 23] and semantic segmentation [5, 55].
In the path of pursuing better performance,current deep learning models generally grow deeper and wider [13, 45].
However, such heavy modelsare clumsy to deploy in practice due to the limitations of computational and memory resources.
For an efficient model with competitive performance to those larger models, knowledge distillation(KD) [16] has been proposed to boost the performance of the efficient model (student) by distillingthe knowledge of a larger model (teacher) during training.
The essence of knowledge distillation relies on how to formulate and transfer the knowledge fromteacher to student.
The most intuitive yet effective approach is to match the probabilistic prediction(response) scores between the teacher and student via Kullback–Leibler (KL) divergence [16].
In thisway, the student can be guided with more informative signals during training, and is thus expected tohave more promising performance than that being trained stand-alone.
Besides this vanilla predictionmatch, other works [11, 14, 34, 41] also investigate the knowledge within intermediate representationsto further boost the distillation performance, but this usually induces additional training cost as aconsequence.
For example, OFD [14] proposes to distill the information via multiple intermediatelayers, but requires additional convolutions for feature alignments; CRD [41] introduces a contrastiveloss to transfer pair-wise relationships, but it needs to hold a memory bank for all 128-d features of
ImageNet images, and produces additional 260M FLOPs of computation cost.
[Figure]
Recently, a few studies [8, 29, 39] have been performed to address the poor learning issue ofthe student network when the student and teacher model sizes significantly differ.
For example, TAKD [29] proposes to reduce the discrepancy of teacher and student by resorting to an additionalteaching assistant of moderate model size; DGKD [39] further improves TAKD by densely gatheringall the assistant models to guide the student.
However, increasing the model size is only one ofthe popular approaches to have a stronger teacher.
There lacks a thorough analysis on the trainingstrategies to derive a stronger teacher and their effect on KD.
Most importantly, a generic enoughsolution is preferred to address the difficulty of KD brought by stronger teachers, rather than strugglingto deal with different types of stronger teachers (with larger model size or stronger training strategy)individually.
To understand what makes a stronger teacher and their effect on KD, we systematically study theprevalent strategies for designing and training deep neural networks, and show that:
• Beyond scaling up the model size, a stronger teacher can also be derived through advanced
training strategies, e
g. , label smoothing and data augmentation [52].
However, given a stronger teacher, the student’s performance on the vanilla KD could be dropped, even worse than training from scratch without KD, as shown in Figure 1.
• The discrepancy between teacher and student tends to get fairly larger when we switch
their training strategy to a stronger one (see Figure 2).
In this case, an exact recovery of predictions via KL divergence could be challenging and lead to the failure of vanilla KD.
• Preserving the relation of predictions between teacher and student is sufficient and effective.
When transferring the knowledge from teacher to student, what we really care about ispreserving the preference (relative ranks of predictions) by the teacher, instead of recoveringthe absolute values accurately.
Correlation between teacher and student predictions could be
favored to relax the exact match of KL divergence and distill the intrinsic relations.
In this paper, we thus leverage the Pearson correlation coefficient [33] as a new match manner toreplace the KL divergence.
In addition, besides the inter-class relations in prediction vector (see
Figure 3), with the intuition that different instances have different spectrum of similarities with respectto each class, we also propose to distill the intra-class relations for further boosting the performanceas Figure 3.
Concretely, for each class, we gather its corresponding predicted probabilities of allinstances in a batch, then transfer this relation from teacher to student.
Our proposed method (dubbed
DIST) is super simple, efficient, and practical, which can be implemented with only several lines ofcode (see Appendix A.1) and has almost the same training cost as the vanilla KD.
As a result, thestudent can be liberated from the burden of matching the exact output of a strong teacher, but only beguided appropriately to distill those truly informative relations.
K L D i v e
r
g
e n c e ( τ = 1 ) 0.
[Figure]
[Figure]
[Figure] [Figure]
[Figure]
t r o n g
[Figure]
τ
[Figure]
Extensive experiments are conducted on benchmark datasets to verify our effectiveness on varioustasks, including image classification, object detection, and semantic segmentation.
Experimentalresults show that our DIST significantly outperforms vanilla KD and those sophisticatedly-designedstate-of-the-art KD methods.
For example, with the same baseline settings on ImageNet, our DISTachieves the highest 72.07% accuracy on ResNet-18.
With the stronger strategy, our method obtains82.3% accuracy on the recent transformer Swin-T [27], improving KD by 1%.
Revisiting Prediction Match of KD
In vanilla knowledge distillation [16], the knowledge is transferred from a pre-trained teacher modelto a student model by minimizing the discrepancy between the prediction scores of the teacher andstudent models.
Formally, with the logits Z (s) ∈ R B × C and Z (t) ∈
R B × C of student and teacher networks, where
B and C denote batch size and the number of classes, respectively, the vanilla KD loss [16] isrepresented as
[Equation]
where KL refers to Kullback–Leibler divergence with
[Equation]
being the probabilistic prediction vectors, and τ is the temperature factor to control the softness oflogits.
In addition to the teacher’s soft targets in Eq.(1), KD [16] stated that it is beneficial to train thestudent together with ground-truth labels, and the overall training loss is composed of the originalclassification loss L cls and KD loss L KD , i.e. ,
[Equation]
where L cls is usually the cross-entropy loss between the predictions of student network and groundtruth labels, α and β are factors for balancing the losses.
2.1 Catastrophic discrepancy with a stronger teacher
As illustrated in Section 1, the effect of a teacher on KD has not been sufficiently investigated,especially when the performance of pre-trained teacher grows stronger, such as with larger model sizeor being trained with more advanced and competing strategies, e.g. , label smoothing, mix-up [52], tttssstttsss Conventional KD Instance Relation Inter-class Relation
Intra-class Relation classes i n s t a n c e s s 2 tcorr. ttt 3 sss 3 ts… corr. Input Teacher
Student
Output ttt 3 sss
Existing methods
Our DIST corr.
corr.
corr. corr. corr. corr.
corr. corr.
[Figure]
auto augmentations [9], etc . With this regard, as Figure 2, we train ResNet-18 and ResNet-50standalone with strategy B1 and strategy B2 1 , and obtain 4 trained models ( R18B1 , R18B2 , R50B1 ,and R50B2 with accuracies 69.76%, 73.4%, 76.13%, and 78.5%, respectively), then compare theirdiscrepancy using KL divergence ( τ = 1 and τ = 4 ) on the predicted probabilities Y . We have thefollowing observations: • The outputs of ResNet-18 do not change much with the stronger strategy compared toResNet-50.
This implies that the representational capacity limits the student’s performance,and it tends to be fairly challenging for the student to exactly match the teacher’s outputs astheir discrepancy becomes larger. • When the teacher and student models are trained with a stronger strategy, the discrepancy between teacher and student would be larger.
This indicates that when we adopt KD with a stronger training strategy, the misalignment between KD loss and classification loss would be severer, thus disturbing the student’s training.
As a result, the exact match ( i.e. , the loss reaches the minimal if and only if the teacher and studentoutputs are exactly identical) with KL divergence seems way too overambitious and demanding sincethe discrepancy between student and teacher can be considerably huge.
Since the exact match can bedetrimental with a stronger teacher, our intuition is to develop a relaxed manner for matching thepredictions between the teacher and student.
DIST: Distillation from A Stronger Teacher
3.1 Relaxed match with relations
The prediction scores indicate the teacher’s confidence (or preference) over all classes.
For a relaxedmatch of predictions between the teacher and student, we are motivated to consider what we reallycare about for the teacher’s output.
Instead of the exact probabilistic values, actually, during inference,we are only concerned about their relations , i.e. , relative ranks of predictions of teacher.
In this way, for some metric d ( · , · ) with R C × R C → R + , the exact match can be formulated thatd ( a , b ) =
if a = b for any two prediction vector as Y (s) i, : and Y (t) i, : in the KL divergence of Eq.(1).
Then as a relaxed match, we can introduce additional mappings ϕ ( · ) and ψ ( · ) with R C
→ R C suchthat
[Equation]
Therefore, d ( a , b ) = 0 does not necessarily require a and b should be exactly the same.
Nevertheless,since we care about the relation within a or b , the mappings ϕ and ψ should be isotone and do notaffect the semantic information and inference result of the prediction vector.
With this regard, a simple yet effective choice for the isotone mapping is the positive linear transformation, namely,
[Equation]
where m 1 , m 2 , n 1 , and n 2 are constants with m 1 × m 2 > 0 .
As a result, this match could be invariantunder separate changes in scale and shift for the predictions.
Actually, to satisfy the property Eq.(5),we can thus adopt the widely-used Pearson’s distance as the metric, i.e. ,
[Equation]
ρ p ( u , v ) is the Pearson correlation coefficient between two random variables u and v ,
[Equation]
[Equation]
where Cov ( u , v ) is the covariance of u and v , ¯ u and Std ( u ) denote the mean and standard derivationof u , respectively.
In this way, we can define the relation as correlation .
More specifically, and the original exact matchin vanilla KD [16] can thus be relaxed and replaced by maximizing the linear correlation to preservethe relation of teacher and student on the probabilistic distribution of each instance, which we callinter-class relation .
Formally, for each pair of prediction vector Y (s) i, : and Y (t) i, : , the inter-relation losscan be formulated as
[Equation]
Some isotone mappings or metrics can also be used to relax the match as Eq.(4), such as cosinesimilarity investigated empirically in Section 4.5; other more advanced and delicate choices could beleft as future work.
3.2 Better distillation with intra-relations
Besides the inter-class relation, where we transfer the relation of multiple classes in each instance,the prediction scores of multiple instances in each class are also informative and useful.
This scoresindicate the similarities of multiple instances to one class.
For instance, suppose we have threeimages containing “cat”, “dog”, and “plane”, respectively, and they have three prediction scores onthe ‘cat’ class, denoted as e , f , and g .
Generally, the picture “cat” should have the largest score to the“cat” class, while the “plane” should have the smallest score since it is inanimate. This relation of“ e > f > g ” could also be transferred to the student.
Besides, even for the images from the sameclass, the intrinsic intra-class variance of the semantic similarities is actually also informative.
Itindicates the prior from the teacher that which one is more reliable to cast in this class .
Therefore, we also encourage to distill this intra-relation for better performance.
Actually, defineprediction matrix Y (s) and Y (t) with each row as Y (s) i, : and Y (t) i, : , then the above inter-relation is tomaximize the correlation row-wisely (see Figure 3).
In contrast, for intra-relation, the correspondingloss is thus to maximize the correlation column-wisely, i.e. ,
[Equation]
[Table] [Table] As a result, the overall training loss L tr can be composed of the classification loss, inter-class KDloss, and intra-class KD loss, i.e. ,
[Equation]
where α , β , and γ are factors for balancing the losses.
In this way, via the relation loss, we haveendowed the student with freedom more or less to match the teacher network’s output adaptively,thus boosting the distillation performance to a great extent.
Experiments
4.1 Experimental settings
Training strategies.
The training strategies of image classification task are summarized in Table 1.
CIFAR-100.
For fair comparisons, we use the same training strategies (referred to A1 in Table
1)and pretrained models following CRD [41].
ImageNet.
B1: for comparisons with previous KDmethods, we train our baselines with the same simple training strategy as CRD [41].
B2: to validatethe effectiveness of KD methods on modern training strategies, we follow EfficientNet [40] anddesign a training strategy B2, which can significantly improve the performance compared to B1.
B3:the strategy B3 is used for training Swin-Transformers [27], and contains even more stronger dataaugmentations and regularization.
Loss weights.
On CIFAR-100 and ImageNet, we set α = 1 , β = 2 , and γ = 2 in Eq.(10).
On objectdetection and semantic segmentation, these three factors are all equal to 1 .
For KD [16], we setα = 0 .
9 , β = 1 in Eq.(3), and use a default temperature τ = 4 .
4.2 Image Classification
Baseline results on ImageNet.
We first compare our method with prior works using the baselinesettings.
As shown in Table 2, our DIST significantly outperforms prior KD methods.
Note thatour method is only conducted on the outputs of models, and has a similar computational cost as
KD [16].
Nevertheless, it even achieves better performance compared to those sophisticatedlydesigned methods.
For example, CRD [41] needs to preserve a memory bank for all 128-d featuresof ImageNet images, and produces additional 260M FLOPs of computation cost; SRRL [48] and
Review [7] require additional convolutions for feature alignments.
The implementation of DIST canbe found in Appendix A.1, which is quite simple compared to these methods.
Distillation from stronger teacher models.
As the stronger teachers come from larger model sizesand stronger strategies, we here first conduct experiments to compare our DIST with the vanilla KDon different scales (model sizes) of ResNets with baseline strategy B1.
As shown in Table 3, whenthe teacher goes larger, the ResNet-18 students perform even worse than that with a medium-sized
ResNet-50 teacher.
Nevertheless, our DIST shows an upward trend with larger teachers, and theimprovements compared to KD also become more significant, indicating that our DIST tackles betteron the large discrepancy between the student and larger teacher.
[Table]
Distillation from stronger training strategies.
Recently, the performance of models on ImageNethas been significantly improved by the sophisticated training strategies and strong data augmentations( e.g. , TIMM [44] achieves 80.4% accuracy on ResNet-50 while the baseline strategy B1 only obtains76.1%).
However, most of the KD methods still conduct experiments with simple training settings.
Itis seldomly investigated whether the KD methods are suitable to the advanced strategies.
In this way,we conduct experiments with advanced training strategies and compare our method with vanilla KD,instance relation-based RKD [30], and SRRL [48].
We first train traditional CNNs with strong strategies, and also use a strong ResNet-50 with 80.1%accuracy trained by [44] as the teacher.
As results shown in Table 4, on both similar architectures(ResNet-18, ResNet-34) and dissimilar architectures (MobileNetV2, EfficientNet-B0), our DISTcan achieve the best performance.
Note that RKD and SRRL can perform worse than training fromscratch, especially when the students are small (ResNet-18 and MobileNet) or the architectures ofteacher and student are fairly different (ResNet-50 and Swin-L), this might be because they focus onthe intermediate features, which can be more challenging for the student to recover teacher’s featurescompared to predictions.
Furthermore, we experiment on the recent state-of-the-art Swin-Transformer [27].
The results showthat our DIST gains improvements on even more stronger models and strategies.
For example, with
Swin-L teacher, our method improves ResNet-50 and Swin-T by 1.7% and 1.0%, respectively.
CIFAR-100.
The results on CIFAR-100 dataset in Table 5 show that, by distilling on the predictedlogits, our method even outperforms those sophisticatedly-designed feature distillation methods.
4.3 Object Detection
We further investigate the effectiveness of DIST on downstream tasks.
We conduct experiments on
MS COCO object detection dataset [25], and simply leverage our DIST as an additional supervisionon the final predictions of classes.
Following [37, 53], we use the same standard training strategiesand utilize Cascade Mask R-CNN [2] with ResNeXt-101 backbone as the teacher for two-stagestudent of Faster R-CNN [23] with ResNet-50 backbone; while for one-stage RetinaNet [24] with
ResNet-50 backbone, the RetinaNet with ResNeXt-101 backbone is utilized as the teacher.
As shown in Table 6, our DIST achieves competitive results on COCO validation set.
For comparisons,we train the vanilla KD under the same settings as our DIST, the results show that our DISTsignificantly outperforms vanilla KD by simply replacing the loss functions.
Moreover, by combining
[Table]
[Table] DIST with mimic, which minimizes the mean square error between FPN features of teacher andstudent [Table] we can even outperform the state-of-the-art KD methods designed for object detection.
4.4 Semantic Segmentation
We also perform experiments on semantic segmentation, a challenging dense prediction task.
Following [37, 43, 47], we train DeepLabV3 [6] and PSPNet [55] with ResNet-18 backbone on Cityscapesdataset, and adopt our DIST on the predictions of classification head using a teacher with ResNet101 backbone of DeepLabV3.
As the results summarized in Table 7, with only the supervision ofclass predictions, our DIST can significantly outperform existing knowledge distillation methodson semantic segmentation task.
For example, our DIST outperforms recent state-of-the-art method
CIRKD [47] by 1.58% on PSPNet-R18.
This demonstrate our effectiveness on relation modeling.
4.5 Ablation studies
Effects of inter-class and intra-class correlations.
This paper proposes two types of relations: interclass and intra-class relations.
To validate the effectiveness of each relation, we conduct experimentsto train students with these relations separately.
The results on Table 8 verify that, both inter-class andintra-class relations can outperform the vanilla KD; also, the performance could be further boostedby combining them together. [Table] [Table]
[Table]
Effect of intra-class relation in vanilla KD.
To investigate the effectiveness of intra-class relationin vanilla KD, we adopt experiments to train our DIST using KL divergence as the relation metric,denoted as DIST (KL div.)
As the results summarized in Table 8, adding intra-class relation in thevanilla KD can also improve the performance (from 71.21% to 71.62%).
However, when the studentis trained with intra-class relation only, the improvement of using KL divergence is less significantthan using Pearson correlation (70.61% vs. 71.55%), since the means and variances of intra-classdistributions could be varied.
More ablation studies can be found in Section A.3.
Conclusion
This paper presents a new knowledge distillation (KD) method named DIST to implement betterdistillation from a stronger teacher.
We empirically study the catastrophic discrepancy problembetween the student and a stronger teacher, and propose a relation-based loss to relax the exact matchof KL divergence in a linear sense.
Our method DIST is simple yet effective in handling strongteachers.
Extensive experiments show our superiority in various benchmark tasks.
For example,
DIST even outperforms state-of
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Training details
are provided in the paper.
Core code of the method and training logs are included in
supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Standard deviations on CIFAR-100 are reported.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code and training logs are included.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/ A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
A AppendixA.1 Implementation of DIST
This section presents the implementation code of DIST, as shown in Figure 4.
With only the outputlogits of student and teacher models, our DIST leverages quite simple inputs and computations, andis easy to implement.
[Table]
Distillation from A Stronger Teacher A. References of Compared KD Methods
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
B. Implementation of DIST
This section presents the implementation code of DIST, as shown in Figure 5.
With only the output logits of student and teacher models, our DIST leverages quite simple inputs and computations and is easy to implement.
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
(a
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
y_
[Figure]
[Figure]
[Figure]
(y_
[Figure]
y_t
[Figure]
[Figure]
[Figure]
_class_
[Figure]
(y_s, y_t):
[Figure]
[Figure]
[Figure]
(y_s.
[Figure]
[Figure]
[Figure]
y_t.
[Figure]
[Figure]
(nn
[Figure]
A.2 Related work
There exist some KD methods [4, 30, 34] which transfer the relations between instances from teacherto student (see Figure 3 for illustration).
The purpose of these methods is to learn the similarityrelationships between instances from teacher, e.g. , the semantic spaces of instances with the sameclass should be close, while the instances of different classes should have larger distances.
To distillthis instance relation, CCKD [34] proposes to measure the correlation between each instance pairwisely, then minimizes the difference of correlation scores between teacher and student; RKD [30]formulates the instance relation via the distance and angle between each pair; 4 introduces a localitypreserving loss to embed the teacher’s instance relation into student.
Nevertheless, in this paper, we focus on the probabilistic prediction instead of intermediate features,and leverage the relation on a new perspective to relax and replace the KL divergence in almost all
KD methods, which help us to achieve better performance especially when the student is trainedwith a stronger teacher.
Besides, different from previous works which measure relations betweeninstances of teacher and student individually, we propose inter-class relation and intra-class relationsimultaneously to better distill from a teacher.
A.3 More ablation studies
Correlations between teacher and student.
To validate the effectiveness of our correlation-basedloss, we measure the correlations between teacher and student models, where the student modelsare trained by plain classification loss, KD, and our DIST.
We choose commonly used Pearsoncorrelation coefficient, Spearman’s [10] and Kendall’s Tau [20] rank correlation coefficients as themetrics of correlation.
As summarized in Figure 5, our DIST obtains higher inter-class and intra-classcorrelations compared to baselines.
Effect of training students with KD loss only.
Training student with only the KD loss can betterreflect the distillation ability and the information richness of supervision signals.
{+As results in Table 
+}
[Figure]
show that, when the student is trained with only the KD loss, our DIST significantly outperforms thevanilla KD.
Without using the ground-truth labels, it can even outperform the standalone trainingaccuracy, which indicates the effectiveness of our DIST in distilling those truly-beneficial relations.
[Table]
Using cosine similarity in DIST.
In our method, the relation matching can be any function with thesame form of Eq.(4).
We simply adopt a commonly used Pearson correlation as our relation metric in
DIST.
Here we conduct experiments to investigate the efficacy of our method with cosine similarity.
[Table]
Both cosine similarity and Pearson correlation coefficient can evaluate the relations between teacherand student.
Compared to the scale invariance in cosine similarity, the Pearson correlation has anadditional shift-invariance by centering the vectors first (see Eq.(7)), and it could be more robust tothe distribution changes.
We conduct experiments to compare these two metrics in our DIST andtrain the models with or without label smoothing.
Since recent studies [3, 36] state that KD with hightemperatures is incompatible with label smoothing, we also train the models with KD ( τ = 1 ).
Asshown in Table 10, adopting DIST with Pearson correlation achieves higher accuracies comparedto KD and DIST with cosine similarity, especially when the teacher and student are trained withlabel smoothing (the predicted probabilistic distributions would be shifted by it).
As a result, thescale-and-shift-invariant Pearson correlation may be a better metric for measuring relations in DIST.
A.4 Comparisons of training speed
We compare the training speed of our DIST with vanilla KD [16], RKD [30], CRD [41], and
SRRL [48], as summarized in Table 11.
Our DIST has almost the same highest training speed as thevanilla KD, outperforming other feature-based KD methods.
[Table]
A.5 Landscapes of matching functions
As discussed in our main text, the matching functions such as KL divergence and MSE are usedto match the outputs between student and teacher in KD.
In DIST, we propose to relax the matchin KD with relational matching functions, i.e. , Pearson distance.
Here we visualize the landscapesof KL divergence, MSE, cosine distance, and Pearson distance to better show our effectiveness onrelaxing the constraints.
As shown in Figure 6 and Figure 7, we randomly generate a 2-dimensionalvector as the output logits of student and teacher, then we adjust the first element in the teacher andstudent logits (namely, Z ( s ) 1 and Z ( t ) 1 ) independently and fix the remaining elements, to draw the lossor gradient landscapes w.r.t.
Z ( s ) 1 and Z ( t ) 1 .
Specifically, when Z ( s ) 1 =
Z ( t ) 1 , the logits of student andteacher are identical, and the loss values of these functions should be zero.
When we change thevalues of logits, the curves shows that how the losses are changed by it, i.e. , the sensitivity of loss tothe distribution shifts.
The landscapes of gradient in Figure 7 show that, the PCC in our DIST has
-optimal regions and has the largest area on the region of optima.
Thisindicates that our DIST can get optimized quickly with wider ranges of Z ( t ) 1 and Z ( s ) 1 , thus it wouldhave smaller conflict to the task loss.
A.6 Discussion
Limitations.
DIST can improve multi-class classification consistently in various tasks such asclassification, object detection, and semantic segmentation.
However, it would be less-effective onbinary classification task, as the task only contains two classes and the information in inter-classrelation is limited.
Societal impacts.
Investigating the efficacy of the proposed method would consume considerablecomputing resources.
These efforts can contribute to increased carbon emissions, which could raiseenvironmental concerns.
However, the proposed knowledge distillation method can improve theperformance of light-weight compact models, where replacing the heavy models with light models inproduction could save more energy consumption, and it is necessary to validate the efficacy of DISTadequately.
[Figure]
[Figure]