Shadow Knowledge Distillation: Bridging Ofﬂine and Online Knowledge Transfer
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Knowledge distillation can be generally divided into ofﬂine and online categories according to whether teacher model is pre-trained and persistent during the distillationprocess.
Ofﬂine distillation can employ existing models yet always demonstratesinferior performance than online ones.
In this paper, we ﬁrst empirically showthat the essential factor for their performance gap lies in the reversed distillationfrom student to teacher, rather than the training fashion.
Ofﬂine distillation canachieve competitive performance gain by ﬁne-tuning pre-trained teacher to adaptstudent with such reversed distillation.
However, this ﬁne-tuning process stillcosts lots of training budgets.
To alleviate this dilemma, we propose SHAKE, asimple yet effective SHA dow K nowl E dge transfer framework to bridge ofﬂineand online distillation, which trades the accuracy with efﬁciency.
Speciﬁcally,we build an extra shadow head on the student’s backbone to mimic the predictions of pre-trained teacher as its shadow.
Then, this shadow head is leveragedas a proxy teacher to perform bidirectional distillation with student on the ﬂy.
In this way, SHAKE not only updates this student-aware proxy teacher with theknowledge of pre-trained model but also greatly optimizes costs of augmentedreversed distillation.
Extensive experiments on classiﬁcation and object detectiontasks demonstrate that our technique achieves state-of-the-art results with different
CNNs and Vision Transformer models.
Additionally, our method shows strongcompatibility with multi-teacher and augmentation strategies by gaining additionalperformance improvement.
Code will be made publicly.
Introduction
Deep Neural Networks (DNNs) have achieved great success in tackling a variety of tasks [19, 10].
Despite appealing performance, prevailing DNN models usually have large numbers of parameters, bringing heavy costs of computation.
To alleviate this problem, many network compressionmethods [21, 28] have been proposed among which Knowledge Distillation (KD)
[14] has recentlyattracted increased attention.
The aim of KD is to transfer the learnt knowledge of a high-capacity teacher model to a low-capacitystudent model.
Numerous ofﬂine methods [1, 14] use a two-stage training process that begins withtraining a teacher model and then keeping it ﬁxed to distill student model (see KD in Figure 1 (a)).
Besides ofﬂine fashion, recent online methods [20, 4] adopt a one-stage training process, jointlytraining the student and teacher/peer models using bidirectional distillation like DML [39] in Figure 1(b).
These online distillations always surpass the ofﬂine distillations under the same teacher model.
However, some large teacher models trained from scratch would bring some difﬁculties ( e.g. , highcomputational resources and unstable optimizations) for distillation, especially for tasks that relyon large transformers ( e.g. , BERT [36], GPT-3 [2] and ViT-MoE [30]).
Therefore, two questions
[Figure]
[Table]
naturally arise: (1) Why is there a performance gap between ofﬂine and online distillation?
(2) Howthe performance of ofﬂine distillation can be advanced with design techniques of online KD methods?
To clarify the ﬁrst question, we compare KD and DML regarding the training fashion of teacherand distillation loss in Table 1.
Contrary to the common belief, we empirically observe that trainingfashion may not affect the distillation performance since DML † obtains similar performance as KD(70.55% vs. 70.66%).
Instead, the reversed distillation from the student model yields signiﬁcantaccuracy gains for KD † than KD (71.76% vs. 70.66%) and DML than DML † (71.52% vs. 70.55%).
Then, we analyze the output discrepancy of the teacher-student for different methods, and the reverseddistillation reduces the gap from 1.12 of KD to 0.76 of KD † .
Therefore, the main reasons behindthe performance gap lie in two aspects: (a) Teacher models in conventional ofﬂine pipelines arenot optimized for the student model.
Thus, they could only provide general knowledge, which maybe suboptimal for the particular student.
(b) The reversed distillation changes the universal teachermodel into a student-aware one and bridges the teacher-student gap, which clearly justiﬁes:
Teachers should teach students in accordance with their aptitude and shouldn’t follow the samepattern.
For the second question, ﬁne-tuning pre-trained teacher model with reversed distillation is a straightforward way to bridge the two training fashions.
However, in most scenarios, ﬁne-tuning the wholenetwork still needs lots of training budges (more 4 × costs than KD in Table 1).
Some trade-offsin reducing the ﬁne-tuning parameters or epochs involve performance loss.
Thus, how to augmentreversed distillation without much extra overhead is an important issue for the application.
Besidesﬁne-tuning, building a proxy teacher model (see Figure 2 (a)) to inherit the knowledge of pre-trainedmodels and receive reversed distillation from students also enjoys the same beneﬁts.
As shownin Table 1 (H), this proxy teacher model (SHAKE (alone)) can achieve competitive gains thanﬁne-tuning the whole teacher model (KD † ).
Recent weight sharing strategies in AutoML and caneffectively save training overhead and accelerate the convergence of the model.
This encourages us toadopt the same student architecture and weight sharing strategy to generate the proxy teacher model.
Thus, we allow this proxy teacher to share the student’s backbone but use an individual shadow headto preserve the diversity of logits representations (see Figure 2 (b)).
As shown in Table 1 (I),
such T
[Figure]
cat T
[Figure]
T
T
Tcat
[Figure]
Student
Shadow head
Task head
KD
CE
Share
[Figure] [Figure]
T' T
S S
S cat
[Figure]
sharing strategy in SHAKE (share) presents three beneﬁts: (1) more than 3 × training accelerationthan DML and ﬁne-tuning whole teacher model.
(2) No need for architecture selection costs for proxyteacher.
(3) Additional accuracy gains than individual proxy teacher because knowledge inherited byproxy teachers also directly improves the representation of students’ backbone in the weight-sharingprocess.
Based on the above observations, we propose SHA dow K nowl E dge distillation (SHAKE), a noveland effective logits distillation framework.
Our SHAKE builds a proxy teacher model and updates itsweights via the original teacher model predictions.
In this way, SHAKE enjoys the same beneﬁts withteacher ﬁne-tuning to mine the knowledge of the pre-trained model and can perform bidirectionalsupervision with student.
To optimize training costs, this proxy teacher shares the backbone with thestudents but leverages an individual head to preserve the diversity of logits representations.
This headis named shadow head since it imitates the original teacher model just like its shadow.
During training,we only need to additionally train this shadow head with small training budgets (only 1.18 × coststhan KD).
After training, all teacher models and shadow head can be deprecated, and student modelcan be separately applied for inference without any overhead.
Moreover, we extend the SHAKE tomulti-teacher model scenarios using multiple shadow heads to inherit diverse knowledge.
In principle, SHAKE alters the chain of knowledge transfer from pre-trained teacher → student in
KD to pre-trained teacher → proxy teacher (cid:29) student .
Other adaptive KD [25] only employs middlenetworks to distill as large teacher → middle teacher → student , which is not optimized for studentsand requires multi-step training.
The merits of SHAKE lie in three-fold.
First, it effectively reducesthe teacher-student capability gap with reversed distillation, bringing signiﬁcant gains when pretrained teacher models are available.
Second, for the scenario without pre-trained teacher models, SHAKE also enables the ofﬂine KD methods more effective to alleviate the unstable optimizationissues of online KD methods.
Thus, SHAKE bridges ofﬂine and online KD methods and enjoys theadvantages of both methods.
Third, SHAKE achieves favorable trade-offs between accuracy andtraining budget.
By contrast, other adaptive KD [25] needs sequentially training multiple models withlots of additional training time and resources.
We hope that these intriguing observations in SHAKEwould expand the application of KD and facilitate future research for KD work to some extent.
We conduct extensive experiments on classiﬁcation and detection to verify the superiority of theproposed method.
SHAKE achieves a consistent and signiﬁcant accuracy boost in various neuralnetworks and data augmentations, which outperforms other methods by large margins.
For example,
SHAKE obtains 1 .
21% ∼ 3 .
65% accuracy gains and 3 × ∼ 4 × training acceleration than DML on CIFAR-100.
On the challenging ImageNet dataset, SHAKE can improve the Top-1 accuracy of
ResNet-18 from 69.75% to 72.37% and MobileNet from 70.13% to 72.96% , which are state-of-
the-art performances among KD methods.
On vision transformer architecture [34], our approachachieves 75.22% Top-1 accuracy and 2.76% gain for training a VIT-T model.
On the object detectiontask, SHAKE improves the AP by 1.20 for RetinaNet and 1.02 for Faster R-CNN on MS-COCOdataset, demonstrating the generality of our approach.
In summary, we make the following principle contributions in this paper:
• By analyzing and exploring the difference between ofﬂine and online KD methods, weempirically show that the reversed distillation hinders the performance gain, which ﬁxesthe discrepancy between teacher-student capability.
This motivates us to propose a newSHA dow K nowl E dge distillation (SHAKE) framework to bridge two training fashions,which, to the best of our knowledge, is not achieved in the area of knowledge
distillation.• SHAKE achieves remarkable trade-offs between accuracy and training efﬁciency withan extra shadow head.
The shadow head inherits knowledge from pre-trained models,introduces reversed distillation with students, and accelerates training process by sharingthe student’s backbone.•
We perform thorough evaluations on classiﬁcation and detection.
SHAKE achieves stateof-the-art performances in multiple datasets and architectures (e.g. CNN and vision transformer).
Speciﬁcally, ResNet-18, MobileNet, and VIT-T with SHAKE achieve 72.37%,72.96% and 75.22% Top-1 accuracy on ImageNet, outperforming KD by 1.71%, 2.28% and3.02%, respectively.
Shadow Knowledge Distillation
In this section, we ﬁrst revisit the ofﬂine and online KD methods.
Then, we present the formulationof SHAKE and its expansion for multi-teacher models.
The evolution of our approach is shown in
Figure 2. 2.1 Revisit of ofﬂine and online KD methods
We ﬁrst review the formulations of ofﬂine and online KD.
For simplicity, We choose two typicalframeworks ( i.e. , original KD [14] and DML [39]) for analysis.
Given training dataset ( X, Y ) where
X = { x i } ni =1 are training samples and Y =
{ y i } ni =1 are their labels.
Let f T be the output logits ofthe ﬁxed teacher T and let f S be the output of student S , respectively.
In KD, the student network f Sis trained by minimizing:
[Equation]
where λ is a weight for balancing these two terms.
L CE is the regular cross-entropy objective:
[Equation]
where H ( · , · ) is the cross-entropy loss and σ is the softmax function.
L KL in Eq.
1 is the distillationobjective for transferring knowledge from a teacher to a student:
[Equation]
where τ is a temperature to generate soft labels and KL represents Kullback-Leibler (KL) divergence.
This distillation loss can be considered as a modiﬁed cross-entropy loss using the probabilistic outputsof the pre-trained teacher network as the soft labels instead of the one-hot ground-truth labels.
Different from the one-way distillation in KD, DML presents a two-way knowledge transfer strategyin which the probabilistic outputs from both teacher and student networks can be used to guide thetraining of each other.
DML jointly trains the teacher and student networks in an end-to-end mannervia interleavingly optimizing two objectives:
[Equation]
where the default value of λ is 1 in DML.
2.2 Formulation of SHAKE
As shown in Figure 2 (c), SHAKE proposes a proxy teacher model with output f T (cid:48) via shadow headto augment reverse distillation in the ofﬂine framework.
During the training process, f T (cid:48) is updatedwith the output of the pre-trained teacher model f T and performs mutual distillation with studentmodels f S as:
[Equation]
where each of the loss items has the same form as KD in Eq. 1, and the effects of λ are explored inthe experiment. In contrast to DML, the online teacher f T (cid:48) is not optimized with ground-truth labelsto better maintain knowledge of pre-trained teacher f T and is not applied in inferences.
For multiple teacher models in Figure 2 (d), we build several proxy teachers with outputs ( f T (cid:48) 1 , f T (cid:48) 2 ,..., f Ti (cid:48) ) to follow outputs ( f T 1 , f T 2 ,..., f T i ) of multiple teacher models with variousshadow heads.
The mutual distillation also exists between the task head of the student and thesemultiple shadow heads.
Similar to Eq.
5, the total optimization function for the multi-teacher scenariocan be deﬁned as:
[Equation]
where N is the total number of teacher models.
In addition, SHAKE can be combined with advancedmulti-teacher methods ( e.g. , adaptive weights for different teachers in part IV of Table 2).
Comparison of SHAKE with other KD methods .
Eq.
Eq. 4 and Eq. 1 clearly illustrate the difference of SHAKEand KD/DML.
Compared to KD, SHAKE introduces reverseddistillation so that the teacher can be optimized by the student.
In contrast to DML, SHAKE leverages the knowledge ofpre-trained models, resulting in additional accuracy gains.
Asshown in Figure 3, SHAKE achieves a robust boost than KDand baseline during the training process.
Meanwhile, trainingcurve of DML shows highly dynamic oscillation due to unreliable predictions of its teacher trained from scratch.
Otheradaptive KD methods use the same knowledge transfer wayas KD and do not optimize teacher with feedback supervisionof student.
Thus, these methods including ATKD [25] and
ESKD [5] do not essentially address the teacher-student capability disparity and our SHAKE surpasses them with signiﬁcantmargins ( 1 . 48% ∼ 1 .
59% on ImageNet in Figure 3)..
Experiments
In this section, we ﬁrst evaluate our approach for the classiﬁcation task on CIFAR-100/
ImageNet andthe object detection task on MS-COCO.
Then, comprehensive ablation experiments are performed toanalyze the key design in our SHAKE.
As a novel logits ofﬂine approach, the main competitor of
SHAKE is the original KD [14].
Thus, we perform detailed experimental comparisons between them.
Moreover, we compare the performance with recent advanced KD methods.
For fair comparisons,we use the public codes of these approaches with the same training and data preprocessing settingsthroughout the experiments.
For SHAKE, we set λ and τ as 1 and 4, respectively.
Please refer to thesupplementary materials for more training settings.
3.1 Experiments on CIFAR-
Implementation .
On CIFAR-100 [18], we conduct experiments on various teacher-student modelsunder same or different architecture style with CRD’s settings [33], whose training epochs are 240. [Table] KD [14] 74. KD
[Table]
DML [39]
[Table] DML † 74. [Table] [Table]
[Table]
II
[Table]
[Table]
[Table]
[Table]
[Table]
CRD 77.
III
[Table]
[Table]
[Table]
SHAKE
[Table]
CutMix 78.
IV
KD
[Table]
[Table]
We use a mini-batch size of 64 and a standard SGD optimizer with a weight decay of 0.0005.
Themulti-step learning rate is initialized to 0.05, decayed by 0.1 at 150, 180, and 210 epochs.
Forthe comparison experiments with online KD methods, we adopt the same training settings with OKDDip [4], whose training epochs are 300.
Comparison with ofﬂine and online KD methods .
In the part I of Table 2, we compare ourapproach to some advanced ofﬂine/online KD methods with the same training settings.
For thesame architecture style of the teacher-students, SHAKE obtains 3 .
35% ∼ 5 .
45% absolute accuracygains and outperforms KD with 1 .
41% ∼ 4 .
62% margins and Review with 0 .
70% ∼ 2 .
49% margins.
Besides, on cross-architecture teacher-student pairs, SHAKE achieves more signiﬁcant gains with5 .
58% ∼ 6 .
96% margins than baseline.
Compared to other KD methods, SHAKE outperforms
KD with 2 .
83% ∼ 4 .
06% margins and Review with 0 .
64% ∼ 2 .
46% margins, which illustratesthe effectiveness of SHAKE in reducing the teacher-student network gap.
Compared to DML and
AFD [6] under the same teacher-student pair in Table 2, SHAKE obtains 1 .
21% ∼ 3 .
65% relativegains.
Compare to online KDs ( e.g. , CL [32], ONE [20]) with multiple branches as teacher models,
SHAKE also obtains 0 .
70% ∼ 1 .
77% relative accuracy gains.
These signiﬁcant improvementsdemonstrate the superiority of SHAKE by leveraging the knowledge of pre-trained models.
Orthogonal to other KDs and data augmentations .
As acting on the output logits, SHAKE isorthogonal to feature and relation KD methods because of transferring different knowledge.
Asshown in the part II of Table 2, The combination of FitNets with SHAKE surpasses FitNets + KD with0 .
98% ∼ 3 .
54% margins.
Moreover, for CRD, its combination with SHAKE yields more dramatic [Table] gains than KD.
In addition, SHAKE outperforms KD (see part III of Table 2) under the strong dataaugmentation.
Extension to multiple teacher models.
In the part IV of Table 2, we compare SHAKE and KD withaverage and adaptive weighting under two teacher networks.
The results illustrate that the design of
SHAKE with multi-proxy model can effectively inherit the knowledge of different teachers bringingsigniﬁcant performance gains.
3.2 Experiments on ImageNet
Implementation .
For standard ResNet-18 [12] and MobileNet [15] models, we adopt the sametraining setting with most distillation methods, whose training epochs are 100.
The multi-steplearning rate is initialized to 0.1, decayed by 0.1 at 30, 60, and 90 epochs.
Recent vision transformersachieve great success on different vision tasks [7, 34].
We also extend SHAKE to VIT-T [34] with thesame training settings ( e.g. , data augmentation and distillation token) on ImageNet.
Implementationdetails are available in supplementary materials.
Comparison results .
Table 3 reports the performance of our approach on ImageNet.
SHAKEimproves baseline models of ResNet-18 by 2 .
62% gains in Top-1 accuracy (see Figure 3 for detailedaccuracy curves) and MobileNet by 2 . 83% gains.
Compared to other KD methods, SHAKE outperforms KD with 1 .
71% ∼ 2 .
28% margins and CRD with 1 . 20% ∼ 1 .
56% margins, which supportsthe superiority of SHAKE on the large-scale dataset.
Equipped with the distillation of two teachermodels, SHAKE ‡ obtains 2.98% gain for ResNet-18 and 3.29% gain for MobileNet than baseline.
As shown in Table 3, SHAKE obtains 2.76% accuracy gains than baseline and surpasses KD underthe soft or hard label of the CNN teacher (RegNet [27]), verifying its effectiveness on differentarchitectures.
3.3 Experiments on object detection
Implementation .
We evaluate SHAKE on MS-COCO dataset [23] and use the most popular opensource Detectron2 [22] as the strong baseline.
We apply SHAKE to two-stage detector ( e.g. , Faster
R-CNN [29]) and one-stage detector ( e.g. , RetinaNet [22]), which are widely used object detectionframeworks.
Following the common practice [22], all models are trained with 2 × learning schedule(24 epochs).
For SHAKE, we build an extra shadow head with the same architecture as the originalclassiﬁcation head, which performs distillation in the detector ﬁne-tuning stage.
All distillationperformances are evaluated in Average Precision (AP).
Comparison results .
Table 4 demonstrates the experimental results of the baseline detector andour approach.
As shown in Table 4, our SHAKE improves the AP by 1.20 on RetinaNet and 1.02on Faster R-CNN, which outperform KD [14].
The performance of the object detection greatlydepends on the quality of deep features to locate interested objects, while logits are not capable ofproviding knowledge for object localization.
Thus, SHAKE naturally weaker than Review (recentstate-of-the-art feature-based KDs) and we further introduce Review to obtain satisfactory results.
Itcan be observed that SHAKE can obtain new state-of-the-art results with feature-based KDs.
Thesuccess of challenging object detection tasks demonstrates the generality and effectiveness of ourapproach.
[Table]
3.4 Ablation study
In this section, we isolate the impact of each component of our approach.
All experiments areconducted on the CIFAR-100 dataset.
For each setting, we run 3 times and report Top-1 meanaccuracies.
Detailed ablation study of SHAKE As shown in Table 5, an ablation study has been conductedto demonstrate the individual effectiveness of different components in SHAKE.
It is observed that(a) Reversed distillation optimizes the teacher model to adapt to the student, improving distillationefﬁciency.
SHAKE without reversed distillation only obtains marginal gains.
(b) Shadow headsare effective in preserving the diversity of knowledge, and SHAKE brings accuracy reduction ifthere is no separate shadow head.
(c) Stronger shadow heads for storing and teaching more pretrained knowledge when the capacity gap enlarges.
(d) Backbone sharing settings can accelerate theconvergence of the student model and results in performance gains.
(e) Similar to the self-distillationand regularization methods, SHAKE without pre-trained teacher model also yields slight accuracygains because of more training weights and logit consistency. (f) Our proxy teacher is present toinherit the pre-training knowledge with its supervision.
Under our backbone sharing settings, addingextra supervision from ground truth labels for proxy teacher not conducive to the decoupling ofsupervision from pre-trained teachers/labels and leads to accuracy reduction.
(h) Proxy teachers inour SHAKE to transfer the knowledge of the original pre-trained teacher model for the student modelto learn more easily.
As shown in the Table 5 (h), the extra distillation from pre-trained teachers tostudents based on SHAKE does not yield additional signiﬁcant performance gains.
[Table]
(f) w Ground truth supervision for T’
(h) w Pre-teacher distillation for S
Sensitivity study for temperature τ and weight λ .
τ in Eq.
3 controls the softness of teacher’ssupervision.
As τ gets higher, the output of the softmax function becomes smoother.
As shown in
Figure 4, SHAKE presents superior gains and robustness than the original KD on different τ .
λ
isthe hyper-parameter that balances the weight of KD loss.
The small λ limits distillation gains, andλ = 0 .
5 is the best option for SHAKE.
Analyzing teacher-student similarity.
The similarity between student and teacher network is animportant measure for KD tasks.
We employ KL-divergence as metrics of similarity [5], where lower
KL-divergence implies higher similarity.
Figure 5 presents the similarities and performances betweenthe outputs of ResNet with different depths as teachers ( i.e. , ResNet-110, ResNet-56, ResNet-44and ResNet-32) and the ResNet-20 as student.
The results show that the distillation from SHAKEalways gives higher similarity than KD and DML, resulting in signiﬁcant gains under teacher models [Figure] Teacher DML of different depths.
Figure 6 presents the attention map visualization of ResNet-18 with differentmethods on ImageNet.
The attention maps of SHAKE pay more attention to the important regionsthan KD and baseline, which have high similarity with teacher.
In summary, SHAKE bridges KD and DML to obtain student-friendly distillations, improving teacher-student similarity.
Related Work
Ofﬂine distillations .
In ofﬂine framework [14], the teacher is pre-trained and ﬁxed, and then its softlogits are used as the extra supervision to distill student.
Although the subsequent methods exploredto transfer feature knowledges [31, 38] and relation knowledges [26, 33]), the effective original KDstill outperforms most distillations and is widely used for different tasks [9, 17].
However, pre-trainedteacher model is not optimized for the student, limiting its distillation gains.
To address this issue,our SHAKE enhances KD using a shadow head to introduce optimization from the student.
Online distillations .
Online KD methods simplify the KD process by training all models simultaneously.
DML [39] performs bidirectional distillation for the peer networks and ONE [20] presentsan on-the-ﬂy ensemble distillation among multiple branches.
Subsequent studies focus on how tobalance multiple teacher [4] or construct the online teacher [37].
Online KD methods always obtainbetter performance than ofﬂine ones.
However, large teacher models trained from scratch sometimesperform unstable predictions and bring lots of training costs during distillation.
Therefore, we propose
SHAKE to combine the advantages of the two pipelines to facilitate application.
Adaptive distillations .
Capacity gaps between teacher-student models for their different architectureswould limit distillation gains [24].
There are two types of existing works to alleviate this gap in termsof training paradigms [8] and architectural adaptation [16, 11].
For instance, ESKD [5] proposesstopping the training of the teacher early, and ATKD [25] uses a medium-size teacher assistant toperform sequence distillation with large overheads.
However, these methods do not optimize theteacher model for the student, resulting in minor beneﬁts.
In sharp contrast to these methods, SHAKEis a student-aware ofﬂine KD method and opens a new direction for adaptive distillation design.
Conclusion
In this paper, we present SHAKE, a simple, effective, and new student-aware logits distillation tobridge ofﬂine and online knowledge transfer.
Based on our insight of online KD methods success,
SHAKE achieves this goal by building an extra shadow head as a proxy teacher model to performmutual distillation with the student model.
Thorough evaluations are performed on classiﬁcation anddetection, and SHAKE achieves signiﬁcant performance gains in various neural networks withoutextra inference overheads.
We hope that this elegant and practical approach would inspire futureresearch on knowledge distillation design and understanding.