Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer
Abstract
Knowledge distillation can be generally divided into offline and online categories according to whether teacher model is pre-trained and persistent during the distillation process.
Offline distillation can employ existing models yet always demonstrates inferior performance than online ones.
In this paper, we first empirically show that the essential factor for their performance gap lies in the reversed distillation from student to teacher, rather than the training fashion.
Offline distillation can achieve competitive performance gain by fine-tuning pre-trained teacher to adapt student with such reversed distillation.
However, this fine-tuning process still costs lots of training budgets.
To alleviate this dilemma, we propose SHAKE, a simple yet effective SHA dow K nowl E dge transfer framework to bridge offline and online distillation, which trades the accuracy with efficiency.
Specifically, we build an extra shadow head on the backbone to mimic the predictions of pre-trained teacher as its shadow.
Then, this shadow head is leveraged as a proxy teacher to perform bidirectional distillation with student on the fly.
In this way, SHAKE not only updates this student-aware proxy teacher with the knowledge of pre-trained model, but also greatly optimizes costs of augmented reversed distillation.
Extensive experiments on classification and object detection tasks demonstrate that our technique achieves state-of-the-art results with different CNNs and Vision Transformer models.
Additionally, our method shows strong compatibility with multi-teacher and augmentation strategies by gaining additional performance improvement.
Code is made publicly available at https://lilujunai.github.io/SHAKE/.
Introduction
Deep Neural Networks (DNNs) have achieved great success in tackling a variety of tasks [24, 11].
Despite the appealing performance, prevailing DNN models usually have large numbers of parameters, bringing heavy costs of computation.
To alleviate this problem, many network compression methods [26, 37] have been proposed, among which Knowledge Distillation (KD) [17, 48, 12] has recently attracted increased attention.
The aim of KD is to transfer the learnt knowledge of a high-capacity teacher model to a low-capacity student model.
Numerous offline methods [1, 17] use a two-stage training process that begins with training a teacher model and then keeping it fixed to distill student model (see KD in Figure 1 (a)).
Besides offline fashion, recent online methods [25, 5] adopt a one-stage training process, jointly training the student and teacher/peer models using bidirectional distillation like DML [53] in Figure(b).
These online distillations always surpass the offline distillations under the same teacher model.
However, some large teacher models trained from scratch would bring some difficulties ( e.g. , high computational resources and unstable optimizations) for distillation, especially for tasks that rely on large transformers ( e.g. , BERT [47], GPT-3 [2] and ViT-MoE [39]).
Therefore, two questions
[Figure]
Frozen
Tuned
[Figure]
KD
CE Forward Backward [Table] naturally arise: (1) Why is there a performance gap between offline and online distillation?
How the performance of offline distillation can be advanced with design techniques of online KD methods?
To clarify the first question, we compare KD and DML regarding the training fashion of teacher and distillation loss in Table 1.
Contrary to the common belief, we empirically observe that training fashion may not affect the distillation performance since DML † obtains similar performance as KD (70.55% vs. 70.66%).
Instead, the reversed distillation from the student model yields significant accuracy gains for KD † than KD (71.76% vs. 70.66%) and DML than DML † (71.52% vs. 70.55%).
Then, we analyze the output discrepancy of the teacher-student for different methods, and the reversed distillation reduces the gap from 1.12 of KD to 0.76 of KD † .
Therefore, the main reasons behind the performance gap lie in two aspects: (a) Teacher models in conventional offline pipelines are not optimized for the student model.
Thus, they could only provide general knowledge, which may be suboptimal for the particular student.
(b) The reversed distillation changes the universal teacher model into a student-aware one and bridges the teacher-student gap, which clearly justifies:
Teachers should teach students in accordance with their aptitude and should not follow the same pattern.
For the second question, fine-tuning the pre-trained teacher model with reversed distillation is a straightforward way to bridge the two training fashions.
However, in most scenarios, fine-tuning the whole network still needs lots of training budges (more 4 × costs than KD in Table 1).
Some trade-offs in reducing the fine-tuning parameters or adapter-based methods (see Figure 2 (G)) involve performance loss.
Thus, how to augment reversed distillation without much extra overhead is an important issue for the application.
Besides fine-tuning, building a proxy teacher model (see Figure(H)) to inherit the knowledge of pre-trained models and receive reversed distillation from students also enjoys the same benefits.
As shown in Table 1 (H), this proxy teacher model (SHAKE (alone)) can achieve competitive gains than fine-tuning the whole teacher model (KD † ).
Recent weight-sharing strategies in AutoML [50, 35] can effectively save training overhead and preserve logit consistency. T (G) cat
S
T (I) cat
S
T
Frozen
Tuned
KD
CE Forward Backward Adapter
Shadow (J) cat
S
T(H) cat
S
PT
T
PT Proxy Teacher This encourages us to adopt the same student architecture and weight-sharing strategy to generate the proxy teacher model. Thus, we allow this proxy teacher to share the backbone but use an individual shadow head to preserve the diversity of logits representations (see Figure 2 (I)).
As shown in Table 1 (I), this sharing strategy in SHAKE (share) presents four benefits: (1) more than 3 × training acceleration than DML and fine-tuning the whole teacher model.
(2) No need for architecture selection cost for proxy teacher.
(3) Additional accuracy gains than individual proxy teacher because knowledge inheritance also directly improves the representation in the weight-sharing process.
Based on the above observations, we propose SHA dow K nowl E dge distillation (SHAKE), a novel and effective logits distillation framework.
Our SHAKE builds a proxy teacher model and updates its weights via the original teacher model predictions.
In this way, SHAKE enjoys the same benefits with teacher fine-tuning to mine the knowledge of the pre-trained model and can perform bidirectional supervision with the student.
To optimize training costs, this proxy teacher shares the backbone with the pre-trained teacher but leverages an individual head to preserve the diversity of logits representations.
This head is named shadow head since it imitates the original teacher model just like its shadow.
During training, we only need to additionally train this shadow head with small training budgets (only 1.28 × costs than KD).
After training, all teacher models and shadow head can be deprecated, and student model can be separately applied for inference without any overhead.
Moreover, we extend the SHAKE to multi-teacher model scenarios using multiple shadow heads to inherit diverse knowledge.
In principle, SHAKE alters the chain of knowledge transfer from pre-trained teacher → student in KD to pre-trained teacher → proxy teacher ⇄ student .
Other adaptive KD [32] only employs middle networks to distill as large teacher → middle teacher → student , which is not optimized for students and requires multi-step training.
The merits of SHAKE lie in three-fold.
First, it effectively reduces the teacher-student capability gap with reversed distillation, bringing significant gains when pre-trained teacher models are available.
Second, for the scenario without pre-trained teacher models, SHAKE also enables the offline KD methods to be more effective in alleviating the unstable optimization issues of online KD methods.
Thus, SHAKE bridges the offline and online KD methods and enjoys the advantages of both methods.
Third, SHAKE achieves favorable trade-offs between accuracy and training budget.
By contrast, other adaptive KD [32] needs to sequentially train multiple models with lots of additional training time and resources.
We hope that these intriguing observations in SHAKE would expand the application of KD and facilitate future research for KD work to some extent.
We conduct extensive experiments on classification and detection to verify the superiority of the proposed method.
SHAKE achieves a consistent and significant accuracy boost in various neural networks and data augmentations, which outperforms other methods by large margins.
For example,
SHAKE obtains 2 .
92% ∼ 6 .
75% accuracy gains than baseline and more than 3 × training acceleration than DML on CIFAR-100.
On the challenging ImageNet dataset, SHAKE can improve the Top-  accuracy of ResNet-18 from 69.75% to 72.07% and MobileNet from 70.13% to 72.66%, which are state-of-the-art performances among KD methods.
On vision transformer architecture [45], our approach achieves 75.22% Top-1 accuracy and 2.76% gain for training a VIT-T model.
On the object detection task, SHAKE improves the AP by 1.02 for Faster R-CNN on the MS-COCO dataset, demonstrating the generality of our approach.
In summary, we make the following principle contributions in this paper:
• By analyzing and exploring the difference between offline and online KD methods, we empirically show that the reversed distillation hinders the performance gain, which fixes the discrepancy between teacher-student capability.
This motivates us to propose a new SHA dow K nowl E dge distillation (SHAKE) framework to bridge two training fashions, which, to the best of our knowledge, is not achieved in the area of knowledge distillation.
• SHAKE achieves remarkable trade-offs between accuracy and training efficiency with an extra shadow head.
The shadow head inherits knowledge from pre-trained models, introduces reversed distillation with students, and accelerates the training process.
• We perform thorough evaluations on classification and detection.
SHAKE achieves stateof-the-art performances in multiple datasets and architectures (e
g. CNN and vision transformer).
Specifically, ResNet-18, MobileNet, and VIT-T with SHAKE achieve 72.07%, 72.66% and 75.22% Top-1 accuracy on ImageNet, outperforming KD by 1.41%, 1.98% and 3.02%, respectively.
Shadow Knowledge Distillation
In this section, we first revisit the offline and online KD methods.
Then, we present the formulation of SHAKE and understand why SHAKE work from the perspective of optimization properties and teacher-student similarity.
Finally, we present the formulation of SHAKE and its expansion for multi-teacher models.
The evolution of our approach is shown in Figure 2.
2.1 Revisit of offline and online KD methods
We first review the formulations of offline and online KD.
For simplicity, We choose two typical frameworks ( i.e. , original KD [17] and DML [53]) for analysis.
Given training dataset ( X, Y ) where
X = { x i } ni =1 are training samples and Y =
{ y i } ni =1 are their labels.
Let f T be the output logits of the fixed teacher T and let f S be the output of student S , respectively.
In KD, the student network f S is trained by minimizing:
[Equation]
where λ is a weight for balancing these two terms.
L CE is the regular cross-entropy objective:
[Equation]
where CE ( · , · ) is the cross-entropy loss and σ is the softmax function.
L KL in Eq.
1 is the distillation objective for transferring knowledge from a teacher to a student:
[Equation]
where τ is the temperature to generate soft labels and KL represents Kullback-Leibler (KL) divergence.
This distillation loss can be considered as a modified cross-entropy loss using the probabilistic outputs of the pre-trained teacher network as the soft labels instead of the one-hot ground-truth labels.
Different from the one-way distillation in KD, DML presents a two-way knowledge transfer strategy in which the probabilistic outputs from both teacher and student networks can be used to guide the training of each other.
DML jointly trains the teacher and student networks in an end-to-end manner via interleaving optimizing two objectives:
[Equation]
where the default value of λ is 1 in DML, DML always outperforms KD under the same teacherstudent pair. The common belief argues that the accuracy gap between DML and KD comes from the additional boost of teacher models under mutual training fashion in DML than fixed teachers in KD. However, the significant gains of KD † than KD challenge this belief because the teacher’s accuracy of KD † does not obviously improve, but its teacher-student gap reduces, as shown in Table 1. In summary, introducing reversed distillation is a promising way to realize student-adapted KD methods, and we illustrate how SHAKE implements it in a cost-friendly way in the following section.
2.2 Formulation of SHAKE
As shown in Figure 2 (I), SHAKE proposes a proxy teacher model with output f T ′ via shadow head to augment reverse distillation in the offline framework.
In particular, the hierarchical features ( H T 1 , H T 2 ,..., H T i ) of the pre-trained teacher model are transformed by feature adaptation layer A and then performed a bottom-up feature fusion. For example, taking the three-layer feature fusion example, the proxy teacher’s output f T ′ can be defined as:
[Equation]
where MLP shadow is the fully connected shadow head with the same shape as the pre-trained teacher.
The shadow feature fusion scheme consists of directly feature addition ⊕ and Conv-BN-ReLU as feature adaptation layer A .
In the backpropagation, pre-trained teachers are fixed, and only shadow modules are updated.
Such shadow modules are simple to implement, and its additional costs are similar to other feature distillations [21, 6] and parameter efficient fine-tuning strategies. To achieve better logit alignment, we perform normalization ∥ · ∥ norm and scaling π for the output of the pre-trained teacher, proxy teacher, and student model, which follows advanced distillation.
During the training process, f T ′ is updated with the output of the pre-trained teacher model f T and performs mutual distillation with the student models f S as:
[Equation]
6) where each of the loss items has the same form as KD in Eq. 1, and the effects of λ are explored in the ablation study. The inherit distillation loss term D ( f T ′ , f T ) is presented to mimic the predictions of the pre-trained teacher model.
The detailed minimize distance D can be KL or MSE distance.
The reverse distillation term from student feedback to the proxy teacher and the direct distillation from the proxy teacher to the student play key roles in our framework, which is analyzed in detail from the perspective of training dynamics and teacher-student similarity in the following.
Comparison of SHAKE with other KDs in training dynamics .
Eq.
6, Eq. 4 and Eq. 1 clearly illustrate the difference of SHAKE and KD/DML.
Compared to KD, SHAKE introduces reversed distillation so that the teacher can be optimized by the student.
In contrast to DML, SHAKE leverages the knowledge of pre-trained models, resulting in additional accuracy gains.
As shown in Figure 3, SHAKE achieves a robust boost than KD and baseline during the training process.
Meanwhile, the training curve of DML shows a highly dynamic oscillation due to unreliable predictions of its teacher trained from scratch.
Other adaptive KD methods use the same knowledge transfer way as KD and do not optimize teacher with feedback supervision of student.
Thus, these methods, including ATKD [32] and ESKD [7] do not essentially address the teacher-student capacity disparity, and our SHAKE surpasses them with significant margins ( 1 . 48% ∼ 1 .
59% on ImageNet in Figure 3).
In-depth teacher-student similarity analysis in SHAKE.
The similarity between student and teacher network is an important measure for KD tasks.
Reversed distillation optimizes the teacher model to adapt to the student, reducing the teacher-student gap and improving distillation efficiency. With reversed KD loss, SHAKE surpasses KD and DML with a small teacher-student discrepancy. To verify this insight, we employ KL-divergence as metrics of similarity [7], where lower KL-divergence implies higher similarity. Figure 4 presents the similarities and performances between the outputs of
[Figure]
KD
SHAKE
Teacher
DML
ResNet with different depths as teachers ( e.g. , ResNet-110, ResNet-56, ResNet-44 and ResNet-32) and the ResNet-20 as student.
The results show that the distillation from SHAKE always gives a higher similarity than KD and DML, resulting in significant gains under teacher models of different depths.
Figure 5 presents the Grad-CAM visualization of ResNet-18 with different methods on
ImageNet.
The Grad-CAM maps of SHAKE pay more attention to the important regions than
KD and baseline, which have high similarity with the teacher.
In summary, SHAKE bridges KD and DML to obtain student-friendly distillations, improving teacher-student similarity.
In addition,
Figure 6 illustrates that comparing with KD, applying SHAKE training helps learn more scattered embeddings [33, 43, 31, 3] and logits correlation visualization in Figure 7 clearly demonstrate that teacher model in SHAKE owns student-aware logits.
2.3 Extension to multiple teacher models
As shown in Figure 2 (J), we build several proxy teachers with outputs ( f T ′ 1 , f T ′ 2 ,..., f Ti ′ ) to follow outputs ( f T 1 , f T 2 ,..., f T i ) of multiple teacher models with various shadow heads.
The mutual distillation also exists between the task head of the student and these multiple shadow heads.
Similar to
Eq. 6, the total optimization function for the multi-teacher scenario can be defined as:
[Equation]
where N is the total number of teacher models.
These multiple teacher models and shadows are also present only in the training phase and do not bring extra overhead for inference.
As shown in
Setting.
H of Table 1, this multi-shadow head strategy yields significant accuracy gains with small additional overhead.
Furthermore, SHAKE can be combined with advanced multi-teacher methods ( e.g. , adaptive weights [42] for different teachers in Table 2).
In this section, we first evaluate our approach for the classification task on CIFAR-100/ImageNet and the object detection task on MS-COCO.
Then, comprehensive ablation experiments are performed to analyze the key design in our SHAKE.
As a novel offline logit approach, the main competitor of
SHAKE is the original KD [17].
Thus, we perform detailed experimental comparisons between them.
Moreover, we compare the performance with recent advanced KD methods.
For fair comparisons, we use the public codes of these approaches with the same training and data preprocessing settings throughout the experiments.
For SHAKE, we set λ and τ as 1 and 4, respectively.
Please refer to the supplementary materials for more training settings. [Table] [Table] [Table]
[Table]
ReviewKD [6] [Table] [Table] N/A
[Table]
N/A
N/A
N/A
[Table]
NORM
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
N/A
[Table]
SRRL [19]
[Table]
[Table]
[Table]
[Table]
N/A
N/A
[Table]
[Table]
[Table]
[Table]
[Table]
ONE [25]
[Table]
[Table]
[Table]
N
[Table]
[Table]
[Table]
[Table]
[Table]
/A
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
KD+FitNets 75.
SHAKE+FitNets 76.
KD+CRD 75.
SHAKE+CRD 77.
KD+Mixup 75.
SHAKE+Mixup 76.
KD+CutMix 75.
SHAKE+CutMix 76.
KD+AVER 75.
SHAKE+AVER 76.
KD+AEKD 75.
SHAKE+AEKD 76.
3.1 Experiments on CIFAR-
Implementation .
On CIFAR-100 [23], we conduct experiments on various teacher-student models under the same or different architecture style with CRD’s settings [44], whose training epochs are 240.
We use a mini-batch size of 64 and a standard SGD optimizer with a weight decay of 0.0005.
The multi-step learning rate is initialized to 0.05, and decayed by 0.1 at 150, 180, and 210 epochs.
Comparison with offline and online KD methods .
In Table 2, we compare our approach to some advanced offline/online KD methods with the same training settings.
For the same architecture style of the teacher-students, SHAKE obtains 3 .
35% ∼ 5 .
45% absolute accuracy gains and outperforms KD with 0 .
26% ∼ 2 .
44% margins and Review with 1.72 % margins.
Besides, on cross-architecture [Table] [Table] teacher-student pairs, SHAKE achieves more significant gains with 4 .
70% ∼ 6 .
75% margins than baseline.
Compared to other KD methods, SHAKE outperforms KD with 0 .
66% ∼ 1 .
22% margins, which illustrates the effectiveness of SHAKE in reducing the teacher-student network gap.
Compared to DML and DML † under the same teacher-student pair in Table 2, SHAKE obtains 1 .
79% ∼ 4 .
20% relative gains.
Compare to online KDs ( e.g. , KDCL [14], ONE [25]) with multiple branches as teacher models, SHAKE also obtains 0 .
71% ∼ 3 .
77% relative accuracy gains.
These significant improvements demonstrate the superiority of SHAKE by leveraging the knowledge of pre-trained models.
Orthogonal to other KDs and data augmentations .
As acting on the output logits, SHAKE is orthogonal to feature and relation KD methods because of transferring different knowledge.
As shown in Table 2, the combination of FitNets with SHAKE surpasses KD with 0 .
91% ∼ 2 .
65% margins.
In addition, for CRD, its combination with SHAKE yields more dramatic gains than KD.
For distillation training under strong data augmentation ( e.g. , Mixup and CutMix [41]),SHAKE obtains 0 . 21% ∼ 3 . 05% relative accuracy gains than KD.
Extension to multiple teacher models.
In Table 2, we compare SHAKE and KD with average and adaptive weighting in multiple teacher settings.
The results ( i.e. , 0 . 75% ∼ 4 .
05% gains than KD) illustrate that the design of SHAKE with the multi-proxy model can effectively inherit the knowledge of different teachers bringing significant performance gains.
3.2 Experiments on ImageNet
Implementation .
For the standard ResNet-18 [15] and MobileNet [18] models, we adopt the same training setting as with most distillation methods, whose training epochs are 100.
The multi-step learning rate is initialized to 0.1, and decayed by 0.1 at 30, 60, and 90 epochs.
Recent vision transformers achieve great success on different vision tasks [8, 45].
We also extend SHAKE to VITT [45] with the same training settings ( e.g. , data augmentation and distillation token) on ImageNet.
Implementation details are available in supplementary materials.
Comparison results .
Table 3 reports the performance of our approach on ImageNet.
SHAKE improves the baseline models of ResNet-18 with 2 .
62% gains in Top-1 accuracy (see Figure 3 for detailed accuracy curves) and MobileNet with 2 . 83% gains.
Compared to other KD methods, SHAKE outperforms KD with 1 .
71% ∼ 2 .
28% margins and CRD with 1 . 20% ∼ 1 .
56% margins, which supports the superiority of SHAKE on the large-scale dataset.
Equipped with the distillation of two teacher models, SHAKE † obtains 2.98% gain for ResNet-18 and 3.29% gain for MobileNet than baseline.
As shown in Table 3, SHAKE obtains 2.76% accuracy gains than baseline and surpasses
KD under the soft or hard label of the CNN teacher (RegNet [36]), verifying its effectiveness on different architectures. [Figure]
3.3 Extensive experiments in diverse datasets and tasks
Experiments on CIFAR-10 and Tiny-ImageNet.
In addition to the commonly evaluated classification datasets ( i.e. , CIFAR-100 and ImageNet) in knowledge distillation methods, we also evaluated
SHAKE in CIFAR-10 and Tiny-ImageNet, which have smaller categories or data volume.
As shown in Figure 8, our intriguing observation about Reversed distillation is also present on different data sets: KD † with reversed KD outperforms KD, DML, and KD † . In addition, with a carefully designed shadow module and distillation strategy, our SHAKE outperforms KD † and achieves state-of-the-art results on CIFAR-10 and Tiny-ImageNet. Experiments on object detection . Besides classification tasks, we have also extended SHAKE for object detection applications.
In particular, we evaluate SHAKE on MS-COCO dataset [28] and use the most popular open-source Detectron2 [27] as the strong baseline.
We apply SHAKE to two-stage detector ( e.g. , Faster R-CNN [38]) , which are widely used object detection frameworks.
Following common practice [27], all models are trained with a 2 × learning schedule (24 epochs).
As shown in
Figure 8, our SHAKE improves the AP 1.02 on Faster R-CNN, which outperforms KD [17].
The performance of object detection greatly depends on the quality of deep features to locate interesting objects, while logits are not capable of providing knowledge for object localization.
Thus, SHAKE is naturally weaker than Review [6] (recent state-of-the-art feature-based KDs) and we further introduce Review to obtain satisfactory results.
It can be observed that SHAKE can obtain new state-of-the-art results with feature-based KDs.
The success of challenging object detection tasks demonstrates the generality and effectiveness of our approach.
3.4 Ablation study
In this section, we isolate the impact of each component of our approach.
All experiments are conducted on the CIFAR-100 dataset.
For each setting, we run 3 times and report Top-1 mean accuracies.
Detailed ablation study of SHAKE As shown in Figure 9, an ablation study has been conducted to demonstrate the individual effectiveness of different components in SHAKE.
It is observed that (a) Reversed distillation optimizes the teacher model to adapt to the student, improving distillation efficiency.
SHAKE, without reversed distillation, only obtains marginal gains.
(b) Shadow modules are effective in preserving the diversity of knowledge, and their backbone sharing settings can accelerate the convergence of the student model and results in performance gains.
In addition, the stronger shadow module design for storing and teaching more pre-trained knowledge when the capacity gap enlarges.
(c) Inherited knowledge from pre-trained teacher via distillation optimization beyond sharing weights is important.
Proxy teachers in our SHAKE to transfer the knowledge of the original pre-trained teacher model for the student model to learn more easily.
These results demonstrate that our SHAKE proxy teacher design can indeed serve as a good bridge between the original teacher and student.
Reverse KD reduces the teacher-student gap to facilitate knowledge transfer.
Sensitivity study for temperature τ and weight λ .
τ in Eq.
3 controls the softness of teacher’s supervision.
As τ increases, the output of the softmax function becomes smoother.
As shown in
Figure 10, SHAKE presents superior gains and robustness than the original KD in different τ .
λ is the hyper-parameter that balances the weight of KD loss.
The small λ limits distillation gains, and λ = 0 .
5 is the best option for SHAKE.
Related Work
Offline distillations .
In the offline framework [17], the teacher is pre-trained and fixed, and then its soft logits are used as extra supervision to distill students.
Although subsequent methods explored to transfer feature knowledges [40, 52] and relation knowledges [34, 44]), the effective original KD still outperforms most distillations and is widely used for different tasks [10, 22].
However, the pre-trained teacher model is not optimized for the student, limiting its distillation gains.
To address this issue, our SHAKE enhances KD using a shadow head to introduce optimization from the student.
Online distillations .
Online KD methods simplify the KD process by training all models simultaneously.
DML [53] performs bidirectional distillation for the peer networks and ONE [25] presents an on-the-fly ensemble distillation among multiple branches.
Subsequent studies focus on how to balance multiple teacher [5] or construct the online teacher [49].
Online KD methods always obtain better performance than offline ones.
However, large teacher models trained from scratch sometimes perform unstable predictions and bring lots of training costs during distillation.
Therefore, we propose
SHAKE combine the advantages of the two pipelines to facilitate application.
Adaptive distillations .
Capacity gaps between teacher-student models for their different architectures would limit distillation gains [29].
There are two types of existing works to alleviate this gap in terms of training paradigms [9] and architectural adaptation [20, 13].
For instance, ESKD [7] proposes stopping the training of the teacher early, and ATKD [32] uses a medium-size teacher assistant to perform sequence distillation with large overheads.
However, these methods do not optimize the teacher model for the student, resulting in minor benefits.
In sharp contrast to these methods, SHAKE is a student-aware offline KD method and opens a new direction for adaptive distillation design.
Conclusion
In this paper, we present SHAKE, a simple, effective, and new student-aware logit distillation to bridge offline and online knowledge transfer.
Based on our insight into online KD methods success,
SHAKE achieves this goal by building an extra shadow head as a proxy teacher model to perform mutual distillation with the student model.
Thorough evaluations are performed on classification and detection, and SHAKE achieves significant performance gains in various neural networks without extra inference overheads.
We hope that this elegant and practical approach would inspire future research on knowledge distillation design and understanding.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] .
Our method will add additional training overhead.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our approach is a new knowledge distillation application technique, which helps lightweight models to achieve better performance without additional teacher architectures.
It does not bring negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[No] (b) Did you include complete proofs of all theoretical results?
[No]
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See implementation details in our experiment Section 3.2 and supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 2 and more results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Table 1. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
We evaluate our approach in a generic public dataset, following lots of previous research works.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
The datasets we use does not contain personally identifiable information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
Our approach is a generic training framework, which does not concern human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?