Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer
Abstract
Knowledge distillation can be generally divided into offline and online categories according to whether teacher model is pre-trained and persistent during the distillation process.
Offline distillation can employ existing models yet always demonstrates inferior performance than online ones.
In this paper, we first empirically show that the essential factor for their performance gap lies in the reversed distillation from student to teacher, rather than the training fashion.
Offline distillation can achieve competitive performance gain by fine-tuning pre-trained teacher to adapt student with such reversed distillation.
However, this fine-tuning process still costs lots of training budgets.
To alleviate this dilemma, we propose SHAKE, a simple yet effective SHA dow K nowl E dge transfer framework to bridge offline and online distillation, which trades the accuracy with efficiency.
Specifically, we build an extra shadow head on the backbone to mimic the predictions of pre-trained teacher as its shadow.
Then, this shadow head is leveraged as a proxy teacher to perform bidirectional distillation with student on the fly.
In this way, SHAKE not only updates this student-aware proxy teacher with the knowledge of pre-trained model, but also greatly optimizes costs of augmented reversed distillation.
Extensive experiments on classification and object detection tasks demonstrate that our technique achieves state-of-the-art results with different CNNs and Vision Transformer models.
Additionally, our method shows strong compatibility with multi-teacher and augmentation strategies by gaining additional performance improvement.
Code is made publicly available at https://lilujunai.github.io/SHAKE/.
Introduction
Even though Deep Neural Networks (DNNs) have been shown to achieve great performance in tackling a variety of challenges over computer vision [26, 12] and natural language processing [2], they usually have large numbers of parameters, bringing heavy computation costs.
To alleviate this problem, many network compression methods [28, 45] have been proposed, among which Knowledge
Distillation (KD)
[18, 57, 13] has recently attracted increased attention.
KD aims to transfer knowledge from a high-capacity large model ( i.e. , teacher) to a low-capacity lightweight model ( i.e. , student).
Numerous offline methods [1, 18] use a two-stage training process that begins with training a teacher model and then keeping it fixed to distill student model (see KD in
Figure 1 (a)).
Besides offline fashion, recent online methods [27, 5] adopt a one-stage training process, jointly training the student and teacher/peer models using bidirectional distillation like DML [63] in
Figure 1 (b).
These online distillations always surpass the offline distillations under the same teacher model.
However, some large teacher models trained from scratch would bring some difficulties ( e.g. , high computational resources and unstable optimizations) for distillation, especially for tasks that rely on large transformers ( e.g. , BERT [56], GPT-3 [2] and ViT-MoE [47]).
Therefore, two questions
[Figure]
Frozen
Tuned
[Figure]
KD
CE Forward
Backward
[Table]
naturally arise: (1) Why is there a performance gap between offline and online distillation?
How the performance of offline distillation can be advanced with design techniques of online
KD methods?
To clarify the first question, we compare KD and DML regarding the training fashion of teacher and distillation loss in Table 1.
Contrary to the common belief, we empirically observe that training fashion may not affect the distillation performance since DML † obtains similar performance as KD (70.55% vs. 70.66%).
Instead, the reversed distillation from the student model yields significant accuracy gains for KD † than KD (71.76% vs. 70.66%) and DML than DML † (71.52% vs. 70.55%).
Then, we analyze the output discrepancy of the teacher-student for different methods, and the reversed distillation reduces the gap from 1.12 of KD to 0.76 of KD † .
Therefore, the main reasons behind the performance gap lie in two aspects: (a) Teacher models in conventional offline pipelines are not optimized for the student model.
Thus, they could only provide general knowledge, which may be suboptimal for the particular student.
(b) The reversed distillation changes the universal teacher model into a student-aware one and bridges the teacher-student gap, which clearly justifies:
Teachers should teach students in accordance with their aptitude and should not follow the same pattern.
For the second question, fine-tuning the pre-trained teacher model with reversed distillation is a straightforward way to bridge the two training fashions.
However, in most scenarios, fine-tuning the whole network still needs lots of training budges (more 4 × costs than KD in Table 1).
Some trade-offs in reducing the fine-tuning parameters or adapter-based methods (see Figure 2 (G)) involve performance loss.
Thus, how to augment reversed distillation without much extra overhead is an important issue for the application.
Besides fine-tuning, building a proxy teacher model (see Figure(H)) to inherit the knowledge of pre-trained models and receive reversed distillation from students also enjoys the same benefits.
As shown in Table 1 (H), this proxy teacher model (SHAKE (alone)) can achieve competitive gains than fine-tuning the whole teacher model (KD † ).
Recent weightsharing strategies in AutoML [60, 42, 20, 8] can effectively save training overhead and preserve logit
T (G) cat
S
T (I) cat
S
T
Frozen
Tuned
KD
CE Forward Backward Adapter
Shadow (J) cat
S
T(H) cat
S
PT
T
PT Proxy Teacher consistency.
This encourages us to adopt the same student architecture and weight-sharing strategy to generate the proxy teacher model.
Thus, we allow this proxy teacher to share the backbone but use an individual shadow head to preserve the diversity of logits representations (see Figure 2 (I)).
As shown in Table 1 (I), this sharing strategy in SHAKE (share) presents four benefits: (1) more than 3 × training acceleration than DML and fine-tuning the whole teacher model.
(2) No need for architecture selection cost for proxy teacher.
(3) Additional accuracy gains than individual proxy teacher because knowledge inheritance also directly improves the representation in the weight-sharing process.
Based on the above observations, we propose SHA dow K nowl E dge distillation (SHAKE), a novel and effective logits distillation framework.
Our SHAKE builds a proxy teacher model and updates its weights via the original teacher model predictions.
In this way, SHAKE enjoys the same benefits with teacher fine-tuning to mine the knowledge of the pre-trained model and can perform bidirectional supervision with the student.
To optimize training costs, this proxy teacher shares the backbone with the pre-trained teacher but leverages an individual head to preserve the diversity of logits representations.
This head is named shadow head since it imitates the original teacher model just like its shadow.
During training, we only need to additionally train this shadow head with small training budgets (only 1.28 × costs than KD).
After training, all teacher models and shadow module can be deprecated, and student model can be separately applied for inference without any overhead.
Moreover, we extend the SHAKE to multi-teacher model scenarios using multiple shadow heads to inherit diverse knowledge.
In principle, SHAKE alters the chain of knowledge transfer from pre-trained teacher → student in KD to pre-trained teacher → proxy teacher ⇄ student .
Other adaptive KD [39] only employs middle networks to distill as large teacher → middle teacher → student , which is not optimized for students and requires multi-step training.
The merits of SHAKE lie in three-fold.
First, it effectively reduces the teacher-student capability gap with reversed distillation, bringing significant gains when pre-trained teacher models are available.
Second, for the scenario without pre-trained teacher models, SHAKE also enables the offline KD methods to be more effective in alleviating the unstable optimization issues of online KD methods.
Thus, SHAKE bridges the offline and online KD methods and enjoys the advantages of both methods.
Third, SHAKE achieves favorable trade-offs between accuracy and training budget.
By contrast, other adaptive KD [39] needs to sequentially train multiple models with lots of additional training time and resources.
We hope that these intriguing observations in SHAKE would expand the application of KD and facilitate future research for KD work to some extent.
We conduct extensive experiments on multiple tasks ( e.g. ,classification and detection) and datasets ( e.g. , CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and MS-COCO) to verify the superiority of the proposed method.
SHAKE achieves a consistent and significant accuracy boost in various neural networks and data augmentations, which outperforms other methods by large margins.
For example, SHAKE obtains 2 .
92% ∼ 6 .
75% accuracy gains than baseline and more than 3 × training acceleration than DML on CIFAR-100.
On the challenging ImageNet dataset, SHAKE can augment ResNet-18’s Top-1 performance from 69.75% to 72.07% and MobileNet from 70.13% to 72.66%, which are state-of-the-art results for KD techniques.
On vision transformer architecture [54], our approach achieves 75.22% Top-1 accuracy and 2.76% gain for training a VIT-T model.
On the object detection task, SHAKE improves the AP by 1.02 for Faster R-CNN on the MS-COCO, demonstrating its generality.
In summary, we make the following principle contributions in this paper:
• By analyzing and exploring the difference between offline and online KD methods, we empirically show that the reversed distillation hinders the performance gain, which fixes the discrepancy between teacher-student capability.
This motivates us to propose a new SHA dow K nowl E dge distillation (SHAKE) framework to bridge two training fashions, which, to the best of our knowledge, is not achieved throughout the area of knowledge distillation.
• SHAKE achieves remarkable trade-offs between accuracy and training efficiency with an extra shadow head.
The shadow head inherits knowledge from pre-trained models, introduces reversed distillation with students, and accelerates the training process.
• We perform thorough evaluations on classification and detection.
SHAKE achieves state-ofthe-art performances in various datasets and architectures ( e
g. , CNN and vision transformer).
Specifically, ResNet-18, MobileNet, and VIT-T with SHAKE achieve 72.07%, 72.66% and 75.22% Top-1 accuracy on ImageNet, outperforming KD by 1.41%, 1.98% and 3.02%, respectively.
Shadow Knowledge Distillation
In this section, we first revisit the offline and online KD methods.
Then, we present the formulation of SHAKE and understand why SHAKE work from the perspective of optimization properties and teacher-student similarity.
Finally, we present the formulation of SHAKE for multi-teacher models.
The evolution of our approach is shown in Figure 2.
2.1 Revisit of offline and online KD methods
We first review the formulations of offline and online KD.
For simplicity, We choose two typical frameworks ( i.e. , original KD [18] and DML [63]) for analysis.
Regarding training data ( X, Y ) containing training samples X and labels Y .
Let f T be the output logits of the fixed teacher T and let f S be the output of student S , respectively.
KD equips the student network fS via minimizing:
[Equation]
where λ is used to balance these two terms.
L CE is the regular cross-entropy objective:
[Equation]
where σ is the softmax function and CE ( · , · ) is the cross-entropy loss.
L KL in Eq.
1 is the distillation goal for transmitting knowledge from a teacher to a student:
[Equation]
where τ is the temperature to generate soft labels and KL represents Kullback-Leibler (KL) divergence.
The teacher’s probabilistic outputs can be viewed as the soft labels in this distillation loss rather than the one-hot ground-truth labels, making it a modified cross-entropy loss.
DML provides a two-way knowledge transfer technique in contrast to KD’s one-way distillation. The probabilistic outputs from both teacher and student networks can be used to direct one
another’s training.
DML interleaves two aims to jointly train the teacher and student networks in an end-to-end manner:
[Equation]
where the default value of λ is 1 in DML, DML consistently outperforms KD under the same teacherstudent pair.
The common belief argues that the accuracy gap between DML and KD comes from the additional boost of teacher models under mutual training fashion in DML than fixed teachers in KD.
However, the significant gains of KD † than KD challenge this belief because the teacher’s accuracy of KD † does not obviously improve, but its teacher-student gap reduces, as shown in Table 1.
In summary, introducing reversed distillation is a promising way to realize student-adapted KD methods. We illustrate how SHAKE implements it in a cost-friendly way in the following section.
2.2 Formulation of SHAKE
As shown in Figure 2 (I), SHAKE proposes a proxy teacher model with output f T ′ via shadow head to augment reverse distillation in the offline framework.
In particular, the hierarchical features ( H T 1 , H T 2 ,..., H T i ) of the pre-trained teacher model are transformed by feature adaptation layer A and then performed via a bottom-up feature fusion.
For example, for the three-layer feature fusion, the proxy teacher’s output f T ′ can be defined as:
[Equation]
where MLP shadow is the fully connected layer with the same shape as the pre-trained teacher.
The shadow feature fusion scheme consists of directly feature addition ⊕ and Conv-BN-ReLU as feature adaptation layer A .
In the backpropagation, pre-trained teachers are fixed, and only shadow modules are updated.
Such shadow modules are simple to implement, and its additional costs are similar to other feature distillations [24, 6] and parameter efficient fine-tuning strategies [22].
To achieve better logit alignment, we perform normalization ∥ · ∥ norm and scaling π for the output of the pre-trained teacher, proxy teacher, and student model, which follows advanced distillation [51, 27, 30].
During the training process, f T ′ is updated with the output of the pre-trained teacher model f T and performs mutual distillation with the student models f S as:
[Equation]
Each loss item has the same form as KD in Eq. 1, and the effects of λ are explored in the ablation study.
The inherit distillation loss term D ( f T ′ , f T ) is presented to imitate the probabilistic outputs of pre-trained teacher, which can be defined as KL or MSE distance.
The reverse distillation term from student feedback to the proxy teacher and the direct distillation from the proxy teacher to the student play key roles in our framework, which is analyzed in detail from the perspective of training dynamics and teacher-student similarity in the following
Comparison of SHAKE with other KDs in training dynamics .
Eq.
6, Eq. 4 and Eq.
1 clearly illustrate the difference of
SHAKE and KD/DML.
Compared to KD, SHAKE introduces reversed distillation so that the teacher can be optimized by the student.
In contrast to DML, SHAKE leverages the knowledge of pre-trained models, resulting in additional accuracy gains.
As shown in Figure 3, SHAKE achieves a robust boost than
KD and baseline during the training process.
Meanwhile, the training curve of DML shows a highly dynamic oscillation due to unreliable predictions of its teacher trained from scratch.
Other adaptive KD methods use the same knowledge transfer way as KD and do not optimize teacher with feedback supervision of student.
Thus, these methods, including ATKD [39] and ESKD [7] do not essentially address the teacher-student capacity disparity, and our SHAKE surpasses them with significant margins ( 1 . 18% ∼ 1 . 29% on ImageNet in Figure 3).
In-depth analysis about teacher-student similarity in SHAKE.
The similarity between student and teacher network is an important measure for KD tasks.
Reversed distillation optimizes the teacher model to adapt to the student, reducing the teacher-student gap and improving distillation efficiency.
With reversed KD loss, SHAKE surpasses KD and DML with a small teacher-student discrepancy.
To verify this insight, we employ KL-divergence as metrics of similarity [7], where lower KL-divergence implies higher similarity.
Figure 4 presents the similarities and performances between the outputs of
[Figure]
KD
SHAKE
Teacher
DML
ResNet with different depths as teachers
The results show that the distillation from SHAKE always gives a higher similarity than KD and
DML, resulting in significant gains under teacher models of different depths.
Figure 5 presents the
Grad-CAM visualization of ResNet-18 with different methods on ImageNet.
The Grad-CAM maps of SHAKE pay more attention to the important regions than KD and baseline, which have high similarity with the teacher.
In summary, SHAKE bridges KD and DML to obtain student-friendly distillations, improving teacher-student similarity.
In addition, Figure 6 illustrates that comparing with KD, applying SHAKE training helps learn more scattered embeddings [40, 52, 38, 3] and logits correlation visualization in Figure 7 clearly demonstrate that teacher model in SHAKE owns student-aware logits.
2.3 Extension to multiple teacher models
As shown in Figure 2 (J), we build several proxy teachers with outputs ( f T ′ 1 , f T ′ 2 ,..., f Ti ′ ) to follow outputs ( f T 1 , f T 2 ,..., f T i ) of multiple teacher models with various shadow heads.
Mutual distillation also exists between the task head of the student and these multiple shadow heads.
Similar to Eq.
6, the total optimization function for the multi-teacher scenario can be defined as:
[Equation]
where N is the sum of the total teachers.
These multiple teacher models and shadows are present only in the training phase and do not bring extra overhead for inference.
As shown in Setting (J) of Table 1, this multi-shadow head strategy yields significant accuracy gains with small additional overhead.
Furthermore, SHAKE can be combined with advanced multi-teacher methods ( e.g. , adaptive weights [50] for different teachers in Table 2).
Experiments
In this part, we first assess our SHAKE for classification and object detection.
After that, extensive ablations are conducted to investigate our SHAKE’s principal design.
As a novel offline logit approach, the main competitor of SHAKE is the original KD [18].
Thus, we perform detailed experimental comparisons between them.
Moreover, we compare the performance with recent advanced KD methods.
We employ the open-source versions of these methods with the same training settings for fair comparisons.
We choose λ and τ to be 1 and 4 in SHAKE, respectively.
[Table]
N/A
N/A
N/A 77.
NORM [59] 75.
N/A 77.
SRRL [21] 75.
N/A
N/A 74.
T f -FD [29] 74.
N/A 70.
N/A
N/A
N/A
ONE [27] 74.
N/A 72.
N/A
KDCL [15] 74.
MetaDistil [65]
N/A 71.
N/A 73.
N/A 74.
DML [63] 75.
DML † 74.
KD [18] 74.
KD † (ours) 75.
SHAKE(ours) 76.
Std & Gain
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
KD+FitNets 75.
SHAKE+FitNets 76.
KD+CRD 75.
SHAKE+CRD 77.
KD+Mixup 75.
SHAKE+Mixup 76.
KD+CutMix 75.
SHAKE+CutMix 76.
KD+AVER 75.
SHAKE+AVER 76.
KD+AEKD 75.
SHAKE+AEKD 76.
3.1 Experiments on CIFAR-
Implementation .
With CRD’s settings [53], whose training epochs are 240, we perform experiments on several teacher-student models on CIFAR-100, either using the same architecture style or a different one.
We employ a conventional SGD optimizer with a weight decay of 0.0005 and a mini-batch size of 64.
Initialized at 0.05, the multi-step learning rate decrements by 0.1 at 150, 180, and 210 epochs.
Comparison with offline and online KD methods .
In Table 2, we compare our approach to some advanced offline/online KD methods with the same training settings.
For the same architecture style of the teacher-students, SHAKE obtains 3 .
35% ∼ 5 .
45% absolute accuracy gains and outperforms
KD with 0 .
26% ∼ 2 .
44% margins and Review with 1.72 % margins.
Besides, SHAKE achieves more significant gains on cross-architecture teacher-student pairs with 4 .
70% ∼ 6 .
75% margins than
[Table] [Table]
baseline.
Compared to other KD methods, SHAKE outperforms KD with 0 .
66% ∼ 1 .
22% margins, which illustrates the effectiveness of SHAKE in reducing the teacher-student network gap.
Compared to DML and DML † under the same teacher-student pair in Table 2, SHAKE obtains 1 .
79% ∼ 4 .
20% relative gains.
Compare to online KDs ( e.g. , KDCL [15], ONE [27]) with multiple branches as teacher models, SHAKE also obtains 0 .
71% ∼ 3 .
77% relative accuracy gains.
These significant improvements demonstrate the superiority of SHAKE by leveraging the knowledge of pre-trained models.
Orthogonal to other KDs and data augmentations .
As acting on the output logits, SHAKE is orthogonal to feature and relation KD methods because of transferring different knowledge.
As shown in Table 2, the combination of FitNets with SHAKE surpasses KD with 0 .
91% ∼ 2 .
65% margins.
In addition, for CRD, its combination with SHAKE yields more dramatic gains than KD.
For distillation training under strong data augmentation ( e.g. , Mixup and CutMix [49]), SHAKE obtains 0 .
21% ∼ 3 .
05% relative accuracy gains than KD.
Extension to multiple teacher models.
In Table 2, we compare SHAKE and KD with average and adaptive weighting in multiple teacher settings.
The results ( i.e. , 0 .
75% ∼ 4 .
05% gains than KD) illustrate that the design of SHAKE with the multi-proxy model can effectively inherit the knowledge of different teachers bringing significant performance gains.
3.2 Experiments on ImageNet
Implementation .
For the standard ResNet-18 [16] and MobileNet [19] models, we employ the same training configurations, whose the training epochs are 100, as most distillation techniques.
The learning rate is initialized at 0.1, decreasing by 0.1 every 30 epochs.
Recent vision transformers achieve great success on different vision tasks [9, 54].
We also extend SHAKE to VIT-T [54] with the same training settings ( e.g. , data augmentation and distillation token) on ImageNet.
Implementation details are available in supplementary materials.
Comparison results .
Table 3 reports the performance of our approach on ImageNet.
SHAKE improves the baseline models of ResNet-18 with 2 .
32% gains in Top-1 accuracy (see Figure 3 for detailed accuracy curves) and MobileNet with 2 . 53% gains.
Compared to other KD methods, SHAKE outperforms KD with 1 .
41% ∼ 1 .
98% margins and CRD with 0 . 90% ∼ 1 .
26% margins, which supports the superiority of SHAKE on the large-scale dataset.
Equipped with the distillation of two teacher models, SHAKE † obtains 2.78% gain for ResNet-18 and 2.89% gain for MobileNet than baseline.
As shown in Table 4, SHAKE obtains 2.76% accuracy gains than baseline.
It surpasses KD under the soft or hard label of the CNN teacher (RegNet [44]), verifying its effectiveness on different architectures.
[Figure]
3.3 Extensive experiments in diverse datasets and tasks
Experiments on CIFAR-10 and Tiny-ImageNet.
In addition to the commonly evaluated classification datasets ( i.e. , CIFAR-100 and ImageNet) in knowledge distillation methods, we also evaluated
SHAKE in CIFAR-10 and Tiny-ImageNet, which have smaller categories and data volumes.
As shown in Figure 8, our intriguing observation about Reversed distillation is also present on different data sets: KD † with reversed KD outperforms KD, DML, and KD † .
In addition, with a carefully designed shadow module and distillation strategy, our SHAKE outperforms KD † and achieves best results.
Experiments on object detection .
Besides classification tasks, we also extend SHAKE for object detection applications.
In particular, we evaluate SHAKE on MS-COCO dataset [34] and use the most popular open-source Detectron
[33] as the strong baseline.
We apply SHAKE to two-stage detector ( i.e. , Faster R-CNN [46]), which are widely used object detection frameworks.
Following common practice [33], all models are trained with a 2 × learning schedule (24 epochs).
As shown in
Figure 8, our SHAKE improves the AP 1.02 on Faster R-CNN, which outperforms KD [18].
The performance of object detection greatly depends on the quality of deep features to locate interesting objects, while logits are not capable of providing knowledge for object localization.
Thus, SHAKE is naturally weaker than Review [6] (recent excellent feature-based KDs), and we further introduce
Review to obtain satisfactory results.
It can be observed that SHAKE can obtain new best results with feature-based KDs.
The success of challenging object detection tasks demonstrates the generality and effectiveness of our approach.
3.4 Ablation study
Detailed ablation study of SHAKE As shown in Figure 9, an ablation study has been conducted to demonstrate the individual effectiveness of different components in SHAKE.
It is observed that (a) Reversed distillation optimizes the teacher model to adapt to the student, improving distillation efficiency.
SHAKE, without reversed distillation, only obtains marginal gains.
(b) Shadow modules effectively preserve the diversity of knowledge, and their backbone-sharing settings can accelerate the student model’s convergence and result in performance gains.
In addition, the stronger shadow module design for storing and teaching more pre-trained knowledge when the capacity gap enlarges.
(c) Inherited knowledge from pre-trained teacher via distillation optimization beyond sharing weights is important.
Proxy teachers in our SHAKE transfer the knowledge of the original pre-trained teacher model more efficiently.
These results demonstrate that our proxy teacher design can indeed serve as a good bridge between the original teacher and student.
Reverse KD reduces the teacher-student gap to facilitate knowledge transfer.
Sensitivity study for temperature τ and weight λ .
τ in Eq.
3 controls the softness of teacher’s supervision.
As τ increases, the output of the softmax function becomes smoother.
As shown in
Figure 10, SHAKE presents superior gains and robustness than the original KD in different τ .
λ balances the implications of KD loss.
The small λ limits distillation gains, and λ = 0 .
5 is the best option for SHAKE.
Related Work
Offline distillations .
In the offline framework [18], the teacher is pre-trained and non-updated, and then its soft logits are used as extra supervision to distill students.
Although subsequent methods explored to transfer feature knowledges [48, 62] and relation knowledges [41, 53]), the effective original KD still outperforms most distillations and is widely used for different tasks [11, 25, 30, 31].
However, the pre-trained teacher model is not optimized for the student, limiting its gains.
To address this issue, our SHAKE enhances KD using a shadow head to introduce optimization from the student.
Online distillations .
Online KD methods simplify the KD process by training all models simultaneously.
DML [63] performs bidirectional distillation for the peer networks and ONE [27] presents an on-the-fly ensemble distillation among multiple branches.
Subsequent studies focus on how to balance multiple teacher [5] or construct the online teacher [58, 32, 37].
Online KD methods always obtain better performance than offline ones.
However, large teacher models trained from scratch sometimes perform unstable predictions and bring lots of training costs during distillation.
Therefore, we propose that SHAKE combine the advantages of the two pipelines to facilitate application.
Adaptive distillations .
Capacity gaps between teacher-student models for their different architectures would limit distillation gains [36].
There are two types of existing works to alleviate this gap in terms of training paradigms [10] and architectural adaptation [23, 14].
For instance, ESKD [7] proposes stopping the training of the teacher early, and ATKD [39] uses a medium-size teacher assistant to perform sequence distillation with large overheads.
However, these methods do not optimize the teacher model for the student, resulting in minor benefits.
Recent Meta-KD [65, 64, 35] implement quite complex meta-optimization for whole teacher networks with a lot of extra overhead.
In sharp contrast to these methods, SHAKE is a student-aware offline KD method and opens a new direction for adaptive distillation design.
Conclusion
In this paper, we present SHAKE, a new student-aware logit distillation that is easy to use and effective for bridging offline and online knowledge transfer.
Based on our insight into online KD methods success, SHAKE achieves this goal by building an extra shadow head as a proxy teacher model to perform mutual distillation with the student model.
Thorough evaluations are performed on classification and detection, and SHAKE achieves significant performance gains in various neural networks without extra inference overheads.
In future work, we would make the best effort to explore the application of SHAKE for different specific tasks [43].
We hope that this elegant and practical approach would inspire future research on knowledge distillation design and understanding.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] .
Our method will add additional training overhead.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our approach is a new knowledge distillation application technique, which helps lightweight models to achieve better performance without additional teacher architectures.
It does not bring negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[No] (b) Did you include complete proofs of all theoretical results?
[No] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See implementation details in our experiment Section 3.2 and supplemental materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 2 and more results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Table 1. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
We evaluate our approach in a generic public dataset, following lots of previous research works.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
The datasets we use does not contain personally identifiable information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
Our approach is a generic training framework, which does not concern human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]