Shadow Knowledge Distillation: Bridging Ofﬂine and Online Knowledge Transfer
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Knowledge distillation can be generally divided into ofﬂine and online categories according to whether teacher model is pre-trained and persistent during the distillationprocess.
Ofﬂine distillation can employ existing models yet always demonstratesinferior performance than online ones.
In this paper, we ﬁrst empirically showthat the essential factor for their performance gap lies in the reversed distillationfrom student to teacher, rather than the training fashion.
Ofﬂine distillation canachieve competitive performance gain by ﬁne-tuning pre-trained teacher to adaptstudent with such reversed distillation.
However, this ﬁne-tuning process stillcosts lots of training budgets.
To alleviate this dilemma, we propose SHAKE, asimple yet effective SHA dow K nowl E dge transfer framework to bridge ofﬂineand online distillation, which trades the accuracy with efﬁciency.
Speciﬁcally,we build an extra shadow head on the student’s backbone to mimic the predictions of pre-trained teacher as its shadow.
Then, this shadow head is leveragedas a proxy teacher to perform bidirectional distillation with student on the ﬂy.
In this way, SHAKE not only updates this student-aware proxy teacher with theknowledge of pre-trained model, but also greatly optimizes costs of augmentedreversed distillation.
Extensive experiments on classiﬁcation and object detectiontasks demonstrate that our technique achieves state-of-the-art results with different
CNNs and Vision Transformer models.
Additionally, our method shows strongcompatibility with multi-teacher and augmentation strategies by gaining additionalperformance improvement.
Code will be made publicly available.
Introduction
Deep Neural Networks (DNNs) have achieved great success in tackling a variety of tasks [22, 12].
Despite appealing performance, prevailing DNN models usually have large numbers of parameters,bringing heavy costs of computation.
To alleviate this problem, many network compression methods [24, 35] have been proposed among which Knowledge Distillation (KD)
[17, 47, 13] has recentlyattracted increased attention.
The aim of KD is to transfer the learnt knowledge of a high-capacity teacher model to a low-capacitystudent model.
Numerous ofﬂine methods [1, 17] use a two-stage training process that begins withtraining a teacher model and then keeping it ﬁxed to distill student model (see KD in Figure 1 (a)).
Besides ofﬂine fashion, recent online methods [23, 5] adopt a one-stage training process, jointlytraining the student and teacher/peer models using bidirectional distillation like DML [52] in Figure 1(b).
These online distillations always surpass the ofﬂine distillations under the same teacher model.
However, some large teacher models trained from scratch would bring some difﬁculties ( e.g. , highcomputational resources and unstable optimizations) for distillation, especially for tasks that relyon large transformers ( e.g. , BERT [46], GPT-3 [2] and ViT-MoE [37]).
Therefore, two questions
[Figure]
[Table]
naturally arise: (1) Why is there a performance gap between ofﬂine and online distillation?
(2) Howthe performance of ofﬂine distillation can be advanced with design techniques of online KD methods?
To clarify the ﬁrst question, we compare KD and DML regarding the training fashion of teacherand distillation loss in Table 1.
Contrary to the common belief, we empirically observe that trainingfashion may not affect the distillation performance since DML † obtains similar performance as KD(70.55% vs. 70.66%).
Instead, the reversed distillation from the student model yields signiﬁcantaccuracy gains for KD † than KD (71.76% vs. 70.66%) and DML than DML † (71.52% vs. 70.55%).
Then, we analyze the output discrepancy of the teacher-student for different methods, and the reverseddistillation reduces the gap from 1.12 of KD to 0.76 of KD † .
Therefore, the main reasons behindthe performance gap lie in two aspects: (a) Teacher models in conventional ofﬂine pipelines arenot optimized for the student model.
Thus, they could only provide general knowledge, which maybe suboptimal for the particular student.
(b) The reversed distillation changes the universal teachermodel into a student-aware one and bridges the teacher-student gap, which clearly justiﬁes:
Teachers should teach students in accordance with their aptitude and shouldn’t follow the samepattern.
For the second question, ﬁne-tuning pre-trained teacher model with reversed distillation is a straightforward way to bridge the two training fashions.
However, in most scenarios, ﬁne-tuning the wholenetwork still needs lots of training budges (more 4 × costs than KD in Table 1).
Some trade-offs inreducing the ﬁne-tuning parameters or adapter-based methods involve performance loss.
Thus, howto augment reversed distillation without much extra overhead is an important issue for the application.
Besides ﬁne-tuning, building a proxy teacher model (see Figure 2 (a)) to inherit the knowledge ofpre-trained models and receive reversed distillation from students also enjoys the same beneﬁts.
Asshown in Table 1 (H), this proxy teacher model (SHAKE (alone)) can achieve competitive gains thanﬁne-tuning the whole teacher model (KD † ).
Recent weight sharing strategies in AutoML [49, 33 ?] and can effectively save training overhead and preserve logit consistency.
This encourages us toadopt the same student architecture and weight sharing strategy to generate the proxy teacher model.
Thus, we allow this proxy teacher to share the student’s backbone but use an individual shadow headto preserve the diversity of logits representations (see Figure 2 (b)).
As shown in Table 1 (I), this
T' cat
T cat
T
T
Tcat
[Figure]
Proxy teacher
Student
Shadow head
Task head
KD
CE
Share
[Figure] [Figure]
S
T' T
S S
S cat
[Figure]
sharing strategy in SHAKE (share) presents four beneﬁts: (1) more than 3 × training accelerationthan DML and ﬁne-tuning the whole teacher model.
(2) No need for architecture selection costsfor proxy teacher.
(3) Additional accuracy gains than individual proxy teacher because knowledgeinherited by proxy teachers also directly improves the representation of students’ backbone in theweight-sharing process.
In the shared backbone setting, the reversed distillation works withadditional regularization [53] and logit consistency [29].
Based on the above observations, we propose SHA dow K nowl E dge distillation (SHAKE), a noveland effective logits distillation framework.
Our SHAKE builds a proxy teacher model and updates itsweights via the original teacher model predictions.
In this way, SHAKE enjoys the same beneﬁts withteacher ﬁne-tuning to mine the knowledge of the pre-trained model and can perform bidirectionalsupervision with student.
To optimize training costs, this proxy teacher shares the backbone with thestudents but leverages an individual head to preserve the diversity of logits representations.
This headis named shadow head since it imitates the original teacher model just like its shadow.
During training,we only need to additionally train this shadow head with small training budgets (only 1.18 × coststhan KD).
After training, all teacher models and shadow head can be deprecated, and student modelcan be separately applied for inference without any overhead.
Moreover, we extend the SHAKE tomulti-teacher model scenarios using multiple shadow heads to inherit diverse knowledge.
In principle, SHAKE alters the chain of knowledge transfer from pre-trained teacher → student in
KD to pre-trained teacher → proxy teacher (cid:29) student .
Other adaptive KD [30] only employs middlenetworks to distill as large teacher → middle teacher → student , which is not optimized for students andrequires multi-step training.
The merits of SHAKE lie in three-fold.
First, it effectively reduces theteacher-student capability gap with reversed distillation, bringing signiﬁcant gains when pre-trainedteacher models are available.
Second, for the scenario without pre-trained teacher models, SHAKEalso enables the ofﬂine KD methods more effective to alleviate the unstable optimization issuesof online KD methods.
Thus, SHAKE bridges the ofﬂine and online KD methods and enjoys theadvantages of both methods.
Third, SHAKE achieves favorable trade-offs between accuracy andtraining budget.
By contrast, other adaptive KD [30] needs to sequentially train multiple models withlots of additional training time and resources.
We hope that these intriguing observations in SHAKEwould expand the application of KD and facilitate future research for KD work to some extent.
We conduct extensive experiments on classiﬁcation and detection to verify the superiority of theproposed method.
SHAKE achieves a consistent and signiﬁcant accuracy boost in various neuralnetworks and data augmentations, which outperforms other methods by large margins.
For example,
SHAKE obtains 1 .
21% ∼ 3 .
65% accuracy gains and more than 3 × training acceleration than DMLon CIFAR-100.
On the challenging ImageNet dataset, SHAKE can improve the Top-1 accuracy of
ResNet-18 from 69.75% to 72.37% and MobileNet from 70.13% to 72.96% , which are state-of
the-art performances among KD methods.
On vision transformer architecture [44], our approachachieves 75.22% Top-1 accuracy and 2.76% gain for training a VIT-T model.
On the object detectiontask, SHAKE improves the AP by 1.20 for RetinaNet and 1.02 for Faster R-CNN on the MS-COCOdataset, demonstrating the generality of our approach.
In summary, we make the following principle contributions in this paper:
• By analyzing and exploring the difference between ofﬂine and online KD methods, weempirically show that the reversed distillation hinders the performance gain, which ﬁxesthe discrepancy between teacher-student capability.
This motivates us to propose a newSHA dow K nowl E dge distillation (SHAKE) framework to bridge two training fashions,which, to the best of our knowledge, is not achieved in the area of knowledge
distillation.• SHAKE achieves remarkable trade-offs between accuracy and training efﬁciency withan extra shadow head.
The shadow head inherits knowledge from pre-trained models,introduces reversed distillation with students, and accelerates training process by sharingthe student’s backbone.•
We perform thorough evaluations on classiﬁcation and detection.
SHAKE achieves stateof-the-art performances in multiple datasets and architectures (e
g. CNN and vision transformer).
Speciﬁcally, ResNet-18, MobileNet, and VIT-T with SHAKE achieve 72.37%,72.96% and 75.22% Top-1 accuracy on ImageNet, outperforming KD by 1.71%, 2.28% and3.02%, respectively.
Shadow Knowledge Distillation
In this section, we ﬁrst revisit the ofﬂine and online KD methods.
Then, we present the formulationof SHAKE and its expansion for multi-teacher models.
The evolution of our approach is shown in
Figure 2.2.1 Revisit
of ofﬂine and online KD methods
We ﬁrst review the formulations of ofﬂine and online KD.
For simplicity, We choose two typicalframeworks ( i.e. , original KD [17] and DML [52]) for analysis.
Given training dataset ( X, Y ) where
X = { x i } ni =1 are training samples and Y =
{ y i } ni =1 are their labels.
Let f T be the output logits ofthe ﬁxed teacher T and let f S be the output of student S , respectively.
In KD, the student network f Sis trained by minimizing:
[Equation]
where λ is a weight for balancing these two terms.
L CE is the regular cross-entropy objective:
[Equation]
where H ( · , · ) is the cross-entropy loss and σ is the softmax function.
L KL in Eq.
1 is the distillationobjective for transferring knowledge from a teacher to a student:
[Equation]
where τ is the temperature to generate soft labels and KL represents Kullback-Leibler (KL) divergence.
This distillation loss can be considered as a modiﬁed cross-entropy loss using the probabilisticoutputs of the pre-trained teacher network as the soft labels instead of the one-hot ground-truth labels.
Different from the one-way distillation in KD, DML presents a two-way knowledge transfer strategyin which the probabilistic outputs from both teacher and student networks can be used to guide thetraining of each other.
DML jointly trains the teacher and student networks in an end-to-end mannervia interleavingly optimizing two objectives:
[Equation]
where the default value of λ is 1 in DML.
2.2 Formulation of SHAKE
As shown in Figure 2 (c), SHAKE proposes a proxy teacher model with output f T (cid:48) via shadow headto augment reverse distillation in the ofﬂine framework.
During the training process, f T (cid:48) is updatedwith the output of the pre-trained teacher model f T and performs mutual distillation with the studentmodels f S as:
[Equation]
where N is the total number of teacher models.
For multiple teacher models in Figure 2 (d), we build several proxy teachers with outputs ( f T (cid:48) 1 , f T (cid:48) 2 ,..., f Ti (cid:48) ) to follow outputs ( f T 1 , f T 2 ,..., f T i ) of multiple teacher models with variousshadow heads.
The mutual distillation also exists between the task head of the student and thesemultiple shadow heads.
Similar to Eq.
5, the total optimization function for the multi-teacher scenariocan be deﬁned as:
[Equation]
(6) where N is the total number of teacher models.
Furthermore, SHAKE can be combined with advancedmulti-teacher methods ( e.g. , adaptive weights [40] for different teachers in part IV of Table 2).
[Equation]
ATKD (70.78%),
ESKD (70.89%), DML (71.13%) and our SHAKE (72.37%) for
ResNet-18 with single ResNet-34 as teacher on ImageNet.
Comparison of SHAKE with other KD methods .
Eq.
Eq. 4 and Eq. 1 clearly illustrate the difference of SHAKEand KD/DML.
Compared to KD, SHAKE introduces reverseddistillation so that the teacher can be optimized by the student.
In contrast to DML, SHAKE leverages the knowledge ofpre-trained models, resulting in additional accuracy gains.
Asshown in Figure 3, SHAKE achieves a robust boost than KDand baseline during the training process.
Meanwhile, trainingcurve of DML shows a highly dynamic oscillation due to unreliable predictions of its teacher trained from scratch.
Otheradaptive KD methods use the same knowledge transfer wayas KD and do not optimize teacher with feedback supervisionof student.
Thus, these methods, including ATKD [30] and
ESKD [7] do not essentially address the teacher-student capacity disparity, and our SHAKE surpasses them with signiﬁcantmargins ( 1 . 48% ∼ 1 .
59% on ImageNet in Figure 3)..
Experiments
In this section, we ﬁrst evaluate our approach for the classiﬁcation task on CIFAR-100/
ImageNet andthe object detection task on MS-COCO.
Then, comprehensive ablation experiments are performedto analyze the key design in our SHAKE.
As a novel ofﬂine logit approach, the main competitor of
SHAKE is the original KD [17].
Thus, we perform detailed experimental comparisons between them.
Moreover, we compare the performance with recent advanced KD methods.
For fair comparisons,we use the public codes of these approaches with the same training and data preprocessing settingsthroughout the experiments.
For SHAKE, we set λ and τ as 1 and 4, respectively.
Please refer to thesupplementary materials for more training settings.
3.1 Experiments on CIFAR-
Implementation . On
CIFAR-100 [21], we conduct experiments on various teacher-student modelsunder the same or different architecture style with CRD’s settings [43], whose training epochs are240.
We use a mini-batch size of 64 and a standard SGD optimizer with a weight decay of 0.0005.
The multi-step learning rate is initialized to 0.05, decayed by 0.1 at 150, 180, and 210 epochs.
[Table]
Comparison with ofﬂine and online KD methods .
In part I of Table 2, we compare our approachto some advanced ofﬂine/online KD methods with the same training settings.
For the same architecture style of the teacher-students, SHAKE obtains 3 .
35% ∼ 5 .
45% absolute accuracy gainsand outperforms KD with 1 .
41% ∼ 4 .
62% margins and Review with 0 .
70% ∼ 2 .
49% margins.
Besides, on cross-architecture teacher-student pairs, SHAKE achieves more signiﬁcant gains with5 .
58% ∼ 6 .
96% margins than baseline.
Compared to other KD methods, SHAKE outperforms
KD with 2 .
83% ∼ 4 .
06% margins and Review with 0 .
64% ∼ 2 .
46% margins, which illustratesthe effectiveness of SHAKE in reducing the teacher-student network gap.
Compared to DML and
AFD [8] under the same teacher-student pair in Table 2, SHAKE obtains 1 .
21% ∼ 3 .
65% relativegains.
Compare to online KDs ( e.g. , CL [42], ONE [23]) with multiple branches as teacher models,
SHAKE also obtains 0 .
70% ∼ 1 .
77% relative accuracy gains.
These signiﬁcant improvementsdemonstrate the superiority of SHAKE by leveraging the knowledge of pre-trained models.
Orthogonal to other KDs and data augmentations .
As acting on the output logits, SHAKE isorthogonal to feature and relation KD methods because of transferring different knowledge.
Asshown in Part II of Table 2, the combination of FitNets with SHAKE surpasses FitNets + KD with0 .
98% ∼ 3 .
54% margins.
Moreover, for CRD, its combination with SHAKE yields more dramaticgains than KD.
In addition, SHAKE outperforms KD (see part III of Table 2) under strong dataaugmentation.
Extension to multiple teacher models.
In the part IV of Table 2, we compare SHAKE and KD withaverage and adaptive weighting in two teacher networks.
The results illustrate that the design of
SHAKE with multi-proxy model can effectively inherit the knowledge of different teachers bringingsigniﬁcant performance gains.
3.2 Experiments on ImageNet
Implementation .
For the standard ResNet-18 [15] and MobileNet [18] models, we adopt the sametraining setting as with most distillation methods, whose training epochs are 100.
The multi-step
[Table] [Table]
learning rate is initialized to 0.1, decayed by 0.1 at 30, 60, and 90 epochs.
Recent vision transformersachieve great success on different vision tasks [9, 44].
We also extend SHAKE to VIT-T [44] with thesame training settings ( e.g. , data augmentation and distillation token) on ImageNet.
Implementationdetails are available in supplementary materials.
Comparison results .
Table 3 reports the performance of our approach on ImageNet.
SHAKEimproves the baseline models of ResNet-18 with 2 .
62% gains in Top-1 accuracy (see Figure 3 fordetailed accuracy curves) and MobileNet with 2 . 83% gains.
Compared to other KD methods, SHAKEoutperforms KD with 1 .
71% ∼ 2 .
28% margins and CRD with 1 . 20% ∼ 1 .
56% margins, whichsupports the superiority of SHAKE on the large-scale dataset.
Equipped with the distillation of twoteacher models, SHAKE ‡ obtains 2.98% gain for ResNet-18 and 3.29% gain for MobileNet thanbaseline.
As shown in Table 3, SHAKE obtains 2.76% accuracy gains than baseline and surpasses
KD under the soft or hard label of the CNN teacher (RegNet [34]), verifying its effectiveness ondifferent architectures.
3.3 Experiments on object detection
Implementation .
We evaluate SHAKE on MS-COCO dataset [26] and use the most popular opensource Detectron2 [25] as the strong baseline.
We apply SHAKE to two-stage detector ( e.g. , Faster
R-CNN [36]) and the one-stage detector ( e.g. , RetinaNet [25]), which are widely used object detectionframeworks.
Following common practice [25], all models are trained with a 2 × learning schedule(24 epochs).
For SHAKE, we build an extra shadow head with the same architecture as the originalclassiﬁcation head, which performs distillation in the detector ﬁne-tuning stage.
All distillationperformances are evaluated in Average Precision (AP).
Comparison results .
Table 4 demonstrates the experimental results of the baseline detector and ourapproach.
As shown in Table 4, our SHAKE improves the AP by 1.20 on RetinaNet and 1.02 on
Faster R-CNN, which outperform KD [17].
The performance of object detection greatly dependson the quality of deep features to locate interested objects, while logits are not capable of providingknowledge for object localization.
Thus, SHAKE naturally weaker than Review (recent state-ofthe-art feature-based KDs) and we further introduce Review to obtain satisfactory results.
It can beobserved that SHAKE can obtain new state-of-the-art results with feature-based KDs.
The success ofchallenging object detection tasks demonstrates the generality and effectiveness of our approach.
3.4 Ablation study
In this section, we isolate the impact of each component of our approach.
All experiments areconducted on the CIFAR-100 dataset.
For each setting, we run 3 times and report Top-1 meanaccuracies.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Detailed ablation study of SHAKE As shown in Table 5, an ablation study has been conductedto demonstrate the individual effectiveness of different components in SHAKE.
It is observed that(a) Reversed distillation optimizes the teacher model to adapt to the student, improving distillationefﬁciency.
SHAKE without reversed distillation only obtains marginal gains.
b
Shadow headsare effective in preserving the diversity of knowledge, and SHAKE brings accuracy reduction ifthere is no separate shadow head
Stronger more
trained knowledge when capacity sharing can accelerate theconvergence
Similar
SHAKE without also yields slight accuracygains because of more training weights and logit proxy teacher is present toinherit the pre-training knowledge with its supervision.
Under our backbone sharing settings, addingextra supervision from ground truth labels for teacher not conducive to the decoupling ofsupervision from trained teachers/labels and leads to accuracy reduction
Proxy teachers inour SHAKE to transfer the knowledge of the original pre-trained teacher model for the student modelto learn more easily.
As shown in the Table 5 (H), the extra distillation from pre-trained teachers tostudents based on SHAKE does not yield additional signiﬁcant performance gains.
(G) The originalpre-teacher model would perform ( i.e. , KD) weaker than SHAKE for direct distillation because ofthe gap with the students.
As shown in Table 5 (G), addition of pre-trained teachers’ supervision forstudent based on SHAKE does not result in additional accuracy gains.
These results demonstrate thatour SHAKE proxy teacher design can indeed serve as a good bridge between the original teacher andstudent.
Reverse KD reduces teacher-student gap to facilitate knowledge transfer.
Sensitivity study for temperature τ and weight λ .
τ in Eq.
3 controls the softness of teacher’ssupervision.
As τ increases, the output of the softmax function becomes smoother.
As shown in
Figure 4, SHAKE presents superior gains and robustness than the original KD in different τ .
λ
isthe hyper-parameter that balances the weight of KD loss.
The small λ limits distillation gains, andλ = 0 .
5 is the best option for SHAKE.
Analyzing teacher-student similarity.
The similarity between student and teacher network is animportant measure for KD tasks.
We employ KL-divergence as metrics of similarity [7], where lower
KL-divergence implies higher similarity.
Figure 5 presents the similarities and performances betweenthe outputs of ResNet with different depths as teachers ( i.e. , ResNet-110, ResNet-56, ResNet-44and ResNet-32) and the ResNet-20 as student.
The results show that the distillation from SHAKEalways gives a higher similarity than KD and DML, resulting in signiﬁcant gains under teachermodels of different depths.
Figure 6 presents the Grad-cam visualization of ResNet-18 with differentmethods on ImageNet.
The Grad-cam maps of SHAKE pay more attention to the important regions
[Figure] [Figure]
than KD and baseline, which have high similarity with teacher.
In summary, SHAKE bridges KDand DML to obtain student-friendly distillations, improving teacher-student similarity.
In addition,
Figure 7 illustrates that comparing with KD, applying SHAKE training helps learn more scatteredembeddings [31, 41, 28, 3] and logits correlation visualization in Figure 8 clearly demonstrate thatteacher model in SHAKE owns student-aware logits.
Related Work
Ofﬂine distillations .
In the ofﬂine framework [17], the teacher is pre-trained and ﬁxed, and then itssoft logits are used as extra supervision to distill student.
Although subsequent methods exploredto transfer feature knowledges [38, 51] and relation knowledges [32, 43]), the effective original KDstill outperforms most distillations and is widely used for different tasks [11, 20].
However, thepre-trained teacher model is not optimized for the student, limiting its distillation gains.
To addressthis issue, our SHAKE enhances KD using a shadow head to introduce optimization from the student.
Online distillations .
Online KD methods simplify the KD process by training all models simultaneously.
DML [52] performs bidirectional distillation for the peer networks and ONE [23] presentsan on-the-ﬂy ensemble distillation among multiple branches.
Subsequent studies focus on how tobalance multiple teacher [5] or construct the online teacher [48].
Online KD methods always obtainbetter performance than ofﬂine ones.
However, large teacher models trained from scratch sometimesperform unstable predictions and bring lots of training costs during distillation.
Therefore, we propose
SHAKE to combine the advantages of the two pipelines to facilitate application.
Adaptive distillations .
Capacity gaps between teacher-student models for their different architectureswould limit distillation gains [27].
There are two types of existing works to alleviate this gap in termsof training paradigms [10] and architectural adaptation [19, 14].
For instance, ESKD [7] proposesstopping the training of the teacher early, and ATKD [30] uses a medium-size teacher assistant toperform sequence distillation with large overheads.
However, these methods do not optimize theteacher model for the student, resulting in minor beneﬁts.
In sharp contrast to these methods, SHAKEis a student-aware ofﬂine KD method and opens a new direction for adaptive distillation design.
Conclusion
In this paper, we present SHAKE, a simple, effective, and new student-aware logits distillation tobridge ofﬂine and online knowledge transfer.
Based on our insight of online KD methods success,
SHAKE achieves this goal by building an extra shadow head as a proxy teacher model to performmutual distillation with the student model.
Thorough evaluations are performed on classiﬁcation anddetection, and SHAKE achieves signiﬁcant performance gains in various neural networks withoutextra inference overheads.
We hope that this elegant and practical approach would inspire futureresearch on knowledge distillation design and understanding.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] .
Our method will add additional
training overhead.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our
approach is a new knowledge distillation application technique, which helps lightweight
models to achieve better performance without additional teacher architectures.
It does
not bring negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[No]
(b) Did you include complete proofs of all theoretical results?
[No]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Seeimplementation details in our experiment Section 3.2 and supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 2 and more results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Table 1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’re
using/curating?
[N/A]
We evaluate our approach in a generic public dataset, following
lots of previous research works.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
The datasets we use does not contain
personally identiﬁable information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
Our approach is a generic training framework, which does not
concern human subjects.
(b) Did you describe any potential participant
risks, links to Institutional