NORM: K NOWLEDGE D ISTILLATION VIA N- TO -O NE R EPRESENTATION M ATCHING
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing feature distillation methods commonly adopt the One-to-one Representation Matching between each pre-selected teacher-student layer pair.
In this paper, we present N -to- O ne R epresentation M atching (NORM), a new two-stage knowledge distillation method, which relies on a linear Feature Transform (FT) module.
In view of preserving the intact information learnt by the teacher network, during training, our FT module consisting of two linear layers is merely inserted after the last convolutional layer of the student network.
The ﬁrst linear layer projects the student representation to a feature space having N times feature channels than the teacher representation from the last convolutional layer, and the second linear layer contracts the expanded output back to the original feature space.
By sequentially splitting the expanded student representation into N non-overlapping segments having the same number of feature channels as the teacher’s, they can be forced to approximate the intact teacher representation simultaneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair.
After training, such an FT module will be merged into the subsequent fully connected layer thanks to its linear property, introducing no extra parameters or architectural modiﬁcations to the student network at inference.
Extensive experiments on CIFAR100 and ImageNet with various teacher-student network pairs show competitive performance of NORM.
Code will be released.
I NTRODUCTION
Knowledge distillation (KD), an effective way to train compact yet accurate neural networks through knowledge transfer, has attracted increasing research attention recently.
Bucilˇa et al.
(2006) and Ba &
Caruana (2014) made early attempts in this direction.
Hinton et al.
(2015) presented the well-known
KD using a teacher-student framework, which starts with pre-training a large network (teacher), and then trains a smaller target network (student) on the same dataset by forcing it to match the logits predicted by the teacher model.
Many subsequent methods follow this two-stage KD scheme but use hidden layer features as extra knowledge, while others use a one-stage KD scheme in which teacher and student networks are trained from scratch jointly (Guo et al., 2021).
In this paper, we focus on two-stage feature distillation (FD) research for supervised image classiﬁcation tasks.
Existing two-stage FD methods primarily use feature maps (Romero et al., 2015), or attention maps (Zagoruyko & Komodakis, 2017), or other forms of features (Chen et al., 2021a) at one or multiple hidden layers as knowledge representations.
Generally, modern neural network architectures engineered on the ImageNet classiﬁcation dataset (Russakovsky et al., 2015) adopt a multi-stage design paradigm.
At a pair of the same staged layers, a teacher network typically has more output feature channels than a student network while keeping the same spatial feature size.
All feature dimensions may be different at a cross-stage layer pair.
To align the feature dimensions, there have been many feature transform (FT) designs.
However, prevailing teacher FT designs cause information loss due to dimension reduction, as studied in (Heo et al., 2019a; Tian et al., 2020).
More importantly, we observe that existing FD methods adopt the One-to-one Representation Matching (ORM) between each pre-selected teacher-student layer pair, indicating that only one knowledge transfer route is introduced.
We argue that this leaves considerable room to promote two-stage FD research.
Driven by the above analysis, in this paper, we present a new two-stage feature distillation method dubbed N -to- O ne R epresentation M atching (NORM) that relies on a simple FT module consisting
𝐶 𝑚 𝐶 𝑚 (3) 𝐶 𝑠 𝐶 𝑡 𝑇 𝑡 𝑑(·) 𝐹 𝑠 𝐹 𝑡 𝑇 𝑠 or or 𝐶 𝑡 (1) 𝐶 𝑡 𝐶 𝑠 𝑑(·) 𝑇 𝑠 𝐹 𝑠 𝐹 𝑡 𝐶 𝑡 (2) 𝐶 𝑠 𝑑(·) 𝐶 𝑠
𝑇 𝑡 𝐹 𝑠 𝐹 𝑡
Student network
Teacher network (Pre-trained and fixed) G A P 𝑌 GA P 𝑊 𝑠𝑒
Teacher network (Pre-trained and fixed)
𝐹 𝑠 𝐶 𝑠 𝑁×𝐶 𝑡 𝐹 𝑠𝑒 𝐶 𝑡 𝐶 𝑡 𝐶 𝑡 𝐹 𝑡
𝑑
(·) 𝑊 𝑓𝑐 𝑊 𝑓𝑐 =
𝑊 𝑓𝑐 (𝑊 𝑠𝑐 𝑊 𝑠𝑒 +𝐼) 𝑌 𝐶 𝑡 𝐶 𝑡
Training
Inference with merging …
+ 𝑊 𝑠𝑐
Feature transform
Student network GA P G A P of two linear layers.
An architectural comparison of popular ORM schemes and NORM is depicted in Figure 1.
When formulating NORM, we leverage three basic principles: (1) using as fewer FTs as possible; (2) enabling many-to-one feature mimicking ﬂow via student representation expansion and splitting; (3) making FT module absorbable.
With the ﬁrst principle, our FT module is merely inserted after the last convolutional layer of the student network.
In this way, the intact information learnt by the teacher network is preserved, and knowledge transfer ﬂow only needs to be considered between the last convolutional layer pair.
With the second principle, our FT module starts with a linear layer that projects the student representation to a feature space having N times feature channels than the teacher representation.
This allows NORM to introduce many parallel knowledge transfer routes between a single teacher-student layer pair via simple student feature splitting and group-wise feature mimicking operations.
With the third principle, our FT module ends with another linear layer that projects the expanded student representation back to the original feature space, and it does not contain any non-linear activation functions, making all of its operations linear.
As a result, after training the FT module can be directly merged into its subsequent fully connected layer, without introducing any extra parameters or architectural modiﬁcations to the student network at inference.
We evaluate NORM on popular image classiﬁcation benchmarks, achieving competitive results.
On the CIFAR-100 dataset (Krizhevsky & Hinton, 2009), the student models trained by NORM show a mean improvement of 2 .
88% over 7 teacher-student pairs of the same typed network architectures.
Over 6 teacher-student pairs of different typed network architectures, the mean improvement of
NORM even reaches 5 .
81% , and the maximal gain is 6 .
Leading results are obtained on the large-scale ImageNet dataset.
With NORM, the ResNet18 | MobileNet | ResNet50-1/4 model reaches 72 .
14% | 74 .
26% | 68 .
03% top-1 accuracy when using a pre-trained ResNet34 | ResNet50 | ResNet50 as the teacher model, showing 2 .
01% | 4 .
63% | 3 .
03% absolute gain compared to the standard training scheme, without heavy extra training cost.
Thanks to its simplicity and compatibility, the performance of NORM could be further improved when combining it with other popular distillation strategies like logits based supervision (Hinton et al., 2015) and contrastive learning (Tian et al., 2020).
R ELATED WORK
Two-stage KD methods.
This category of KD methods ﬁrst assumes that a pre-trained teacher network is available, and then uses its learnt representation as extra supervision to guide the training of a student network.
The vanilla KD (Hinton et al., 2015) uses the logits output from the teacher network as soft supervision.
FitNets (Romero et al., 2015) show that the feature maps from hidden layers can also be used as hints to improve distillation performance.
AT (Zagoruyko & Komodakis, 2017) proposes to use spatial attention maps instead of source feature maps.
Many follow-up methods intend to improve the knowledge representation via different techniques, such as feature encoding (Yim et al., 2017; Chen et al., 2021a; Srinivas & Fleuret, 2018), feature selection (Heo et al.,
2019a;b; Chen et al., 2021b), distribution learning (Ahn et al., 2019; Huang & Wang, 2017; Passalis & Tefas, 2018;
Malinin et al., 2020; Yang et al., 2021), attention rephrasing (Kim et al., 2018; Ji et al., 2021), and reuse of teacher classiﬁer (Chen et al., 2022).
Besides, some works (Peng et al., 2019;
Yin et al., 2020; Park et al., 2019; Tung & Mori, 2019) explore the use of sample relations.
It is also worth noting that several recent works (Yue et al., 2020; Liu et al., 2020b; Deng et al., 2022) propose reinforcement learning based searching methods to improve feature distillation process.
One-stage KD methods.
This category of KD methods adopts an online training framework that does not require to pre-train the teacher model.
ONE (Lan et al., 2018) presents a multi-branch training strategy using an on-the-ﬂy branch ensemble to guide the training of individual branches.
DML (Zhang et al., 2018b) uses a mutual learning strategy to jointly train a set of peer models from scratch.
During training, every peer model acts as a teacher of the others.
DCM (Yao & Sun, 2020) shows that the logits from auxiliary branches of teacher and student networks can improve mutual distillation performance.
Many one-stage KD variants have been presented recently, including but not limited to (Guo et al., 2020a; Wu & Gong, 2021; Chung et al., 2020;
Malinin et al., 2020).
Other KD variants.
Extending knowledge distillation methodology from standard supervised learning to other learning scenarios has gained broad attention.
BANs (Furlanello et al., 2018) improve the training of a target network via a progressive self-distillation formulation.
CRD (Tian et al., 2020) formulates an effective structural representation matching loss based on contrastive learning.
Xu et al.
(2020) further presented a way to combine self-supervision and contrastive learning.
Besides, there also exist various KD variants for life-long learning (Li & Hoiem, 2016), semi-supervised learning (Pham et al., 2021), few-shot learning (Shen et al., 2021), data-free learning (Lopes et al., 2017), adversarial learning (Chung et al., 2020) and distributed learning (Anil et al., 2018).
Although these methods are still tailored to image classiﬁcation, KD has also been applied to handle other tasks, such as semantic segmentation (Liu et al., 2019b), object detection (Wang et al., 2019), image super-resolution (Liu et al., 2020a), and neural machine translation (Kim & Rush, 2016).
This work focuses on two-stage KD research for supervised image classiﬁcation tasks.
Speciﬁcally, we attempt to shift the prevailing one-to-one feature representation matching paradigm to a many-toone alternative conditioned on a single teacher-student layer pair.
This makes our method differ with existing KD methods both in motivation and formulation.
M ETHOD 3. B ACKGROUND : ONE - TO - ONE REPRESENTATION MATCHING
We ﬁrst review the representation matching of existing two-stage feature distillation (FD) methods in a general formulation.
Suppose that we have a pre-trained teacher network T , a target student network S , a given image classiﬁcation dataset X and its ground truth label set Y .
For a pre-selected teacher-student layer pair (the same staged layers of T and S , as default), let F t ∈ R H × W × C t | F
s ∈ R
H
× W × C s denote the output feature maps for the teacher
| student network, where H , W and C t | C s are the channel height, width and number, respectively.
Let T t | T s denote the feature transform (FT) for the teacher
| student network, which projects F t | F s to the same feature space, respectively.
Then, the representation matching loss of existing FD methods, in general, can be deﬁned as:
[Equation]
Various distance metrics d ( · ) , such as l 2 -norm distance (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Heo et al., 2019b), l 1 -norm distance (Kim et al., 2018) and maximum mean discrepancy (Huang & Wang, 2017) are popularly used in FD research.
Furthermore, existing FD methods usually apply the feature representation matching to one or multiple pre-selected teacher-student layer pairs.
Under this context, it is clear that the use of FT and the design of FT are two key problems for the feature representation matching.
To the ﬁrst problem, there are three choices: applying one or multiple FTs to (1) the student network (Romero et al., 2015); (2) the teacher network (Yue et al., 2020); (3) both teacher and student networks (Zagoruyko & Komodakis, 2017; Heo et al., 2019b;
Srinivas & Fleuret, 2018; Heo et al., 2019a; Chen et al., 2021a).
They are illustrated in Figure(left ﬁgure).
Comparatively, the last choice is much more popular than the other two.
To the second problem, there exist many FT designs (Guo et al., 2021) that generate aligned feature maps, or attention maps, or other forms of features used as the knowledge representation for matching.
However, according to Eq. 1, we can see that existing FD methods use the One-to-one Representation Matching
(ORM).
More speciﬁcally, they perform global single-shot feature mimicking process, meaning that there exists only one knowledge transfer route between any teacher-student layer pair.
We conjecture that if we can formulate an effective mechanism to introduce multiple global knowledge transfer routes between any teacher-student layer pair, it will offer more chances to inject the intact teacher’s knowledge into the student network, and improved FD performance could be attained.
3. N- TO - ONE REPRESENTATION MATCHING .
Motivated by the above analysis, we present N -to- O ne R epresentation M atching (NORM), a new two-stage feature distillation method that relies on a simple FT module.
In the formulation of NORM, we rethink the representation matching mechanism from the following perspectives: (1) regarding the use of FT, where to place it?
(2) regarding the design of FT, how to adapt it for introducing many parallel global knowledge transfer routes between any pre-selected teacher-student layer pair?
(3) regarding the distillation performance, how to make the accuracy-efﬁciency tradeoff?
To the ﬁrst and third questions, our basic principle is “using as fewer FTs as possible” , partially inspired by (Romero et al., 2015; Heo et al., 2019a; Tian et al., 2020).
Accordingly, we merely insert a FT module after the last convolutional layer of the student network.
In this way, NORM preserves the intact information learnt by the teacher network, and knowledge transfer ﬂow only needs to be considered between the last convolutional layer pair.
Note that the design of directly plugging a FT module into the student network is in sharp contrast to existing FD methods which typically use their FTs as auxiliary branches that will be discarded at inference.
To the second question, our basic principle is “enabling many-to-one knowledge mimicking ﬂow via student representation expansion and splitting” .
Accordingly, we construct a student FT module which starts with a linear layer that projects the student representation to a feature space having N times feature channels than the teacher representation.
By sequentially splitting the expanded student representation into N non-overlapping segments having the same number of feature channels as the teacher’s, we can force them to approximate the intact teacher representation simultaneously, formulating an effective many-to-one feature matching mechanism conditioned on a single teacher-student layer pair.
To the third question, we additionally use a “making FT module absorbable” principle.
Accordingly, our student FT module ends with another linear layer that projects the expanded student representation back to the original feature space, and it does not contain any non-linear activation functions.
As a result, such an FT module will be merged into its subsequent fully connected layer after training, without introducing any extra parameters or architectural modiﬁcations to the student network at inference.
An overview of NORM is depicted in Figure 1 (right ﬁgure).
Now, we provide the formulation of NORM following the notations in Eq.
Given a teacher-student layer pair (last convolutional layer pair, as default) with the output feature maps F t ∈ R H × W × C t and F s ∈ R
H
× W × C s , let T s ( W se , W sc ) denote our student FT module consisting of two linear layers.
The ﬁrst linear layer uses a 1 × 1 convolutional kernel W se ∈ R 1 × 1 × C s × NC t along the channel dimension to project each pixel in F s to a desired channel dimension NC t , producing an expanded student representation F
se ∈ R
H
× W × NC t having N times feature channels than the teacher representation.
The second linear layer uses another 1 × 1 convolutional kernel W sc ∈ R 1 × 1 × NC t × C s to project each pixel in F se back to the original channel dimension C s , producing F sc ∈ R H × W × C s .
Mathematically, the student FT module with the input F s can be written as:
[Equation]
where ∗ denotes the convolution operation.
Next, we sequentially split the expanded student representation F se into N non-overlapping segments F ise ∈
R H × W × C t , 1 ≤ i ≤ N having the same number of feature channels as the teacher’s.
This allows us to force N student feature segments to approximate the intact teacher representation simultaneously by minimizing an l 2 -norm distance metric, formulating our many-to-one representation matching mechanism conditioned on a single teacher-student layer pair.
Speciﬁcally, our many-to-one representation matching loss is deﬁned as:
[Equation]
and the total training loss of NORM to be minimized is deﬁned as:
[Equation]
where L ce denotes the standard cross-entropy loss of the student network supervised by the ground truth labels, and α is a positive coefﬁcients to weight the feature distillation loss L norm .
Interpretation of NORM.
The formulation of NORM can be interpreted as a novel way of learning a dynamically mixed feature ensemble over multiple augmented views of the same student representation by forcing them to mimic the intact teacher representation simultaneously.
More precisely, in NORM, the ﬁrst linear layer W se acts as a set of independently initialized feature transforms to generate N channel-expanded views of the student feature F s whose representation abilities are then parallelly augmented by the distillation supervision from the intact teacher feature F t , and the second linear layer W sc performs a learnable ensemble of N distillation-augmented student feature views via fully connected channel mixing operations (which make the feature ensemble F sc has the same size to F s , guaranteeing the absorbable property of NORM at inference).
Our mixed student feature ensemble learning further beneﬁts from the standard cross-entropy loss supervised by the ground truth labels.
In order to understand what enables the distillation effectiveness, we systematically study the major components of NORM in the Experiments section.
Augmented optimization of NORM.
Thanks to its simplicity, NORM can be readily enhanced by additionally introducing a popular logits based distillation loss (Hinton et al., 2015) into Eq. 4:
[Equation]
where L logits is the cross-entropy loss which matches the logits of the student network to the target logits produced by the pre-trained teacher network, and β is a positive coefﬁcient. Typically, for each training sample, the logits of either teacher or student network are the class probabilities from the softmax layer softened with a temperature of T .
Besides, NORM can be also combined with a recent contrastive distillation loss (Tian et al., 2020) for improved performance, as we tested in experiments.
Implementation and Inference.
As our FT module does not have any non-linear activation functions, we empirically ﬁnd that inserting it after the last convolutional layer of different student networks will mostly lead to obvious accuracy drop in the standard training regime, but with NORM the performance of ﬁnal student models will be improved signiﬁcantly.
To suppress the accuracy drop issue, we add a linear residual connection (identity mapping, as shown in Figure 1) from the input to the output of our student FT module, while maintaining the absorbable property.
In implementation, we use this student FT module as our default setting to NORM, and set N = 8 for all main experiments (the choice of N is studied in Figure 2).
For inference, let W fc ∈ R C s × C fc denote the learnt parameters of the fully connected layer after the student FT module T s ( W se , W sc ) , we can directly merge the student FT module into its subsequent fully connect layer by W fc =
W fc ( W sc W se + I ) , where
I ∈ R C s × C s is an identity matrix.
Note that modern neural networks typically have a global average pooling layer before the fully connected layer, and it does not affect the absorbable property of our student FT module at inference.
We put the proofs of them in the Appendix .
Differences with network re-parameterization methods.
The absorbable property of our student
FT module is based on the network re-parameterization.
In recent years, there have been lots of network re-parameterization methods that are presented in the deep learning ﬁeld.
ACNet (Ding et al., 2019) uses an absorbable multi-branch block based on 1D asymmetric convolutions to replace the square convolution.
RepVGG (Ding et al., 2021) presents an absorbable multi-branch block to replace a stack of 3 × 3 convolutions in VGG-like networks (Simonyan & Zisserman, 2015).
This category of methods focuses on strengthening the learning power of basic convolutions via equivalent multibranch alternatives, particularly from the perspective of network structure engineering in the standard training regime.
Clearly, our method differs with them both in focus, formulation and application.
ExpandNets (Guo et al., 2020b) and WIN (Zhou et al., 2020) leverage over-parameterization designs based on linear expansion and contraction to improve the training of a thin network, which are more closely related to our work.
ExpandNets expand all convolutional and fully connected layers during training, and convert the expanded network back to the original one at inference.
WIN ﬁrst trains a wider network generated by uniformly expanding the width of all building blocks in a given thin network, and uses it as the teacher.
Then, it inserts a pair of linear expansion and contraction layers between any two neighboring building blocks of the given thin network, and considers the sequential stacks of expanded building blocks as a set of subnetworks, which are trained progressively one by one using one-to-one feature representation matching.
ExpandNets and WIN all adopt the vanilla
KD (Hinton et al., 2015) to further ﬁne-tune their trained models.
In distinct contrast to them, our method aims to advance two-stage feature distillation research by presenting a novel many-to-one representation matching strategy conditioned on a single teacher-student layer pair. Accordingly, our [Table] [Table]
68 CC 73.
[Table] [Table] 71 VID 74. [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
48 PKT 74.
[Table]
[Table]
88 AB 72.
[Table]
[Table]
[Table]
[Table]
94 FT
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
FSP
[Table]
n
[Table]
a
[Table]
[Table]
[Table]
[Table]
[Table]
23 NST
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
53 CRD
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
94 SRRL n/
[Table]
[Table]
n/a
[Table]
[Table]
[Table]
[Table]
SemCKD n/
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
43 ReviewKD 76.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
84 SimKD n/a 75.
n/a n/a n/a
[Table]
[Table]
[Table]
n/a
[Table]
n/a 73.
n/a n/a NORM(w/1plainFT) 75.
[Table]
[Table]
[Table]
[Table]
64 NORM 75.
[Table]
[Table]
+
[Table]
[Table]
method inserts only one linear FT module to the last convolutional layer of a student network, without need of multi-layer feature mimicking and progressive training with multiple restarts.
Furthermore, our method is applicable to various teacher-student pairs with both the same type and different type network architectures.
Comparatively, our method is a more simple and easy to use, and it (w/o the vanilla KD) outperforms ExpandNets and WIN with large margins on ImageNet (see Table 6).
E XPERIMENTS 4. M AIN EXPERIMENTS
Datasets and experimental setups.
Our experiments are performed on CIFAR-100 (Krizhevsky &
Hinton, 2009) and ImageNet (Russakovsky et al., 2015) datasets.
CIFAR-100, which consists of 50,000 training images and 10,000 test images with 100 classes, is a popular classiﬁcation dataset for KD research.
Following the settings of CRD (Tian et al., 2020), we use 13 teacher-student pairs having either the same type or different type network architectures (see Table 1,2) for experiments.
Each experiment is conducted for 5 separate runs, and we report top-1 mean recognition rate on the test set.
ImageNet contains over 1.2 million images for training and 50,000 images for validation, including 1,000 image classes.
Comparatively, ImageNet is much more challenging than CIFAR-100.
Following the settings of (Tian et al., 2020; Yang et al., 2021; Chen et al., 2021b), we use 2 popular teacher-student pairs (see Table 3) for experiments.
We report top-1 recognition rate on the validation set.
For comprehensive comparisons, we compare our method with existing mainstream two-stage and one-stage KD methods, including the vanilla KD (Hinton et al., 2015), FitNets (Romero et al., 2015), AT (Zagoruyko & Komodakis, 2017), SP (Tung & Mori,
Results comparison on CIFAR-100.
Table 1 shows the results comparison on CIFAR-100 withteacher-student pairs having the same type network architectures.
In average, NORM brings 2 .
88% top-1 gain to the baseline student models, with the maximal gain of 3 . 99% .
Table 2 further provides [Table] [Table] the results comparison with 6 teacher-student pairs having different type network architectures.
In average, NORM brings 5 .
81% top-1 gain to the baseline student models, with the maximal gain of 6 . 92% .
Generally, NORM shows competitive performance compared to these state of the art
KD methods which usually adopt multi-layer feature distillation schemes.
Note that many of these competing KD methods also use the vanilla logits based distillation (Hinton et al., 2015) as the extra supervision during training.
Besides, CRD, SRRL and SSKD utilize the contrastive learning to augment knowledge distillation process.
In Table 1 and Table 2, we also explore the compatibility of NORM with these two types of distillation regularization.
We can see that: (1) the vanilla logits based distillation can further improve the performance of NORM (denoted as NORM+KD), showing 0 .
46% extra gain in average, with the maximal extra gain of 0 .
61% ; (2) the contrastive learning can also improve the performance of NORM (denoted as NORM+CL) in most cases, showing 0 .
18% extra gain in average, with the maximal extra gain of 0 .
Finally, NORM+KD achieves the best and the second best results on 7 and 4 of 13 teacher-student pairs, respectively.
Results comparison on ImageNet.
Table 3 shows the results comparison on ImageNet.
We can see that our NORM always shows the best performance compared to state of the art KD methods including both two-stage and one-stage families.
Speciﬁcally, for the teacher-student pair of
ResNet34 → ResNet18, NORM improves top-1 accuracy of the student model from 70 .
13% to 72 .
14% with an absolute gain of 2 .
01% , outperforming the current best performing method DistPro by a margin of 0 .
Taking a pre-trained ResNet50 as the teacher model and a MobileNet (Howard et al., 2017) as the target student, the MobileNet model trained by NORM attains 74 .
26% top-1 accuracy, showing 4 .
63% gain to the baseline trained individually.
Compared to ResNet34 → ResNet18, the capacity gap of the teacher and student network becomes larger for ResNet50 → MobileNet.
Under this context, it can be observed that the top-1 margin of NORM against the current best performing method DistPro is pronounced, reaching 1 . 00% .
4. A BLATION STUDY
To have a deep analysis of NORM, we further provide a lot of ablative experiments mostly performed on CIFAR-100, unless otherwise stated.
For the experiments on CIFAR-100, we run our methodtimes for each setting with random initialized seeds, and report top-1 mean recognition rate.
On ImageNet with ResNet34 → ResNet18, we insert 2 FT modules ( N = 8 ) after conv5 x (our default design) and conv4 x of ResNet18, and compare the following two settings: (1) using
NORM to the same staged teacher-student layer pairs; (2) forcing the student representations expanded from conv4 x and conv5 x to match the teacher representation learnt from conv5 x. Plain
FT does not have a linear residual connection.
[Figure] The selection of N and α .
In the formulation, NORM has two hyper-parameters: N to control the desired number of representation matching routes between a single teacher-student layer pair and α to weight the many-to-one representation loss L norm .
Accordingly, our ﬁrst two sets of ablative experiments on CIFAR-100 are conducted for the selection of N and α .
Speciﬁcally, we use a teacher-student pair of ResNet32x4 → ResNet8x4.
Figure 2 and Figure 3 show the performance comparison of NORM with different settings of N and α , respectively.
From Figure 2, we can observe that: (1) increasing N from 1 to 2 brings 2 .
28% extra accuracy gain; (2) this gain becomes larger when N gradually increases, and reaches the ﬁrst peak value at N = 8 ( 3 . 02% to N = 1 and 3 .
62% to the baseline student model).
To balance training accuracy and efﬁciency, we set N = 8 for all other experiments both on CIFAR-100 and ImageNet.
According to Figure 3, we set α = 10 for all other experiments on CIFAR-100.
For the experiments on ImageNet, we empirically set α = 8 .
The role of the many-to-one representation matching.
In NORM, the many-to-one representation matching is performed after sequentially splitting the expanded student representation into N nonoverlapping segments having the same number of feature channels as the teacher’s.
Given N student segments, is it necessary to force them to simultaneously approximate the teacher representation?
Again, we use the teacher-student pair of ResNet32x4 → ResNet8x4 with N = 8 to explore this question.
From Figure 4, we can see: the more the segments used to match the teacher representation, the larger the accuracy gain.
Speciﬁcally, using all 8 segments outperforms using only 1 of 8 segments by a margin of 0 . 98% .
Moreover, using 1 of 8 segments for representation matching ( 75 . 51% vs.
47% ) is much better than N = 1 in Figure 2.
These results validate the advantage of our design.
Applying NORM to more teacher-student layer pairs.
Recall that NORM performs the proposed many-to-one representation matching between a single teacher-student layer pair, as our FT module is merely inserted after the last convolutional layer of the student network.
A natural question is how about the performance when applying NORM to multiple teacher-student layer pairs.
We study this question on ImageNet with ResNet34 as teacher and ResNet18 as student.
In the experiments, we add two FT modules ( N = 8 ) after conv5 x (our default design) and conv4 x of ResNet18, and consider the following two settings: (1) applying NORM to the same staged teacher-student layer pairs simultaneously; (2) forcing two different staged student representations to simultaneously match the teacher representation from conv5 x.
Architectural illustrations of these two NORM variants are referred to the Appendix . From the results shown in Table 4 we can see that these two variant [Table]
The results of ExpandNets and Win on ImageNet are obtained from their original papers (Guo et al., [Table] designs lead to worse results than our default design no matter the FT module has a linear residual connection or not, although both of them improve the student accuracy.
At ﬁrst blush, this might seem surprising, since multi-layer feature matching strategies are widely used in KD research (Zagoruyko & Komodakis, 2017;
Yim et al., 2017; Heo et al., 2019b; Srinivas & Fleuret, 2018; Heo et al., 2019a;
Chen et al., 2021a).
This is due to our linear FT modules, as explained next.
The effect of linear FT modules.
Someone may concern that the model accuracy improvement may mainly from inserting a linear FT module into each student network.
We conduct ablative experiments on ImageNet to explore this concern.
In the experiments, we add 1 or 2 linear FT modules into
ResNet18 (after conv4 x and conv5 x) ﬁrst, then train each of them from scratch individually.
We consider the linear FT module without or with a linear residual connection, separately.
Table 5 shows the results.
We can observe that adding one linear FT module without a linear residual connection into ResNet18 leads to obvious accuracy drop (1.03 % ), and the accuracy drop becomes more serious when adding two modules.
The accuracy drop issue is suppressed by our default FT module having a linear residual connection which only brings marginal model accuracy improvement.
Similarly, adding two default FT modules into ResNet18 shows slightly worse model accuracy compared to just adding one.
Furthermore, in Table 1 and Table 2, we provide the individually trained baseline results of adding our two types of FT modules separately to many different student networks on
CIFAR-100, and similar observations can be found.
These experimental observations are mainly due to the structure of our FT modules.
Note that our FT design does not contain popular non-linear activation functions which is critical to stabilize and improve the training process, in order to gain the absorbable property for maintaining efﬁcient inference.
Because of this, applying NORM to more teacher-student layer pairs does not achieve further improved performance compared to NORM conditioned on the last convolutional pair, as shown in Table 4.
Performance comparison with network re-parameterization methods.
In Section 3.2, we discuss the connections and differences of NORM and network re-parameterization methods.
In Table 6, we further compare their performance on ImageNet.
Clearly, our method obtains more accurate student models than network re-parameterization based methods that also apply knowledge distillation during training.
Speciﬁcally, for the MobileNet (student), NORM without KD reaches 74 .
26% topaccuracy with a pre-trained ResNet50 as teacher, while ExpandNets (Guo et al., 2020b) with KD gets 70 .
47% top-1 accuracy using a pre-trained ResNet152 as teacher; for the ResNet50-1/4 (student),
NORM without KD reaches 68 .
03% top-1 accuracy with a pre-trained ResNet50 (teacher),
while
WIN (Zhou et al., 2020) with KD gets 67 .
50% top-1 accuracy using a pre-trained wide teacher which is 4 times larger than the student.
More experiments and discussions.
Please note that in the Appendix , we provide more ablative experiments, for a better understanding of NORM.
The limitations of NORM are also discussed.
C ONCLUSION
In this paper, we present NORM, a new two-stage feature distillation method.
It relies on a linear feature transform module that is merely inserted after the last convolutional layer of the student network to enable a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair via feature expansion, splitting and group-wise mimicking.
Thanks to its linear property, after training such a feature transform module will be merged into the subsequent fully connected layer, maintaining the same student network architecture at inference.
Extensive experiments on popular image recognition benchmarks show that NORM can attain promising performance in both distillation accuracy and efﬁciency.
We hope NORM would inspire the community to pay more attention to many-to-one representation matching research.
A A PPENDIX A. P ROOFS OF THE ABSORBABLE PROPERTY OF OUR FEATURE TRANSFORM MODULES
Here we provide the proofs to show that our feature transform (FT) module without or with a linear residual connection (identity mapping) could be merged into the subsequent fully connect layer of the student network after training.
Given a student network S , let T s ( W se , W sc ) denote a FT module (without a linear residual connection) inserted after the last convolutional layer of S , let W fc denote the weight matrix of the subsequent fully connected layer, let
F ∈ R
H
× W × C s denote the input feature maps of T s , where H ,
W and C s are the channel height, width and number, respectively.
For inference, the mathematical operations from the FT module to the subsequent fully connected layer can be deﬁned as:
[Equation]
where ∗ denotes the convolution operation, W se ∈ R 1 × 1 × C s × NC t , W sc ∈ R 1 ×
1 × NC t × C s denote two point-wise convolutional kernels for channel expansion and contraction, respectively; W fc ∈ R C s × C fc in the fully connected layer can be also seen as a point-wise convolutional kernel; GAP denotes global average pooling operation; F (cid:48)
∈
R H × W × C s denotes the output feature maps of the
FT module; O ∈ R C fc denotes the output vector of the fully connect layer.
We use F i,j ∈ R C s to denote the vector of channel pixels at spatial coordinate ( i, j ) of F .
For the point-wise convolutional kernel W se , we have:
[Equation]
Therefore, the vector of channel pixels at spatial coordinate ( i, j ) of F (cid:48) can be calculated as:
[Equation]
then we can write:
[Equation]
where W (cid:48) fc =
W fc W sc W se .
Now, it is clear that we can directly merge the FT module (without a linear residual connection) into its subsequent fully connect layer of the student network at inference.
For the FT module with a linear residual connection, the calculation of F (cid:48) is deﬁned as:
[Equation]
where I ∈ R C s × C s is an identity matrix.
Then we have:
[Equation]
where W (cid:48) fc =
W fc ( W sc W se + I ) .
Again, it is clear that we can directly merge the FT module (with a linear residual connection) into its subsequent fully connect layer of the student network at inference.
A. D ATASETS AND IMPLEMENTATION DETAILS
Recall that our main experiments are conducted on two image classiﬁcation datasets: CIFAR100 (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015).
This section provides the experimental details.
A.2. I MAGE CLASSIFICATION ON
CIFAR-
CIFAR-100, which consists of 50,000 training images and 10,000 test images with 100 classes, is a popular classiﬁcation dataset for KD research.
Following the settings of CRD (Tian et al., 2020), we use 13 teacher-student pairs having either the same type or different type network architectures (see Table 1,2 in the main manuscript) for experiments.
Each experiment is conducted for 5 separate runs, and we report top-1 mean recognition rate on the test set.
For fair comparisons, we use the same training settings to conduct the experiments with our method, following CRD.
Speciﬁcally, for each teacher-student pair, the model is trained by the stochastic gradient descent (SGD) optimizer for 240 epochs, with a batch size of 64, a weight decay of 0.0005 and a momentum of 0.9.
The initial learning rate is set to 0.1 and decreased by a factor of 10 at epoch 150, 180 and 210.
All models are trained on an Intel Xeon Silver 4214R CPU server using a single GPU (NVIDIA
GeForce RTX 3090).
A.2. I MAGE CLASSIFICATION ON I MAGE N ET
ImageNet is much more challenging than CIFAR-100, which contains over 1.2 million images for training and 50,000 images for validation, including 1,000 image classes.
Following the settings of (Tian et al., 2020; Yang et al., 2021; Chen et al., 2021b), we use 2 popular teacher-student pairs (see
Table 3 in the main manuscript), namely ResNet34 → ResNet18 and ResNet50 → MobileNet Howard et al.
(2017), for experiments.
For fair comparisons, we adopt the standard data augmentation to train and evaluate each network.
For training, we ﬁrst resize the input images to 256 × 256 , then randomly sample 224 × 224 image crops or their horizontal ﬂips.
We standardize the cropped images with mean and variance per channel.
For evaluation, we use the center crops of the resized images, and report top-1 recognition rate on the ImageNet validation set.
Training setup for ResNet34 → ResNet18.
The model is trained by SGD optimizer for 100 epochs, with a batch size of 256, a weight decay of 0.0001 and a momentum of 0.9.
The initial learning rate is set to 0.1 and decreased by a factor of 10 every 30 epochs.
Training setup for ResNet50 → MobileNet.
The model is trained by SGD optimizer for 100 epochs, with a batch size of 256, a weight decay of 0.0001 and a momentum of 0.9.
The initial learning rate is set to 0.1 and and scheduled to arrive at zero with a cosine decaying strategy.
All models are trained on an Intel Xeon Silver 4214R CPU server with 2 NVIDIA GeForce RTX 3090 GPUs.
A. A PPLYING NORM TO MORE TEACHER - STUDENT LAYER PAIRS
Since our FT module is merely inserted after the last convolutional layer of a student network,
NORM performs the proposed many-to-one representation matching between a single teacher-student layer pair.
A natural question is how about the performance when applying NORM to multiple teacher-student layer pairs.
In the main manuscript [Figure] we provide a set of ablative experiments to explore this question on ImageNet with ResNet34 as teacher and ResNet18 as student (see Tablein the main manuscript). In the experiments
[Figure] we add two linear student FT modules ( N = 8 ) after conv4 x and conv5 x (our current design) of ResNet18, and consider the following two settings: (1) using the same staged teacher-student representations for many-to-one matching simultaneously; (2) forcing two different staged student representations to match the teacher representation from conv5 x. For the teacher-student layer pair of conv5 x → conv4 x [Figure] we apply a 2 × 2 average pooling with a stride of 2 to the expanded student representation in order to match the spatial dimension of the teacher representation.
Figure 5 shows an architectural overview of these two NORM variants.
[Figure]
and consider the following two settings: (1) applying NORM to two same staged teacher-student layer pairs (after conv4 x and conv5 x) simultaneously (the ﬁgure in the ﬁrst row); (2) forcing the student representations expanded from conv4 x and conv5 x to match the teacher representation from conv5 x
(the ﬁgure in the second row).
For the teacher-student layer pair of conv5 x → conv4 x, we apply a 2 × 2 average pooling with a stride of 2 to the expanded student representation in order to match the spatial dimension of the teacher representation.
A. C ONTRACTING TEACHER REPRESENTATION .
In current formulation, NORM enables the many-to-one representation matching via expanding the student representation to have N times feature channels than the teacher’s.
Naturally, a reversed way for enabling the many-to-one representation matching is to contract the teacher representation to
On ImageNet with a teacher-student pair ResNet34 → ResNet18, we compare NORM ( N = 8 and α = 8 ) with a reversed NORM variant which enables the many-to-one representation matching via contracting the teacher representation to have 1 /N feature channels than the student’s.
[Table] [Table]
have 1 /N times feature channels than the student’s.
In principle, such a reversed NORM variant is more efﬁcient than NORM.
However its performance is limited by the information missing caused by dimension reduction to the teacher representation.
To validate this problem, we perform a set of ablative experiments on ImageNet with ResNet34 as teacher and ResNet18 as student.
We compare NORM (using a linear FT module either with or without a linear residual connection) and the reversed NORM (using a popular FT consisting of one 1 × 1 convolutional layer). Table 7 shows detailed results, from which we ﬁnd the reversed NORM can also improve the baseline student but performs obviously worse than NORM. Speciﬁcally, NORM outperforms the reversed NORM by a top-1 margin of 1 . 17% and 1 .
32% for the FT module without and with a linear residual connection, respectively.
A. P ERFORMANCE COMPARISON WITH NETWORK RE - PARAMETERIZATION METHODS
In Table 6 of the main manuscript, we compare the performance of NORM with two network reparameterization methods that also apply knowledge distillation during training, on the ImageNet dataset.
For the student MobileNet, a pre-trained ResNet50 is used as the teacher in NORM, while a more powerful ResNet152 is used as the teacher in the paper of ExpandNets (Guo et al., 2020b) when performing the logits based KD.
Even with a less powerful teacher model, the MobileNet model trained by our NORM without the logits based KD is obviously more accurate than that trained by ExpandNets+KD, showing 3 .
79% top-1 gain.
For the student ResNet50-1/4, we follow basic training settings of WIN
(Zhou et al., 2020).
Speciﬁcally, a pre-trained ResNet50 is used as our teacher.
The student model is trained by SGD optimizer for 100 epochs, with a batch size of 256, a weight decay of 0.0001 and a momentum of 0.9.
The initial learning rate is set to 0.1 and and scheduled to arrive at zero with a cosine decaying strategy.
Strong augmentations like label smoothing (Szegedy et al., 2016) and mixup (Zhang et al., 2018a) are not used in our method.
After training, for the ResNet50-1/4 (student), NORM without KD reaches 68 .
03% top-1 accuracy with a pre-trained ResNet50 (teacher), while WIN (Zhou et al., 2020) with KD gets 67 .
50% top-1 accuracy using a pre-trained wide teacher which is 4 times larger than the student.
A. D IFFERENT DISTANCE METRICS .
For simple implementation of NORM, we use the l 2 -norm distance metric to compute the many-toone representation matching loss L norm deﬁned in Eq.
3 of the main manuscript.
In Table 8, we compare the performance of our method with three different distance metrics including l 2 -norm (our choice), l 1 -norm and mean maximum discrepancy (MMD).
In the experiments, we use the teacher-student pair of ResNet32x4 → ResNet8x4 with N = 8 .
Comparatively, l 2 -norm is superior to l 1 -norm.
With MMD, NORM reaches 76 .
62% accuracy for the student model, showing 0 .
13% improvement to l 2 -norm.
This indicates that, with a better distance metric, our method might yield a student model with higher accuracy.
We currently choose the l 2 -norm distance metric owing to its simplicity and effectiveness. [Table]
A. T RAINING COST COMPARISON
Table 9 shows a comparison of the training cost of NORM and the individual training.
We can see that the training cost of NORM for the student ResNet18, MobileNet and ResNet50-1/4 is 2 .
78 × , 2 . 48 × , 2 . 39 × than the individual training, with the teacher network ResNet34, ResNet and ResNet50-1/2, respectively.
These results indicate that NORM is efﬁcient since it uses the standard teacher-student training framework in which the pre-trained teacher model runs in the forward phase of the whole training procedure.
The forward time of the teacher can be saved by ofﬂine representation pre-computing.
For easy implementation, we did not use it.
A. V ISUALIZATION RESULTS
NORM achieves the feature distillation goal with the proposed many-to-one representation matching mechanism at a single teacher-student layer pair.
This means that the feature distribution of the student network would be more similar to that of the teacher network after NORM training, compared to the individual training.
Under this context, it is necessary to study the learnt feature distributions with and without NORM.
To this end, we use a teacher-student model pair (ResNet110 → ResNet32) well trained by NORM ( N = 8 and α = 10 ) on the CIFAR-10 dataset (a subset of CIFAR-100, which is popularly used to analyze the learnt feature distributions in KD research) and all images in the validation set, and conduct a set of experiments to analyze the learnt last-layer feature distributions using t-SNE (Maaten & Hinton, 2008).
Comparative visualization results are shown in Fig.
6, which we can observe that NORM shows relatively strong feature capability one image category in the learnt feature distribution.
A. L IMITATIONS OF NORM
Despite of its simple formulation and effectiveness, NORM has two limitations.
The major limitation is applying the many-to-one representation matching to multiple teacher-student layer pairs cannot achieve further improved distillation performance, unlike many existing KD methods.
This is due to the linear property of our student FT design, as we explored in the ablative experiments (see Table [Table] and Table 5 in the main manuscript).
Besides, applying the many-to-one representation matching to multiple teacher-student layer pairs will lead to increased training cost because the many-to-one representation matching also needs to be computed multiple times. A. M ORE EXPERIMENTS AND DISCUSSIONS FOR THE REBUTTAL
In this section, we provide a lot of extra experiments and discussions provided for the rebuttal.
A.10. A SYSTEMATIC STUDY OF NORM
So far, we have provided several sets of ablative experiments to show: (1) a large N value is much better than N = 1 , see Figure 2;
(2) Given the expanded student representation consisting of N feature segments, the more the feature segments used to match the teacher representation, the larger the accuracy gain, see Figure 4; (3) Inserting a linear FT module into different student networks usually does not bring accuracy improvement under individual training, see Table 1, Table 2 and Table 5.
In order to better understand what enables the distillation effectiveness, here we systematically study the major components of NORM from the following four more aspects.
All following ablative experiments are performed on ImageNet dataset with ResNet34 as teacher and
ResNet18 as student, N = 8 and α = 8 .
The role of the learnable ensemble layer.
In light of the interpretation of NORM given in the
Method section, the second linear layer W sc of our student FT module performs a learnable ensemble of N distillation-augmented student feature views via fully connected channel mixing operations.
Accordingly, we conduct a set of ablative experiments to explore the role of this component.
In the experiments, we compare NORM with 3 different student FT designs: (a) our default two-layer student FT module, (b) a two-layer student FT module with the ﬁxed ensemble layer (i.e., feature channels from different segments are sequentially averaged), and (c) a three-layer student FT module (i.e., having two same-size linear layers as a learnable ensemble).
Detailed results are summarized in Table 10.
It can be seen that all three student FT modules bring clear accuracy improvements, validating the importance of the second linear layer for ensembling N augmented student feature views.
Comparatively, NORM with the learnable ensemble layer is better than NORM with the ﬁxed ensemble layer, and further improved result is attained when using a more complex learnable ensemble (two-layer).
The role of the student FT module initialization.
Note that N expanded student feature segments generated by the ﬁrst linear layer of our student FT module enable the many-to-one representation matching.
Next, we perform a set of ablative experiments to study the effect of different weights initialization strategies for the ﬁrst linear layer.
In the experiments, we compare NORM withdifferent weights initialization strategies: (a) the default weights initialization we used, (b) initialization with a higher weights variance (8 × of the default), and (c) initialization with the same weights to N segments.
Detailed results are summarized in Table 11.
It can be seen that all three weights initialization strategies get promising performance, showing the robustness of our method to different initialization strategies.
Comparatively, initialization with a higher weights variance tends to bring a bit more gain.
Whats more, initialization with the same weights to N = 8 segments is much better than N = 1 .
This beneﬁts from the second linear layer which uses fully connected operations for dynamically mixed feature ensembling.
As a results, the gradients are different to the weight groups initialized with the same values, producing different student feature segments (we also conﬁrm this by recording and comparing them on the ﬂy) to match the teacher representation.
[Table]
[Table]
The role of the designs to construct the many-to-one representation matching.
NORM enables the many-to-one representation matching via expanding the student representation to have N times feature channels than the teachers.
We can also construct multiple student-to-teacher matching routes in other ways.
Accordingly, we perform another set of ablative experiments to study the effect of different designs to construct the many-to-one representation matching.
In the experiments, we compare 5 different designs: (a) our default design, (b) a reversed design (contracting the teacher representation by a single FT to have 1 /N times feature channels than the students, without using our linear student FT module), (c) a reversed design (using N independent FTs to the teacher representation, without using our linear student FT module), (d) a paired design (using N independent
FTs to the teacher representation and another N independent FTs to the student representation), and (e) a paired design (using N independent FTs to the teacher representation, and using our linear student FT module).
Detailed results are summarized in Table 12.
It can be seen that all ﬁve designs improve the performance of the student network, validating the importance of the many-to-one representation matching concept.
Comparatively, our default design achieves the best performance, and the designs that apply a single FT or multiple FTs to the teacher representation show obviously worse performance due to the information loss (that is, it is important to preserve the intact pre-trained teacher representation).
The role of the ways to split the expanded student representation.
In NORM, for simplicity, we sequentially split the expanded student representation into N feature segments.
Our last set of ablative experiments is to study the effect of different feature splitting strategies.
In the experiments, we test NORM with 3 different feature splitting strategies: (a) sequential feature splitting, (b) random feature splitting, and (c) importance-based feature splitting is which we ﬁrst sort feature channels in descending order based on the learnt mean values of channel-wise batch normalization parameters at the ﬁfth epoch, and then use sequential feature splitting to enable our many-to-one representation matching.
Detailed results are summarized in Table 13.
It can be seen that all three feature splitting strategies show almost the same performance.
This is because that there is no semantic channel-wise alignment between the teacher and the expanded student representations before the many-to-one representation matching.
Based on all ablative experiments described above, we validate the roles of the major components of our method, and provide a deep understanding of what enables the distillation effectiveness of
NORM.
A.10. C OMPARISON OF SEMANTIC SEGMENTATION RESULTS ON C ITYSCAPES DATASET
In order to test the effectiveness of our method on image segmentation task, we perform two sets of experiments with Cityscapes dataset (Cordts et al., 2016).
Cityscapes dataset contains
[Table]
[Table]
images with a split of 2975, 500 and 1525 images are for training, validation and test, respectively.
In the experiments
[Table]
we test our method with both same type and different type teacher-student network pairs for semantic segmentation, compare it with SKD (Liu et al., 2019a), IFVD (Wang et al., 2020), CWD (Shu et al., 2021), CIRKD (Yang et al., 2022a) and MGD (Yang et al., 2022c).
Speciﬁcally, we test our method using DeepLabV3-ResNet101 (Chen et al., 2017) as teacher network and DeepLabV3-ResNet18 as student network ﬁrst, and then using PSPNet-ResNet101 (Zhao et al., 2017) as teacher network and DeepLabV3-ResNet18 as student network, following the training and test settings used in very recent works of CIRKD and MGD.
We add NORM ( N = 4 for training efﬁciency) after the last feature layer of DeepLabV3-ResNet18.
All models are trained with 8 Nvidia
Tesla V100-SXM3 GPUs.
Detailed results are summarized in Table 14 and Table 15.
It can be seen that our method achieves new state-of-the-art results on these two teacher-student network pairs, validating its effectiveness in handling semantic segmentation task.
A.10. C OMPARISON OF OBJECT DETECTION RESULTS ON MS COCO DATASET
In order to test the effectiveness of our method on object detection task, we also perform two sets of experiments with MS COCO dataset (Lin et al., 2014).
MS COCO dataset (2017) contains 118, training images and 5,000 validation images with 80 object classes.
In the experiments, we test our method with both same type and different type teacher-student network pairs, and compare it with
FKD (Zhang & Ma, 2021), CWD (Shu et al., 2021), FGD (Yang et al., 2022b) and MGD (Yang et al., 2022c).
Speciﬁcally, we test our method using RetinaNet-ResNeXt101 (Lin et al., 2017b) as teacher detector and RetinaNet-ResNet50 as student detector ﬁrst, and then using Cascade Mask
RCNN-ResNeXt101 (He et al., 2017) as teacher detector and Faster RCNN-ResNet50 (Ren et al., 2015) as student detector, following the training and test settings used in very recent papers of FGD and MGD.
We add NORM ( N = 4 for training efﬁciency) after each output of the FPN neck (Lin et al., 2017a) of RetinaNet-ResNet50/Faster RCNN-ResNet50.
All models are trained with 8 Nvidia
Tesla V100-SXM3 GPUs.
Detailed results are summarized in Table 16 and Table 17.
It can be seen that our method achieves new state-of-the-art mAP results on these two teacher-student detector pairs, validating its effectiveness in handling object detection task.
[Table]
[Table]
[Table]
A.10. P ERFORMANCE ON I MAGE N ET DATASET WITH VISION TRANSFORMERS
In the main manuscript, we evaluate our method with 16 teacher-student network pairs (13/3 pairs on
CIFAR-100/ImageNet dataset).
The main reason for this is that in the community, mainstream KD methods for image classiﬁcation, e.g., 19 recent KD methods compared in our work, typically use convolutional architectures for performance evaluation.
We follow them for fair and easy comparisons.
Beneﬁted from the simplicity of our method, we can easily use it to other network architectures.
Accordingly, we perform a set of new experiments on ImageNet dataset to test the effectiveness of our method with recent vision transformer architectures.
Speciﬁcally, we use DeiT-Small (Touvron et al., 2021) as teacher network and DeiT-Tiny as student network, following the settings of VITKD (Yang et al., 2022d) (a very recent work posted on Arxiv in September 2022).
For the student network, we add NORM ( N = 4 for training efﬁciency) to the same layers as VITKD.
Detailed results are summarized in Table 18.
It can be seen that our method also works well on vision transformer architectures, achieving promising results.
Here, it should be noted that the standard training of a popular vision transformer model is very time consuming due to signiﬁcantly larger model size and longer training schedule, compared to that of mainstream convolutional networks (e.g., 300 epochs for DeiTs vs. 100 epochs for ResNets).
One run of training the above teacher-student pair of vision transformers by NORM needs about 2.5 days with 8 Nvidia Tesla V100-SXM3 GPUs.
A.10. M ORE DISCUSSIONS AND EXPERIMENTS .
Recently, self-supervised learning research with self-distillation settings (a particular type of unsupervised learning) has attracted increasing attention.
In this line of research (e.g., SimCL (Chen et al., 2020b), MoCo (He et al., 2020), SwAV (Caron et al., 2020), BYOL (Grill et al., 2020),
DenseCL (Wang et al., 2021), DINO (Caron et al., 2021), iGPT (Chen et al., 2020a), iBOT
(Zhou et al., 2022), BeiT (Bao et al., 2022) and MAE (He et al., 2022)), matching N features of the student to N features of the teacher by referring to dense features of different image patches has been explored.
Indeed, they also have the concept of many parallel matching routes with their corresponding components, but our method differs with them in focus, formulation and application.
Firstly, existing methods of self-supervised learning with self-distillation settings aim to learn a proper visual representation of a single backbone from a large set of unlabeled images, while our work aims to improve the visual representation of a small student network by a pre-trained larger teacher network under condition that both of them are trained on a set of labelled images.
Secondly, in formulation,
[Table]
[Table]
many existing self-supervised learning methods (e.g., SimCL, MoCo, SwAV, BYOL, DenseCL and DINO) deﬁne a contrastive predication task that encourages the encoder (backbone) to attract similar (positive) sample views and dispel different (negative) sample views with their corresponding contrastive losses leveraging data augmentation, and some others (e.g., iGPT, BeiT and MAE) deﬁne a masked reconstruction task that uses a decoder to predict the original image (pixel-wise/patch-wise) given the representation learnt by the encoder (backbone) with their corresponding reconstruction losses leveraging masked image modeling.
iBOT further combines masked image modeling into self-supervised contrastive learning for improved performance.
To these methods such as DenseCL and iBOT, self-distillation is performed between the target model (student) and the online model (teacher, an exponential moving average of student parameters) via contrasting dense sample view pairs of the whole image or the patch (some patches may be masked), and both models share the same encoder (backbone).
In contrast, our method leverages a linear FT module added after the last convolutional layer of a small student network to formulate many-to-one representation matching scheme between a single teacher-student layer via student feature expansion and splitting, and dense mimicking.
All segments of our expanded student features are simultaneously forced to be similar to the teacher features, and there is no contrastive matching and no paired data augmentation to the input image.
Thirdly, in application, the trained encoder of self-supervised methods is typically used to downstream tasks, while the efﬁcient student network trained by our method is for direct deployment.
Besides, iBOT, iGPT, BeiT and MAE with masked image modeling are particularly used to train vision transformers under unsupervised learning regime, while our method is mainly for training efﬁcient convolutional neural networks under supervised learning regime.
Because of the above differences, it is non-trivial to directly combine our method with BYOL, DINO,
Dense CL, iBOT, and etc.
Similarly, in the community of self-supervised representation learning, all aforementioned methods did not compare their performance with mainstream KD methods.
Note that recent work CRD extends contrastive learning into KD research by presenting a novel contrastive distillation loss, and in the main manuscript we already validated the effectiveness of combining our method with CRD (see Table 1 and Table 2).
Combining NORM with masked generative learning.
We notice that a very recent work
VITKD (Yang et al., 2022d) explores the use of the aforementioned masked generative learning for vision transformer based KD research.
Beneﬁted from the simplicity of our method, we can easily combine NORM with VITKD to test our methods effectiveness in training a vision transformer under masked generative learning.
Accordingly, we perform a set of new experiments on ImageNet dataset, using DeiT-Small (Touvron et al., 2021) as teacher network and DeiT-Tiny as student network, following the settings of VITKD.
For the student network, we add NORM ( N = 4 for training efﬁciency) to the same layers as VITKD.
Detailed results are summarized in Table 19.
It can be seen that our method also works well on transformer architectures, achieving promising results.
Here, it should be noted that the standard training of a popular transformer model is very time consuming due to signiﬁcantly larger model size and longer training schedule, compared to that of mainstream convolutional networks (e.g., 300 epochs for DeiTs vs. 100 epochs for ResNets).
One run of training the above teacher-student transformer pair by NORM needs about 2.5 days with 8 Nvidia Tesla
V100-SXM3 GPUs.
We will continue to explore the potential of our method to more teacher-student transformer pairs in the future.