Hilbert Distillation for Cross-Dimensionality Networks
Anonymous Author(s)
AffiliationAddress email
Abstract 3D convolutional neural networks have revealed superior performance in processingvolumetric data such as video and medical imaging.
However, the competitiveperformance by leveraging 3D networks results in huge computational costs, whichare far beyond that of 2D networks.
In this paper, we propose a novel Hilbert curvebased cross-dimensionality distillation approach that facilitates the knowledge of3D networks to improve the performance of 2D networks.
The proposed Hilbert
Distillation (HD) method preserves the structural information via the Hilbert curve,which maps high-dimensional (>=2) representations to one-dimensional continuousspace-filling curves.
Since the distilled 2D networks are supervised by the curvesconverted from dimensionally heterogeneous 3D features, the 2D networks aregiven an informative view in terms of learning structural information embeddedin well-trained high-dimensional representations.
We further propose a Variablelength Hilbert Distillation (VHD) method to dynamically shorten the walkingstride of the Hilbert curve in activation feature areas and lengthen the stride incontext feature areas, forcing the 2D networks to pay more attention to learningfrom activation features.
The proposed algorithm outperforms the current stateof-the-art distillation techniques adapted to cross-dimensionality distillation ontwo classification tasks.
Moreover, the distilled 2D networks by the proposedmethod achieve competitive performance with the original 3D networks, indicatingthe lightweight distilled 2D networks could potentially be the substitution ofcumbersome 3D networks in the real-world scenario.
Introduction
Knowledge distillation aims to transfer knowledge from cumbersome models to lightweight models.
The vanilla knowledge distillation [10] collects the logits that contain the cognizance of wrongclasses in the cumbersome model.
Forcing the lightweight model to mimic the logits can obviouslystrengthen its parsing ability.
Recent efforts further devote to improve the distillation effectivenessby adopting intermediate representations [8,16,29,32,33] and the relation knowledge [20–22,30]of samples as new distillation mediums.
In some real-world scenarios (e.g., video analysis andmedical imaging processing), 3D models typically present overwhelming performance comparedwith common 2D models, yet the 3D models suffer from huge computational costs.
One way tomitigate this problem is by leveraging knowledge distillation to encapsulate the parsing ability of 3Dmodels into 2D models.
However, the cross-dimensionality distillation problem, especially 3D-to-2Ddistillation, is largely unaddressed in the literature.
Most existing methods that utilize intermediate features for distillation are designed by employinga metric function (e.g., mean square error (MSE) and Kullback-Leibler (KL) divergence)
betweenthe pair of feature maps extracted from teacher and student networks.
Unfortunately, they are notapplicable in cross-dimensionality distillation tasks due to the inconsistent dimensionality.
To get ridof the constraint, several dimensionality reduction methods such as pooling and convolution that can completely condense the extra dimension maybe feasible by converting the 3D feature maps (with thesize of D × W × H ) to reduced 2D-from-3D representations (with the size of 1 × W × H ).
However,the reduction tends to dropout the high-dimensional structural information.
As a result, they havelimited applicability to deal with volumetric data in the case of lacking the structural information.
In this paper, we propose a new method, namely Hilbert Distillation (HD) , to explicitly distillstructural information embedded in intermediate features of 3D models.
Our approach preservesboth 3D and 2D features and maps them into one-dimensional representations based on the intrinsicrules of Hilbert curve [9] in the original feature space.
The distilled models with Hilbert curveretain the relative position information in the high-dimensional space, i.e., the mapped data pointsthat are close to each other in the 1D space are also adjacent in the original high-dimensionalspace, thereby the structural information is highly preserved via the proposed HD method.
Hence,the distilled lightweight 2D models obtain significant improvements by learning the convertedone-dimensional representations from the 3D feature maps.
Furthermore, we propose Variablelength Hilbert Distillation (VHD) to more efficiently transfer the structural knowledge by dynamicallyshortening the walking stride when constructing Hilbert curves in activation areas of feature maps [34].
To address the optimization problem in the dynamic process, we also present an approximationapproach by calculating self-adaptive weights when adopting mapping functions of the Hilbert curve.
We conduct extensive cross-dimensionality distillation experiments in activity and medical imageclassification domains, where 3D models are typically applied.
The proposed method further narrowsthe gap between student and teacher networks comparing with the current state-of-the-art in thedistillation benchmark.
More importantly, the distilled 2D models by our method achieve competitiveperformance with the 3D models, suggesting that the structural information learned by the proposedmethod helps promote the generalizability of the 2D models.
Hence, the distilled lightweight modelscould potentially be the substitution of cumbersome models for analyzing the volumetric data.
Related Work
Knowledge Distillation .
The vanilla knowledge distillation [10] focuses on transferring knowledgeembedded in the logits, i.e., the model outputs scores before applying Softmax.
It also providesthe idea that coarse information would be beneficial to the distillation, and proposes the Softmaxwith temperature should be adopted before the calculation of distillation loss.
Succeeding efforts[8,16,23,29,32,33] try to utilize intermediate representations further.
For example, the Attention
Transfer [33] calculates attention maps by accumulating values in feature maps along with the channeldimension.
The attention maps set out the most informative features for the student model and tacklethe problem of different channel sizes between participated models.
Some other works [20–22,30]try to extract the relation knowledge among samples to enable the distillation between models.
For instance, SP [30] calculates the relationships between intermediate feature maps of differentinputs.
Conducting the distillation between student and teacher relationship maps results in decentimprovements to the student model.
Recent works tend to explore deeper knowledge and transfer theknowledge in more complicated ways.
Ji et al.
[13] propose to use feature maps of all layers throughthe integration with self-adaptive weights learned by attention mechanism.
HKD method [35] designsa graph-based scheme to distill holistic knowledge, thus the student model is able to learn individualknowledge and relational knowledge simultaneously.
Dimensionality Reduction .
Most of the mentioned methods are not applicable for crossdimensionality distillation networks due to the inconsistent dimension.
A straightforward wayto cope with this issue is to adopt dimensionality reduction methods.
For example, PCA [11] and
LDA [6] are commonly used to obtain low-dimensional representation by projecting data points to afew principal components.
Some non-linear dimensionality reduction methods such as ISOMAP [28]and LLE [3] are available for more complicated sample distributions.
Recent efforts about transformer [4,24] provide valuable ideas for learning position coding for complex representations such ascontinuous large-scale feature maps.
The reduction can also be performed by following the learnedposition coding.
However, these methods either lose the important structural information for thecross-dimensionality distillation to a certain extent or divide the original high-dimensional space intomuch smaller space where the problem of dimensional inconsistency still exists.
This work refers toa continuous space-filling dimensionality reduction approach, namely Hilbert curve, to address theabove issues.
Details are described in the next section.
[Figure]
Cross-dimensionality Distillation .
Recently, cross-dimensionality distillation has also been studiedin [2, 12, 17] that have considered adopting 3D models as the distillation participants.
However,their methods still follow the 2D-to-2D distillation by applying customized tricks in order to transfer3D representations into the 2D form, resulting the 2D models are improved only marginally afterperforming the severely diluted knowledge.
To this end, we propose a new cross-dimensionalitydistillation method by explicitly extracting and transferring the structural knowledge embedded inhigh-dimensional feature maps.
Method
[Figure]
In this section, we first introduce the Hilbert Distillation for cross-dimensionality scenarios based onthe Hilbert curve.
Then we elaborate the proposed Variable-length Hilbert Distillation by dynamicallyadjusting the stride of Hilbert curve deduction.
Since encoding and optimizing the dynamic progressis challenging, we further present an approximation solution by mapping the conventional Hilbertcurve into Variable-length Hilbert curve using self-adaptive weights.
[Figure]
3.1 Hilbert Curve
[Figure]
The Hilbert curve [9] is a continuous space-filling curve in the Euclidean space, which provides amapping between 1D and high dimension space while preserves space structure fairly well.
Theprinciple behind the Hilbert curve is that the mapped data points which are close to each other in 1Dspace are also close to each other in the original high dimension space.
The construction [1,19] of the Hilbert curve can be different.
In this paper, we adopt LindenmayerSystem [15] (L-System) to describe the construction that produces walking guides recursively.
Thesystem consists of 4 components: 1) two Variables , i.e., A and B ; 2) three Constants , i.e., ▷ denotesmove forward, ⊕ denotes turn left 90 ◦ , and ⊖ denotes turn right 90 ◦ ; 3) the Axiom A , i.e., the startingpoint of recursion; 4) two Production Rules :
Rule 1 : A → ⊕ B
[Equation]
We denote that order p controls the times of recursion as well as the scale of the curve.
When p = 1 ,we can generate the walking guides as ⊕ ▷ ⊖ ▷ ⊖ ▷ ⊕ (note that the Variables A and B are onlythe placeholder for embedding the rules) based on the Production Rule 1 (Eq. 1) from the startingpoint Axiom A .
Thus, the simplest Hilbert curve for a 2 × 2 square space illustrated in Fig.
1(a) canbe constructed by looking right at the start.
If p = 2 , the Production Rules 2112 are embedded into
BAAB allocated in Production Rule 1 separately.
Then we can get the walking guides as ⊕ ⊖ ▷⊕ ▷ ⊕ ▷ ⊖ ▷
[Equation]
⊖ ▷ ⊖ ▷
[Equation]
▷ ⊕ ▷ ⊖ ▷ ⊖ ▷ ⊕ ⊖ ▷ ⊖ ▷ ⊕ ▷ ⊕
▷ ⊖ ⊕ .
It is importantto note that the contiguous Constants ⊕ and ⊖ will be cancelled out and removed from the guides.
Hence, the final walking guides will be ▷ ⊕ ▷ ⊕
[Equation]
⊕ 124 ▷ ⊕ ▷ .
Fig. 1(b) illustrates the Hilbert curve of order p = 2 for a larger square space based on theguides.
Similarly, the Hilbert curve for higher order can be easily constructed via this method.
It canalso be extended and applied in higher dimensional space [1] as shown in Fig.
1(c).
3.2 Hilbert Distillation
Since the above mentioned curve is able to well preserve the structural information of its original space,we facilitate the curve to distill knowledge from dimensionally heterogeneous features.
Specifically,
[Figure]
we aim to supervise the 2D student models by efficiently learning the knowledge that they have neverseen in feature maps extracted from the 3D teacher models.
Fig. 2(a) illustrates the pipeline of the proposed Hilbert Distillation framework.
Let ( D t , W t , H t )and ( W s , H s ) denote the size of the 3D feature map η 3 d and 2D feature map η 2 d respectively.
Wefirst leverage the mapping function of Hilbert curve H n,p (refer to Algorithm 1) to rearrange andstretch the feature maps.
Let parameter n denotes the dimension (we use 2 or 3 in this paper, buttheoretically it could be larger), and p denotes the order that controls the curve length as described in
Sec.3.1.
We define the following rule to determine the value of p using the ceiling function ⌈·⌉ :
[Equation]
After determining the values of n and p , we adopt the Lindenmayer-System described in Sec 3.1 togenerate the Hilbert curve with desired scale, and Algorithm 1 helps deduce the mapping function
H n =2 ,p ( i, j ) = v that gives the pixel-level surjective mapping rule from spatial feature map η 2 dto a one-dimensional representation ˆ η 2 d .
Although our algorithm describes the case of n = 2 , itcan synergistically be expanded to n = 3 to get the 3D mapping function H n =3 ,p ( i, j, k )
=
v withthe construction of Hilbert curve for 3D space.
The output value v is the new position index of theoriginal pixel-level feature in the one-dimensional representation with the length 2 n × p .
Thus, theone-dimensional representations ˆ η 3 d , ˆ η 2 d of the feature maps η 3 d , η 2 d can be calculated by
[Equation]
i, j, k ) where H n =3 ,p ( i, j, k ) =
v
[Equation]
5) ˆ
[Equation]
=
η 2 d ( i, j ) where H n =2 ,p ( i, j ) = v (6)
At this point, we can define the Hilbert Distillation between cross-dimensionality features by calculating the loss between the converted one-dimensional representations as
[Equation]
(7) where R ( · ) is the nearest rescaling function that scales the length of ˆ η 3 d from 2 3 × p to 2 2 × p before thecalculation, and ||·|| 2 and ||·|| 1 are the L2 and L1 normalization.
The division between representationsand its L2 norm is intended to eliminate the distribution difference between features from differentmodality.
As a knowledge distillation method, the distillation process is commonly performed alongwith the training process.
The final loss function for the end-to-end training scheme is given by
[Equation]
8) where L CE is the cross entropy loss corresponding to the original task of the student network, andthe hyperparameter α is a balance weight between training from distillation and hard labels.
As claimed in the interpretability [14,25,34] and attention [33] research areas, only partial featuremaps are activated and crucial for the final task.
This conclusion implies the reason why our method
1 Initialize i = 0 , j = 0 , v =
Save the map H n =2 ,p i,
= v 3 while Algorithm 1 is running do //
Algorithm 1 starts with looking right if ▷ is activated then if look up / down then i
= i
+ 1 / i
= i − else if look right / left then j = j + 1 / j = j − v = v +
Save the map H n =2 ,p ( i, j ) = v else
Waiting for ▷ 14 Output H n =2 ,p works.
In other words, the superiority of the proposed distillation method is that the 2D modelscan directly learn the activation features that preserve the 3D structural information as much aspossible.
As illustrated in Fig. 2(a), the activated “dog features” in 3D feature maps (leftmost) arestill gathered together after the mapping, as shown in the black area of the bar.
Theoretically, ourmethod could be extended to conduct distillation between networks of any dimensionality since the
Hilbert curve-based mapping approach can be applied in any Euclidean space.
In addition, we also provide several tricks to address the following possible minor obstacles whenapplying our method in real-world scenarios:a. What if the activation features in the converted one-dimensional representations of 2D and3D models are not relatively aligned .
Fig.
2 demonstrates an ideal case that both dogs are locatedat the left bottom so that the relative positions of the activation features are well aligned.
In reality, inthe medical imaging task that 3D models are commonly applied, the spatial distribution of humanorgans are always fixed regardless of the data modality.
The activation features from 2D and 3Dfeature maps are well aligned in the final stage of our method.
For another task using 3D models, suchas video recognition, a simple length-preserving fully connection layer (FC) for any one-dimensionalrepresentation before the distillation loss calculation is able to address this issue.b.
What if the feature maps do not exactly fulfill a square or cube space .
As the vanilla Hilbertcurve is only applicable in the space with a side length equals a power of 2, we always construct
Hilbert curve for a minimum space that can hold the feature maps according to the side length whichcan be calculated by Eqs.
3 and 4, e.g., constructing curve for 8 × 8 × 8 space according to a 5 × 7 × 7feature map.
The mapping function H only considers the areas where contains features.
3.3 Variable-length Hilbert Distillation
The scale of activation features is relatively small compared with that of context features in mostcases, as the ratio of black and gray parts illustrated in Fig. 2(a).
The proportion of activation featuresin 3D feature maps may even lower than in 2D feature maps.
To make the distillation more focus onthe activation areas where the features are significantly related to semantics, we further propose animprovement for the construction process of Hilbert curve called Variable-length Hilbert Distillation.
Specifically, we dynamically change the walking stride when constructing the Hilbert curve.
Asdemonstrated in Fig.
2(b), we lengthen the walking stride in areas that consist of context features todraw sparser curves, and shorten the stride in areas that consist of activation features to draw densercurves.
Those skipped context features will be represented by the nearest valid features located onthe curves.
Thus, the proportion of activation features in the calculation of the final distillation loss isgreatly amplified, as depicted in the figure.
To this end, we propose to alter the Production Rules
(Eqs. 1 and 2) in the described L-system into the formation as
[Equation]
where ϕ ( A | B ) is the decision function that decides whether to perform recursion, which is given byϕ ( A | B )
=
[Equation]
A | B, if upcoming walking routes that follow the Hilbert curve contain an activation feature,
Null , otherwise.
[Equation]
In order to figure out whether there is an activation feature, we define the Activation Mapping (AM)for a given convolutional layer l .
We first compute the activation weight γ for a single feature map inthe layer l .
For 2D and 3D models, the computations are separately defined as
[Equation]
where m and n denote the numbers of feature maps, z 3 d and z 2 d are the numbers of features in the3D and 2D feature maps, ∂y c ∂η is the gradient of the output score y c of class c with respect to featuremaps η .
Different from the Class Activation Mapping (CAM) proposed by [25], we compute thegradients for all the objective classes and accumulate the results because the knowledge distillationmethod requires preserving informative knowledge as much as possible.
The activation featuresfor wrong classes may also be helpful to the distilled model.
To this end, we calculate the AM byweighted sum of the feature maps, which is given by
AM
[Equation]
The AM generally presents the importance of features in corresponding positions, and the size of AMis the same as the size of the feature map in the layer l .
Moreover, we get the activation if the feature’s
Sigmoid value in AM at the corresponding position is larger than a threshold θ .
The threshold is userdefined based on the required degree of activation.
Fig.
1.(d) illustrates a case when order p = 2 .
For the first round of rule generation with Eq. 9, theinferred walking guides can be ⊕ [ γ ϕ ▷ ]
⊖ A [ γ ϕ ▷ ] A ⊖ [ γ ϕ ▷ ] ⊕ .
Only the two areas marked “ A ”where contain the activation feature are preserved after the execution of ϕ .
The additional factor γ ϕin the guides controls the multiple compensation for the walking steps, which doubles the executionnumber of the following ▷ whenever a neighboring ϕ results in Null .
Hence, the actual walkingguides of the first round are ⊕ ▷ ▷ ⊖
A ▷ A ⊖ ▷ ▷ ⊕ .
In the next round of generation, we can getthe final walking guides as ⊕ ▷ ▷ ▷ ⊖ ▷ ⊖ ▷ ⊕ ▷ ⊕ ▷ ⊖ ▷ ⊖ ▷ ▷ ▷ ⊕ by replacing A with
Eq. 9 recursively.
This idea is the desired Variable-length Hilbert curve depicted in Fig. 1(d).
The
Variable-length Hilbert Distillation (VHD) can be hereby designed by following the framework of
HD and substituting the curve construction process, as the illustrated pipeline in Fig. 2(b).
However, unlike the vanilla Hilbert curve that offers the hard mapping for a space of fixed scale,the Variable-length Hilbert curve is dynamically changed in terms of different activation mapsfrom different inputs, however, it is hardly be performed in forward propagation and optimized inbackpropagation.
In this paper, we propose an approximation approach to achieve the variable-lengthby calculating self-adaptive weights when adopting the mapping function H .
We rewrite the originalmapping process in Eqs.
5 and 6 using the product of the features and the values in AM, as follows,
[Equation]
Finally, the optimizable distillation loss function of the proposed approximation scheme is given as
[Equation]
Similar with HD, we define the end-to-end training of VHD by replacing L hd with L vhd in Eq.8.
AT
FitNet
SK D
IFVD
AFD H D VH D 6 1 .6
T o p1 A c c u r a c y ( % ) a v g c n v m a x (a) ActivityNet
AT
FitNet
SK D
IFVD
AFD H D VH D
T o p1 A c c u r a c y ( % ) a v g c n v m a x
[Figure]
Experiments
4.1 Datasets
3D models present overwhelming performance in some specific tasks, such as video recognitionand medical imaging processing.
Therefore, we demonstrate the effectiveness of the proposed crossdimensionality distillation method on two datasets: ActivityNet [5] dataset for activity classificationand Large-COVID-
[18] dataset for medical imaging classification.
ActivityNet , which is a benchmark video dataset for untrimmed human activity classification.
Weadopt the ActivityNet-100 version that contains 100 human activity classes.
We split the 7000 videosinto 2:1 for training and testing.
Large-COVID-19 , which is a lung Computed Tomography (CT) dataset of COVID-19 for Community Acquired Pneumonia (CAP) classification.
This dataset contains 7593 COVID-19 images,
CAP images, and 6893 normal images.
Most images can make up the 3D CT scan by combiningtheir neighboring images.
We randomly select 20% images as the test set.
4.2 Experiment Setting
Distillation Setup .
Different from existing distillation works that consider the same dimensionalityin the teacher and student models, we adopt ResNet-50 [7] and VGG16 [27] as the 2D studentmodels and well-trained 3D ResNet-50 as the 3D teacher model.
In the training process, we feed therandomly selected images into the 2D models, and the volumetric data that consists of 16 consecutiveframes into the 3D model.
The selected pair of 2D and 3D inputs are restricted in the same category.
Since the category of the entire video (activity) is hard to be classified using only one frame, the 2Dtraining scheme of video processing commonly adopts the combination of 2D CNN + LSTM [26]or 2D CNN with voting strategies.
We follow the former scheme to train the 2D student model bysequentially feeding one of the 16 consecutive frames that are prepared for the 3D teacher modelinto the 2D student model.
The distillation method is applied between 3D CNN and 2D CNN beforeperforming the voting.
We train and test models on an NVIDIA GeForce RTX 3090 GPU (24GB).
Network Architectures .
For the activity classification task, we randomly crop the input frame into256 × 256.
The batch size is set to 16.
The training process lasts 40 epochs.
The hyperparameterα in Eq.
8 is set to 10 3 .
For the medical imaging classification task, we resize the input image into224 × 224 horizontally.
The batch size for training is set to 32.
The number of training epochs is 60.
Different from the activity classification task, we set α to 10.
The selection and performance of thefluctuation of α are also discussed in our experiments.
The models are trained with an initial learningrate 0.01, and the Adam optimization algorithm is employed for both tasks.
4.3 Results and Analysis
To demonstrate the performance and stability of the commonly used distillation methods, we separatethe experiments into two parts based on their applicability in the cross-dimensionality distillation
[Table]
problem.
We first present the performance of the distillation methods using feature maps, i
[Table]
e.,
AFD [13], SKD [16], FitNet [23], IFVD [31]
[Table]
and AT [33].
Since they are not to be applied directlydue to the dimensionality difference problem
[Table]
we employ three alignment functions,
i
[Table]
e., averagepooling (“avg” for short in the figure), max pooling (“max” for short in the figure), and convolutionwith kernel size D × 1 × 1 (“cnv” for short in the figure), to align the 3D feature maps into 2Drepresentations by compacting the values along the dimension D .
Fig.
3 shows the Top-1 accuracy(%) of baselines and the proposed methods for the activity and medical imaging classification tasks.
The range values in the boxes are collected from the last ten epochs of all folds.
The distillationperformance is more stable if the box is shorter.
The proposed methods HD and VHD outperform allthe baselines in terms of all the tasks.
Compared with existing methods, the accuracy of the proposed
HD method improves about 2% on the Large-COVID-19 data and 1.5% on the ActivityNet usingboth homogeneous architecture (3DResNet to ResNet) and heterogeneous architecture (3DResNet to
VGG).
The better performance of the proposed VHD method implies that VHD is able to transfer theactivation knowledge more efficiently.
We can also conclude that the max pooling and convolutionalignment functions should be prioritized when the conventional distillation approaches are not ableto deal with the dimensionality inconsistency in the cross-dimensionality distillation problem.
The goal of many distillation methods (i.e., RKD [20], PKT [21], CCKD [22], and SP [30]) is toexcavate the relation knowledge among the data samples.
Theoretically, they can be directly appliedin cross-dimensionality problems because the calculation between the feature maps of the data can beperformed inside the model.
The scale of calculated relation representations for distillation is onlyrelated to the number of samples.
The premise to be effective of those methods is that the inputs of thedistillation models are consistent, and the student models can learn relational knowledge from teachermodels.
However, this condition does not hold in the cross-dimensionality scenario.
Table 1 providesthe performance of commonly adopted methods.
We can observe that the proposed HD/VHD methodsachieve a remarkable improvement compared with the benchmark methods.
The main reason whythe baselines do not perform well is that the data prepared for 3D and 2D models are usually different.
Note that the vanilla KD [10] is always feasible if distillation is applied in classification tasks becausethe number of logits is only related to the number of classes.
The performance of the vanilla KD canalso be regarded as a benchmark of distillation methods for cross-dimensionality problems.
As shown in Table 1, the fluctuation range of VHD is greater than HD.
The main reason might be the
Variable-length Hilbert curve is not directly obtained in this work.
We approximately achieve thecurve by the products of the corresponding values in Activation Mapping, which is the same as theself-adaptive weights used for the vanilla Hilbert curve.
Although the values deduced by the gradientsof output scores do not perfectly indicate the activation features, VHD always outperforms others inall experiments.
As demonstrated in Fig. 3(b), only VHD can reach over 86% Top-1 accuracy.
4.4 Hyperparameter Tuning
In addition to discussing the effectiveness of our methods, we also conduct tuning experiments todemonstrate the influence of the hyperparameters.
We follow the setup described above, and adopt a3D ResNet-50 as the teacher model to train a ResNet-50 as the devised distillation method.
Unlike
T o pA c c u r a c y ( % )
V H D H D
C h a n n e l (a) ActivityNet 361 22 5 6
T o pA c c u r a c y ( % )
V H D H D
C h a n n e l (b) Large-COVIDllll 4
T o pA c c u r a c y ( % )
V H D H D
L a y e r (c) ActivityNet llll 4
T o pA c c u r a c y ( % )
V H D H D
L a y e r (d) Large-COVID- the range values shown above, here we only present the mean Top-1 accuracy (%) of the distilledstudent networks to compare the fluctuation more clearly.
Hyperparameters in the loss functions .
As shown in Eq.
8, the hyperparameter α controls the tradeoff of the original task and distillation during training.
Fig.
5 illustrates theperformance with respect to various α for both two tasks.
The value of α could be initialized with a relatively large value, i.e., about 10 3 for the medical imaging classification and10 for the activity classification task.
Extensive experiments demonstrate that our method canperform well in the cross-dimensionality problems given any value within a reasonable range.
[Figure]
T o pA c c u r a c y ( % )
V H D H D 1 0 - 3 1 0 - 2 1 0 - 1 1 0 0 1 0 1 1 0 2 1 0 3 1 0 4
V H D H D
T o pA c c u r a c y ( % )
The number of feature maps for distillation .
Sec.
3 introduces thedistillation using a pair of feature maps extracted from 3D and 2Dmodels.
In most cases, a given layer can generate the same numberof feature maps as the convolutional kernel.
The distillation loss in
Eq. 8 can also be calculated in batches between the same numberof dimensionally heterogeneous feature maps.
For architecture homogeneous models such as 3D ResNet and ResNet, the number offeature maps is naturally the same as in the symmetrically positionedlayer.
For architecture heterogeneous models, we suggest applyingextra 1 × 1 convolutional layers to align the channel dimension, i.e.,the number of feature maps.
Figs. 4(a) and (b) show the performanceof our method with respect to various number of feature maps.
Thehighest score in both figures is reached when using 128 feature mapsfor distillation.
Layer selection for distillation .
Finally, we show the effectivenessof our methods with respect to various convolutional layers.
Figs.4(c) and (d) illustrate that the distillation on the second layer (resblock) contributes to the most improvement for the student model.
Itis important to note that the layer distribution (from layer 1 to layer4) in our experiments is from ResNet, thereby this result is shown forreference only, if a different model architecture is applied.
In conclusion, rear layers whose featuremaps are not too small should have priority to be distilled because very small feature maps usually donot have explicit structural knowledge.
Conclusion
Although a considerable improvement has been obtained using cumbersome models when analyzingvolumetric data, the success of cumbersome models comes primarily from the availability of computing power.
Our goal is to distill the cumbersome models into lightweight models while preserving thestructural information in the high-dimensional space.
To this end, we propose a Hilbert curve-basedapproach, which aims to transfer structural information as much as possible from 3D to 2D models.
Furthermore, we design the Variable-length Hilbert Distillation to force the student to focus more onlearning the structural knowledge from activation features.
Extensive experiments demonstrate theeffectiveness of the proposed methods. Appendix
A. More Details About How Hilbert Distillation Works
First, we provide some visualization of Activation Mapping (AM) to support that only partial featuremaps are activated and crucial for the final task. As illustrated in Fig. 6, the high gradients (activationfeatures) indicated by light pixels in AM are always in and near the lung area, demonstrating thatfocusing on distilling the activation features makes sense. Based on this, we propose the Hilbert
Distillation to enable 2D student activation features to learn directly from 3D teacher activationfeatures in a dimensionality-free manner.
The core ability of Hilbert Distillation is from the trait that the Hilbert Curve can reduce the dimensionand preserve the locality of data points.
Fig.
7 shows the case of converting a 16 × 16 2D spacewith two semantic pixel-level classes to 1D space using the Hilbert Curve.
The pixels of class 2indicated by the yellow area are still largely aggregated after finishing the mapping by walking along
[Figure]
[Figure]
with the Hilbert Curve, which means the structural information is well preserved in the dimensionreduction.
Combining the illustration about Figs.
6 and 7, it is easier to understand how the proposed
Hilbert Distillation works through Fig.
As the illustrated 3D-to-2D distillation example on COVIDclassification, the Hilbert Curve can help aggregate lung features into the 1-dimensional space andpreserve the feature continuity.
Then the student features in the 2D lung area can directly learn fromteacher features in the 3D lung area.
B. Computational Costs of Generating Hilbert Mapping Function
The computational cost from generating Hilbert Curve to finishing the Hilbert-based mapping function
H n,p is very low.
We run the generation process on a single process on the Intel(R) Xeon(R)
Silver4216 CPU (2.10GHz) and calculate the computational costs on different sizes of feature maps.
Aspresented in Table 2, the mapping only costs 10ms even in processing a 3D feature map with the size256 × 256 × 256.
The time complexity of the generation process depends on the implementation ofthe Hilbert Curve.
We adopt the most applied approach to generate Hilbert Curve in O (log n ) time.
[Figure]
[Table]
[Table]
C. The Adaptability of 2D-to-2D Distillation Methods on 3D-to-2D Distillation Problems
To the best of our knowledge, there are almost no straightforward solutions for the crossdimensionality distillation problem so far.
In our experiments, we help the regular 2D-to-2D distillation approaches to adapt to 3D-to-2D scenarios for comparison through some extra operations suchas alignment functions.
When considering dealing with 3D-to-2D distillation, we categorize existingavailable 2D-to-2D distillation methods into three types.
Traditional 2D-to-2D intermediate feature maps distillation methods, such as AT, FitNet, SKD,IFVD, and AFD in our experiment (Figure 3).
Since they are not to be applied directly due to thedimensionality difference problem, we employ alignment functions such as average pooling, max
pooling, and convolution to align the 3D feature maps into 2D representations to enable the 3D-to-2D
distillation.
Traditional 2D-to-2D relation knowledge distillation methods, such as PDK, PKT, CCDK, andSP in our experiment (Table 1).
They can be directly applied in 3D-to-2D distillation because the
calculation between the calculation can be performed inside the model.
Contrastive learning based distillation methods, such as CRD [29].
The reason why they are
not included in the experiment is because the student requires the same/augmented samples of theteacher to construct “positive pair”, which is impossible in cross-dimensionality distillation as theinput dimension is different.
D. More Benchmarks
In this section, we provide more experiments on the larger dataset Kinetics-400 to further demonstratethe proposed method’s robustness.
We adopt the same training and distillation setup as ActivityNetdescribed in Sec.
Table 3 presents the results.
The performance of our method is stilloutstanding compared with the previous methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
[Table] (b) Did you describe the limitations of your work?
[Yes]
Sec. 4.3 describes the limitation
that the VHD is not directly applicable in our work.
The proposed approximation
[Table] approach may results in a larger fluctuation range of the distillation performance. [Table]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
There are
no potential negative societal impacts of our work.
[Table] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
[Table]
If you are including theoretical results...
[Table] (a) Did you state the full set of assumptions of all theoretical results?
[Yes]
[Table] (b) Did you include complete proofs of all theoretical results?
[Yes]
[Table]
If you ran experiments...
[Table] (a) Did you include the code, data, and instructions needed to reproduce the main ex- [Table]
perimental results (either in the supplemental material or as a URL)?
[Yes]
In the
[Table] supplemental material, we release the source codes as a URL.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes] Please refer to Secs.
4.1 and 4.2.
[Table] (c) Did you report error bars (e.g., with respect to the random seed after running experi- [Table]
ments multiple times)?
[Yes] We have reported all error bars in the experiments. [Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to the experiment setting in Sec. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We have properly [Table]
cited all the creators such as [5,18].
(b) Did you mention the license of the assets?
[N/A] We only use publicly available assets, [Table] and we have properly cited all the assets. [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
The supplemental material includes the source codes of this work that will be providedas a URL.
There is no other asset to be released.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] All the data used in this work are publicly available, and we have
properly cited such as [5,18].
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
There is no personally identifiable information
[Table]
or offensive content in the data we are using.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]