VTC-LFC: Vision Transformer Compression with Low-Frequency Components
Abstract
Although Vision transformers (ViTs) have recently dominated many vision tasks, deploying ViT models on resource-limited devices remains a challenging problem.
To address such a challenge, several methods have been proposed to compress ViTs.
Most of them borrow experience in convolutional neural networks (CNNs) and mainly focus on the spatial domain.
However, the compression only in the spatial domain suffers from a dramatic performance drop without ﬁnetuning and is not robust to noise, as the noise in the spatial domain can easily confuse the pruning criteria, leading to some parameters/channels being pruned incorrectly.
Inspired by recent ﬁndings that self-attention is a low-pass ﬁlter and low-frequency signals/components are more informative to ViTs, this paper proposes compressing ViTs with low-frequency components.
Two metrics named low-frequency sensitivity (LFS) and low-frequency energy (LFE) are proposed for better channel pruning and token pruning.
Additionally, a bottom-up cascade pruning scheme is applied to compress different dimensions jointly.
Extensive experiments demonstrate that the proposed method could save 40% ∼ 60% of the
FLOPs in ViTs, thus signiﬁcantly increasing the throughput on practical devices with less than 1% performance drop on ImageNet-1K. Code will be available at https://github.com/Daner-Wang/VTC-LFC.git .
Introduction
Recently, Vision transformer (ViT) [14] and its variants [50, 32, 62] have outperformed convolutional neural networks (CNNs) in several vision tasks.
However, ViT models still face the challenge of high computational cost when deployed to resource-limited devices.
Following previous experiences in compressing CNN models, some pruning methods based on sparse learning [68, 61], taylor expansion [60], or automatic searching [9] have been proposed for ViT models to reduce model redundancy via channel pruning.
In addition to the redundancy in parameters, recent literature [48, 28, 41] further points out that some noise tokens mainly encoded task-irrelevant information ( e.g. , background), and some tokens become similar in deeper layers, showing that great redundancy also exists in tokens.
The mainstream works [28, 47, 36] ﬁlter out the less informative tokens to reduce the FLOPs without changing the model structure.
[Figure]
Although aforementioned methods have made great progress in ViT compression in spatial domain, we ﬁnd that they generally suffer from the following two problems: (i) different from CNN pruning which maintains the performance well without ﬁnetuning [34, 13], dramatic performance drop is observed when the same method is applied in ViT pruning; (ii) conducting ViT pruning only in spatial domain is not robust to noise, and as shown in Figure 1, after adding noise in the images, the accuracy of spatial compression dramatically drops.
To make
ViT compression more effective and robust, we propose to conduct ViT compress with the help of frequency domain.
Recent studies [3, 55, 42, 38, 52] have indicated that self-attention (SA) behaves like a low-pass ﬁlter, and low-frequency signals/components are more informative to ViT models.
Inspired by such low-frequency characteristics of ViT, we propose a compression framework named Vision Transformer Compression with
Low-Frequency Components (VTC-LFC) which solves the problem from a new angle and emphasizes the contributions of low-frequency components during compression.
To our best knowledge, this is the ﬁrst work that compresses vision transformers in the frequency domain.
The main contributions of this paper are listed as below:
Channel pruning based on low-frequency sensitivity: Channel pruning is a popular structured pruning strategy that aims to remove redundant parameters
in fully connected layers of ViT. The mainstream works use some evaluation metrics ( e.g. Taylor scores [60], weights norm [61], or sparse factor [68]) to estimate importance scores of parameters.
Recent studies [3, 42, 52] ﬁnd that ViTs are more reliable to the low-frequency components in images, i.e. the low-frequency information is more important for ViTs.
Therefore, we infer that channels that are less effective in encoding low-frequency components will contribute less to the feature representation for ViT models.
Motivated by such a property, we propose a better channel pruning criterion named low-frequency sensitivity (LFS) based on the Taylor scores [60].
Different from the standard Taylor scores which are computed with the original images, LFS ﬁlters out high-frequency components from images and uses only low-frequency components to estimate the importance of model parameters.
In this way, channels that efﬁciently encode low-frequency information are more likely to be preserved, and the compressed model tends to be more robust to noise.
Experimental results show that LFS can alleviate the performance drop after compression without bells and whistles.
Token pruning based on low-frequency energy:
Token compression/sampling aims to select the informative tokens that store more useful information.
The popular methods dynamically select those tokens with high correlation to other tokens ( e.g. the
CLS token) as the informative tokens.
However, it may be sub-optimal because the selected tokens tend to be similar to each other, and the information included in the token itself has been neglected to some extent.
As pointed out by [55, 38], the self-attention module in ViTs behaves like a low-pass ﬁlter, i.e. the tokens with more low-frequency components can pass more information to the next layers.
Inspired by this, we propose improving the attention-based token selection with an extra item, token low-frequency energy (LFE), which quantiﬁes the low-frequency information in tokens.
By correcting the attention scores with LFE of tokens, the selector can better distinguish informative tokens from both the long-term dependency in spatial domain and low-frequency contributions in frequency domain.
Bottom-up Cascade Pruning Framework: To jointly compress channels and tokens of vision transformers, we propose a bottom-up cascade pruning framework.
The model accuracy is further preserved through automatically balancing compression ratios of channel pruning and token pruning block-by-block.
Related work
Vision Transformer.
Inspired by the success of transformers [51] in NLP, the Vision Transformer (ViT) [14] is proposed to encode an image into a sequence of tokens and feed them into the pure transformer architecture.
Several studies have shown that ViT performs better than convolution neural networks (CNN) on image classiﬁcation benchmarks [14, 20] when sufﬁcient training data is provided.
Many follow-up variants of ViT [4, 7, 1, 19, 54, 10, 62, 53, 66] have also been proposed.
For example, DeiT [50] introduces a distillation token structure into ViT, and LV-ViT [24] proposes the token labeling approach for better training of ViT.
In addition to image classiﬁcation,
ViT has also achieved great performance in many other computer vision applications, such as semantic segmentation [11, 56, 12], image retrieval [22, 17], object detection [2, 69] and image reconstruction [8, 59].
However, despite of the outstanding performance in a series of tasks, its high computational cost restricts the deployment of ViT, which motivates the study of lightweight ViT models, including pruning [9, 68, 44, 58], block-weights sharing [26, 64], fast distillation [57], and dynamic prediction architecture [67, 45], among which pruning is a universal approach for almost all model structures.
ViT Pruning.
As an efﬁcient compression approach, pruning [35, 23, 29, 30, 65, 33, 31] has been widely applied on various convolutional neural networks (CNNs) in computer vision.
Pruning approaches [63, 21, 58, 48] have also been proposed for ViT to reduce its model size and inference time.
These methods can be roughly grouped into two categories: 1) Channel pruning , which reduces the number of weights, channels, heads or blocks in ViT. SViTE [9] jointly optimizes parameters and explores connectivity for both unstructured pruning (zeroing weights) and structured pruning (removing heads and channels).
ViT-Slim [6] applies L 1 sparsity on channels and produces compressed ViTs with unstructured heads (the shape of heads is different).
UVC [61] drops heads, channels, and blocks in a uniﬁed framework to achieve a high compression ratio.
VTP [68] transfers the sparse-learning scheme in CNN pruning to compress ViT. NViT [60] generates smaller networks from the DeiT-base with the Taylor-based pruning scheme.
Token pruning , which focuses on dynamically selecting signiﬁcant tokens for different inputs.
Token pruning would signiﬁcantly reduce the computational cost while maintaining all parameters.
TokenLearner [44] adaptively generates a small set of token vectors according to the spatial attention.
EViT [28] downsamples tokens every three blocks and selects tokens with high correlation with the CLS token.
DynamicViT [41] estimates the importance of tokens with an MLP [51] based predictor.
IA-RED 2 [36] introduces a multi-head interpreter to drop uninformative tokens.
SP-ViT [25] softly prunes tokens with token selector modules and packages the redundant tokens into one.
Different from previous methods, this paper compresses ViTs from a novel prospect, frequency domain, to prune both parameters and tokens in a uniﬁed framework.
Frequency domain analysis for ViT and CNN.
The recent explorations [3, 55, 42, 38, 52] of ViTs have indicated that ViTs behave in an opposite way to CNNs in frequency domain.
[3] ﬁnds out that
ViTs perform better than CNNs when only low-frequency components of images are fed into the models, and proposes the HAT method to enhance the capability of ViTs in capturing high-frequency information.
[55] analyzes ViT features from the Fourier spectrum domain and shows that the selfattention module amounts to a low-pass ﬁlter.
[38] also demonstrates that multi-head self-attentions exhibit opposite behaviors to convolutions, and take advantage of both mechanisms to design a novel
AlterNet.
To summarize, all these studies point out that the low-frequency components play an important role in information extraction of ViT.
Methodology 3.1 Preliminary
The necessary notations are deﬁned as below.
As shown in Figure 2, a transformer block contains a multi-head self-attention (MHSA) module with multiple heads and a feed-forward network (FFN) module with two fully-connection layers.
The input images in a mini-batch are denoted as X ∈
R B × 3 × H × W , where B , W and H are the batch size, width and height of images, respectively.
The inputs of MHSA and FFN in the l -th block are denoted as X l, 1 ∈ R B × N l × D and X l, 2 ∈ R B × N l × D , respectively.
N l is the number of tokens, and D is the dimension of a token.
In the l -th block, the linear projection matrices W l,hq , W l,hk , and W l,hv are used to calculate Q l,h (query), K l,h (key), and
V l,h (value) for the h -th attention head.
The parameters of the linear projection module in MHSA are denoted as W lproj , and two linear projection matrices in FFN are W lfc 1 and W lfc 2 .
Our goal is to reduce the channel number of linear projection matrices and the token number N l .
𝑊 !",$ 𝑊 %",$ 𝑊 &",$ 𝑄 ",$ 𝐾 ",$ 𝑉 ",$ 𝑊 !",$ 𝑊 %",$ 𝑊 &",$ 𝑄 ",$ 𝐾 ",$ 𝑉 ",$
L a y e r
N o r m 𝑊 !",$ 𝑊 %",$ 𝑊 &",$ 𝑄 ",$ 𝐾 ",$ 𝑉 ",$ 𝑊 ’()*"
L a y e r
N o r m
C on ca t
H heads 𝑋 ",
[Figure]
Multi
[Figure]
Self Attention (MHSA)
Feed-Forward Network (FFN) 𝑋 ",,
[Figure]
[Figure]
𝑊
[Figure]
ViT Block
[Figure] [Figure]
3.2 Channel Pruning based on Low-Frequency Sensitivity
As previously introduced, low-frequency components in images are more valuable for the feature representation in ViT models, i.e. channels less effective in encoding low-frequency components will contribute less to the feature representation.
Therefore, the key goal is to estimate the sensitivity of a channel to low-frequency components in images.
To achieve this, we propose an evaluation policy named low-frequency sensitivity (LFS) that estimates the importance scores of model parameters by taking more low-frequency components in images into account.
Assume redundant channels have less inﬂuence on model outputs, removing redundant channels should hardly change the value of loss when feeding a set of training images into the model for loss computation.
Thus, the importance of a channel can be quantiﬁed by the difference in loss induced by removing this channel.
Given a number of images X ∈ R B × 3 × H × W randomly sampled from the training dataset D , the importance score I j of a weight w j is formulated as:
[Equation]
where Y ∈ R B × 1 is the label set of data X , L ( · ) denotes the loss function (cross-entropy loss in this paper), M ( X, W ) is the model output, and W indicates all model weights.
However, the score in Equation 1 can only reﬂect the importance on the original whole images.
To separate low-frequency components from images, low-pass ﬁltering is applied on images in the
Fourier spectrum domain before feeding them into ViTs.
The low-frequency components in images ˜
X are formulated as:
[Equation]
where F ( · ) and F − 1 ( · ) denote the fast Fourier transformation (FFT) [40] and the inverse fast Fourier transform (IFFT), respectively, (cid:12) is the Hadamard product, G ( · ) is the low-pass ﬁlter, and σ c ∈ (0 , 1) determines the cutoff frequency of the low-pass ﬁlter which is similar to the radial averaging of the 2D Fourier spectrum as in [16, 15, 5, 46].
Considering that a binary ﬁlter will cause the Ringing effect when the image is transformed back to the spatial domain, Gaussian ﬁlter is chosen for G ( · ) .
In addition to the task-speciﬁc loss, the pruned model shall also provide robust feature representation as the original model.
In other words, the feature representation of the low-frequency images ˜ X shall be as close to that of the original images X as possible.
Hence, apart from the crossentropy loss L for the classiﬁcation task, a knowledge-distillation loss is also taken into account.
Kullback–Leibler (KL) divergence loss KL ( · ) is used to measure the error between the CLS tokens corresponding to the low-frequency image and nature image, respectively.
Denote the two CLS tokens as ˜ T (from low-frequency images) and T (from nature images), and simplify the crossentropy loss L ( M ( X, W ) , Y ) to L ( X ) .
Then, the ﬁnal importance score s j of weight w j , named Low-Frequency Sensitivity (LFS), is formulated as:
[Equation]
where λ is the hyper-parameter for the balance of two loss functions.
Calculating the LFS for each parameter with Equation 3 is infeasible for models with millions of parameters.
Fortunately, the score can be approximated with the ﬁrst-order Taylor expansion [34, 60].
Therefore, the approximated version of LFS is represented as below:
[Equation]
where the gradient terms can be easily obtained in the backward procedure of the model.
The channel importance score can then be approximated by summing over LFS scores of all parameters in the channel, i.e. the LFS of a channel is computed by the sum of ˆ s
[Equation]
where J means the index set of weights in a channel.
3.3 Token Pruning based on Low-Frequency Energy
Token redundancy is another major issue in the ViT compression, and several methods [47, 28, 25] sample informative tokens via analyzing the relationship or attention scores between tokens.
Such a solution is sub-optimal because the selected tokens tend to be similar, and the information included in the token itself has been neglected to some extent.
To address this problem, the Low-Frequency
Energy (LFE) is proposed to make use of the low-frequency preference of ViT for token pruning.
Following other works [47, 28, 25], as shown in Figure 2, the selector is located between the multihead self-attention module and the feed-forward network module.
Inspired by [55], we evaluate the low-frequency ratio of the token after transforming tokens X l, 2 into the frequency domain by applying FFT on each channel of tokens, denoted as X l, 2 b, : ,j = F (cid:16) X l, 2 b, : ,j (cid:17) .
We then quantify the low-frequency information contained in a token by calculating the ratio of its remaining energy to the total energy after low-pass ﬁltering.
Given ﬁlter G with cutoff factor σ t , the LFE is formulated as:
[Equation]
(cid:13)(cid:13)(cid:13) [Equation] where DC [ · ] and LC [ · ] denote the direct-current component and the low-frequency component, respectively.
Intuitively, a token with more low-frequency components will achieve a larger η l,i .
Similar to EViT [28], the attention scores in the spatial domain is also included to evaluate the ﬁnal importance scores of tokens.
For the h -th head in ViT, the attention value is calculated as:
[Equation]
where A l,h is the attention score matrix and d l,h is the output dimension.
The CLS token plays a more signiﬁcant role than other tokens because it is the ﬁnal output feature which collects information from all tokens.
Moreover, the head with denser and larger attention values is more important, i.e. , with a larger proportion.
Thus, our proposed modiﬁed attention score is formulated as:
[Equation]
where θ h, 0 = (cid:80) N − 1 j =
A l,h 0 ,j and θ h, 1 =
A l,h 0 , 0 are the head-weights of the CLS attention value
A l,hi, 0 and the other attention value
A l,hi,j , respectively.
To estimate the importance score of tokens from multiple and diverse aspects, we consider to combine the LFE η l,i and attention score ˆ T l,i to get the ﬁnal importance score of a token as:
[Equation]
Note: the CLS token is the ﬁnal output of the ViT model, and is not involved in the token pruning.
3.4 Bottom-up Cascade Pruning
BlockBlock-
Block-l
Block-L … … …
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
Information in frequency domain
Filtering in frequency domain
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
Information in frequency domain
Filtering in frequency domain
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
Information in frequency domain
Filtering in frequency domain
BlockBlock-
Block-l Block
[Figure]
In our method, the value of LFS and LFE in each block is related to the outputs from previous blocks.
Hence, the compression in one block will inﬂuence the LFS and LFE in its subsequent blocks.
It is sub-optimal to independently determine the pruning ratios and indices of channels and tokens for all blocks at once and ignore their inter-relationship.
Therefore, we design a Bottom-up Cascade
Pruning (BCP) process (Figure 3), which promotes pruning from the ﬁrst block to the last block.
A hyper-parameter, named global allowable drop ε , is set to control the ﬁnal performance drop after pruning.
During compression of each block, the number of tokens is gradually reduced until the performance drop reaches ε t = ρ · ε/L , where ρ is a hyper-parameter to control the accuracy drop caused by token pruning.
Once token pruning is done, channels will be compressed with a similar procedure.
When the performance drop caused by pruning reaches ε/L , the compression for a block is considered as completed.
More details are described in Algorithm 1 in Appendix A.1.
Experiments
In this section, the proposed method is evaluated on the benchmark ImageNet (ILSVRC2012) [43], which is a large dataset containing 1.2M training images and 50k validation images of 1000 classes.
All the experiments are deployed with Pytorch [39] on NVIDIA V100 GPUs.
The code is modiﬁed based on the previous study DeiT 2 .
The ﬂoat operations (FLOPs) of models are evaluated by fvcore
4.1 Experiments on ImageNet
Implementation details.
The proposed method is applied to popular ViT models of three different sizes, DeiT-Tiny, DeiT-Small, and DeiT-Base.
The latest state-of-the-art (SOTA) methods are compared, including SCOP [49], CP-ViT [47], PoWER [18], HVT [37], IA-RED 2 [36], S 2 ViTE [9], EViT [28], and SPViT [21].
In the pruning procedure, the number of training samples used for evaluating the performance drop in BCP is 5000 (randomly sampling 5 training samples from each category), the number of training samples for calculating LFS is 2000, and the cutoff factors σ c and σ t are 0 .
1 and 0 . 85 .
For three models, DeiT-Tiny, DeiT-Small, and DeiT-Base, the global allowable drop ε are 9 . 5 , 14 , and 14 , and the ratio ρ for the allowable drop is 0 .
56 , 0 . 35 , and 0 .
3 respectively.
The removed channels involve the columns (output channels) of W l,hq , W l,hq , W l,hq , and W lfc 1 and rows (input channels) of W lproj , and W lfc 2 .
After pruning, the compressed models are ﬁne-tuned with hard distillation [50] of their corresponding original models.
The base learning rate is set to 0.0001, and most of the other hyper-parameters follow the settings in [9].
We ﬁne-tune the pruned
DeiT-Tiny/DeiT-Small/DeiT-Base models for 300/150/150 epochs.
More detailed settings and results of different epochs are listed in Appendix A.3.
Results and analysis.
The comparison with state-of-the-art methods is shown in Table 1, in which the top-1 accuracy and the reduction ratios of FLOPs are reported.
In all three models, the proposed method achieves the highest reduction ratio in FLOPs than previous methods with less than 1% performance drop.
Compared to the pure token pruning methods like SCOP [49], CP-ViT [47],
PoWER [18], HVT [37], and IA-RED 2 [36], our method achieves not only better performance and more reduction in FLOPs, but also less parameters, which demonstrates the superiority of pruning with frequency domain.
Although EViT [28], SPViT [21], and S 2 ViTE [9] achieves better accuracy on DeiT-Base, their reduction in FLOPs (EViT 34.1%, SPViT 33.1%, S 2 ViTE 33.1% vs Our 57.6%) is much lower than ours.
To evaluate the acceleration on inference speed of our pruning technique, the throughput is assessed on a single V100 GPU with batch size 256 in Table 2.
The DeiT-Tiny/Small/Base model achieves 69.7%/97.0%/107.1% speed up after pruning, which demonstrates the practicability of the proposed compression approach. [Table] EViT [28] − / − − [Table]
HVT [37]
[Table]
IA
[Table]
RED 2 [36]
[Table] [Table] [Table] [Table]
[Figure]
4.2 Ablation Study
Effectiveness of LFS and LFE.
For fair comparison to analyze the effectiveness of the proposed
LFS (for channel pruning) and LFE (for token pruning), we conduct experiments without using
BCP framework in Table 3.
Two state-of-the-art methods NViT
[60] and EViT [28] are selected as baselines for channel pruning and token pruning, respectively.
NViT identiﬁes channels based on the Taylor score and EViT selects tokens according to the attention score.
The channels are pruned globally using the manual pruning rate as NViT, and the number of tokens is determined following the same ratio as EViT.
The main difference between other methods and ours is whether to leverage the characteristics of ViT on low-frequency components.
Table 3 shows the results of only compressing channels, tokens, and both.
For channel pruning, it can be found that LFS outperforms
NViT ∗ with the same FLOPs reduction.
For token pruning, LFE achieves an even higher accuracy than the original DeiT-Small.
Both comparison results demonstrate the superiority of pruning based on information in the frequency domain over using only information in the space domain.
The results of different pruning ratios for channels and tokens are displayed in Figure 4(a) and 4(b), in which the proposed method consistently outperform the other methods under all ratios.
The distributions of importance scores for models are displayed in Appendix A.10 Fig.
The inﬂuence of the cutoff factor, which determines the ratio of saved low-frequency components to all frequency, is also analyzed.
As shown in Figure 4(c), σ c = 0 .
1 is the sweet spot, which further proves the preference of ViT on low-frequency information.
For σ t , 0 . 85 is the best choice.
[Table]
Factor analysis for BCP.
The pruning ratios for channels and tokens are determined by two hyperparameters ε and ρ .
As shown in Table 4, when maintaining ρ = 0 .
35 and increasing ε from 10 to 18, the model parameter size will be reduced from 18.7M to 16.8M, and the performance ranges from 79.8% to 78.9%.
Similarly, when maintaining ε = 14 and increasing ρ from 0.15 to 0.55, both the model size and the FLOPS reduction ratio are increased, while the performance is reduced from 79.9% to 78.7%.
Compared with the result (78.7% accuracy and 55.0% Flops reduction ratio) without
BCP in the last row of Table 3, we can observe that BCP improves the accuracy or compression ratio of the model, which veriﬁes the effectiveness of the proposed BCP strategy.
After comprehensively considering the model parameters size, the FLOPS reduction ratio and model accuracy, we set the ε = 14 and ρ = 0 .
35 in this paper.
These two parameters can be adjusted according to speciﬁc requirements in real-world applications.
Inﬂuence of each module.
To further study how each component affects the proposed method, LFS,
LFE, and BCP are respectively removed from our scheme.
The modiﬁed versions are then applied on the same model.
Since BCP will automatically adjust pruning ratios of each block, it is necessary to introduce additional variables for controlled experiments.
For a fair comparison, we keep the same pruning ratios (determined by our VTC-LFC) of each block in all experiments.
As shown in Table 5, the proposed LFS/LFE/BCP improves the performance by 1.3%/0.6%/0.9% before ﬁne-tuning and 0.1%/0.5%/0.2% after ﬁne-tuning, respectively.
The experimental results show the effectiveness of the proposed modules.
[Table]
Inﬂuence of pruning sequence.
The main reason for pruning from the ﬁrst block to the last one is that the pruning in previous blocks will change the inputs of their subsequent blocks, which may inﬂuence the value of our proposed LFS and LFE.
If the last block is pruned ﬁrst, the selected channels and tokens in this block will need to be re-adjusted when former blocks are compressed.
Instead, if the former blocks are pruned ﬁrstly, the inputs for the subsequent blocks are ﬁxed.
Both pipelines are executed and compared on DeiT-Small, and the results shown in Table 6 demonstrate the advantage of the proposed buttom-up (from ﬁrst to last) pipeline.
[Table]
Inﬂuence of hyper-parameter λ .
The hyper-parameter λ used to balance different terms in LFS during compression is analyzed here.
As listed in Table 7, the models pruned with different λ values are evaluated after pruning.
It can be found that the model achieves the best accuracy when λ is 0.1.
In addition, the proposed method is not very sensitive to the value of λ ( λ > 0 ).
[Table]
Compression on other ViT models.
In addition to DeiT models, the proposed method is also evaluated on LV-ViT [24] and window-based attention model
Swin [32], in which the pruned models are ﬁne-tuned for 150 epochs, respectively.
Speciﬁcally, due to the token downsampling and the window shifting, token pruning is not adapted to Swin yet so that only channel pruning is adopted on Swin.
To simplify, the token labels in the original LV-ViT are not used during ﬁnetuning.
For LV-ViT, our method obtains 0.3% higher (81.8% vs 81.5%) performance and 0.1G lower
FLOPs (3.2G vs 3.3G) than the combination of existing channel pruning (NViT) and token pruning (EViT) approaches.
On Swin, the proposed VTC-LFC achieves 0.2/3.1% higher accuracy than the previous SPViT [21]/STEP [27] with fewer FLOPs (3.3G vs 3.4G/3.5G) and parameters (17.1M vs 25.8M/23.6M).
The results shown in Table 8 demonstrate that proposed method can still achieve better performance as well as lower FLOPs than previous approaches on other ViT architectures.
[Table]
Conclusion and Discussion
This paper reveals the disadvantages of pruning ViTs only in the spatial domain and takes advantage of the preference of ViTs for low-frequency information to conduct compression.
Two metrics, lowfrequency sensitivity and low-frequency energy are proposed to leverage knowledge in the frequency domain for better channel pruning and token pruning.
The comparison with the spatial domain pruning approaches proves that the proposed method can identify the informative channels and tokens more precisely, thus better maintaining the model accuracy.
Additionally, with the proposed bottomup cascade pruning strategy, both channels and tokens are automatically compressed in a uniﬁed framework.
Extensive experiments of different models on ImageNet demonstrate that more than half of computational costs are saved from ViTs, with signiﬁcant improvements in inference efﬁciency.
The comparison with the latest compression methods shows the superiority (better performance and more FLOPs reduction) of the proposed approach.
As a preliminary study about pruning ViTs with the frequency domain, more efﬁcient ways and deeper studies are expected to be explored based on it.
Acknowledgement
This work was supported by Alibaba Group through Alibaba Research Intern Program.