VTC-LFC: Vision Transformer Compression with Low-Frequency Components
Abstract
Although Vision transformers (ViTs) have recently dominated many vision tasks, deploying ViT models on resource-limited devices remains a challenging problem.
To address such a challenge, several methods have been proposed to compress ViTs.
Most of them borrow experience in convolutional neural networks (CNNs) and mainly focus on the spatial domain.
However, the compression only in the spatial domain suffers from a dramatic performance drop without finetuning and is not robust to noise, as the noise in the spatial domain can easily confuse the pruning criteria, leading to some parameters/channels being pruned incorrectly.
Inspired by recent findings that self-attention is a low-pass filter and low-frequency signals/components are more informative to ViTs, this paper proposes compressing ViTs with low-frequency components.
Two metrics named low-frequency sensitivity (LFS) and low-frequency energy (LFE) are proposed for better channel pruning and token pruning.
Additionally, a bottom-up cascade pruning scheme is applied to compress different dimensions jointly.
Extensive experiments demonstrate that the proposed method could save 40% ∼ 60% of the FLOPs in ViTs, thus significantly increasing the throughput on practical devices with less than 1% performance drop on ImageNet-1K. Code will be available at https://github.com/Daner-Wang/VTC-LFC.git .
Introduction
Recently, Vision transformer (ViT) [14] and its variants [50, 32, 62] have outperformed convolutional neural networks (CNNs) in several vision tasks.
However, ViT models still face the challenge of high computational cost when deployed to resource-limited devices.
Following previous experiences in compressing CNN models, some pruning methods based on sparse learning [68, 61], taylor expansion [60], or automatic searching [9] have been proposed for ViT models to reduce model redundancy via channel pruning.
In addition to the redundancy in parameters, recent literature [48, 28, 41] further points out that some noise tokens mainly encoded task-irrelevant information ( e.g. , background), and some tokens become similar in deeper layers, showing that great redundancy also exists in tokens.
The mainstream works [28, 47, 36] filter out the less informative tokens to reduce the FLOPs without changing the model structure.
[Figure] Although aforementioned methods have made great progress in ViT compression in spatial domain, we find that they generally suffer from the following two problems: (i) different from CNN pruning which maintains the performance well without finetuning [34, 13], dramatic performance drop is observed when the same method is applied in ViT pruning; (ii) conducting ViT pruning only in spatial domain is not robust to noise, and as shown in Figure 1, after adding noise in the images, the accuracy of spatial compression dramatically drops.
To make ViT compression more effective and robust, we propose to conduct ViT compress with the help of frequency domain.
Recent studies [3, 55, 42, 38, 52] have indicated that self-attention (SA) behaves like a low-pass filter, and low-frequency signals/components are more informative to ViT models.
Inspired by such low-frequency characteristics of ViT, we propose a compression framework named Vision Transformer Compression with
Low-Frequency Components (VTC-LFC) which solves the problem from a new angle and emphasizes the contributions of low-frequency components during compression.
To our best knowledge, this is the first work that compresses vision transformers in the frequency domain.
The main contributions of this paper are listed as below:
Channel pruning based on low-frequency sensitivity: Channel pruning is a popular structured pruning strategy that aims
to
remove
redundant
parameters
in
fully
connected
layers of ViT. The mainstream works use some evaluation metrics ( e.g. Taylor scores [60], weights norm [61], or sparse factor [68]) to estimate importance scores of parameters. Recent studies [3, 42, 52] find that ViTs are more reliable to the low-frequency components in images, i.e. the low-frequency information is more important for ViTs.
Therefore, we infer that channels that are less effective in encoding low-frequency components will contribute less to the feature representation for ViT models.
Motivated by such a property, we propose a better channel pruning criterion named low-frequency sensitivity (LFS) based on the Taylor scores [60].
Different from the standard Taylor scores which are computed with the original images, LFS filters out high-frequency components from images and uses only low-frequency components to estimate the importance of model parameters.
In this way, channels that efficiently encode low-frequency information are more likely to be preserved, and the compressed model tends to be more robust to noise.
Experimental results show that LFS can alleviate the performance drop after compression without bells and whistles.
Token pruning based on low-frequency energy:
Token compression/sampling aims to select the informative tokens that store more useful information.
The popular methods dynamically select those tokens with high correlation to other tokens ( e.g. the CLS token) as the informative tokens.
However, it may be sub-optimal because the selected tokens tend to be similar to each other, and the information included in the token itself has been neglected to some extent.
As pointed out by [55, 38], the self-attention module in ViTs behaves like a low-pass filter, i.e. the tokens with more low-frequency components can pass more information to the next layers.
Inspired by this, we propose improving the attention-based token selection with an extra item, token low-frequency energy (LFE), which quantifies the low-frequency information in tokens.
By correcting the attention scores with LFE of tokens, the selector can better distinguish informative tokens from both the long-term dependency in spatial domain and low-frequency contributions in frequency domain.
Bottom-up Cascade Pruning Framework: To jointly compress channels and tokens of vision transformers, we propose a bottom-up cascade pruning framework.
The model accuracy is further preserved through automatically balancing compression ratios of channel pruning and token pruning block-by-block.
Related work
Vision Transformer.
Inspired by the success of transformers [51] in NLP, the Vision Transformer (ViT) [14] is proposed to encode an image into a sequence of tokens and feed them into the pure transformer architecture.
Several studies have shown that ViT performs better than convolution neural networks (CNN) on image classification benchmarks [14, 20] when sufficient training data is provided.
Many follow-up variants of ViT [4, 7, 1, 19, 54, 10, 62, 53, 66] have also been proposed.
For example, DeiT [50] introduces a distillation token structure into ViT, and LV-ViT [24] proposes the token labeling approach for better training of ViT.
In addition to image classification, ViT has also achieved great performance in many other computer vision applications, such as semantic segmentation [11, 56, 12], image retrieval [22, 17], object detection [2, 69] and image reconstruction [8, 59].
However, despite of the outstanding performance in a series of tasks, its high computational cost restricts the deployment of ViT, which motivates the study of lightweight ViT models, including pruning [9, 68, 44, 58], block-weights sharing [26, 64], fast distillation [57], and dynamic prediction architecture [67, 45], among which pruning is a universal approach for almost all model structures.
ViT Pruning.
As an efficient compression approach, pruning [35, 23, 29, 30, 65, 33, 31] has been widely applied on various convolutional neural networks (CNNs) in computer vision.
Pruning approaches [63, 21, 58, 48] have also been proposed for ViT to reduce its model size and inference time.
These methods can be roughly grouped into two categories: 1) Channel pruning , which reduces the number of weights, channels, heads or blocks in ViT. SViTE [9] jointly optimizes parameters and explores connectivity for both unstructured pruning (zeroing weights) and structured pruning (removing heads and channels).
ViT-Slim [6] applies L 1 sparsity on channels and produces compressed ViTs with unstructured heads (the shape of heads is different).
UVC [61] drops heads, channels, and blocks in a unified framework to achieve a high compression ratio.
VTP [68] transfers the sparse-learning scheme in CNN pruning to compress ViT. NViT [60] generates smaller networks from the DeiT-base with the Taylor-based pruning scheme.
Token pruning , which focuses on dynamically selecting significant tokens for different inputs.
Token pruning would significantly reduce the computational cost while maintaining all parameters.
TokenLearner [44] adaptively generates a small set of token vectors according to the spatial attention.
EViT [28] downsamples tokens every three blocks and selects tokens with high correlation with the CLS token.
DynamicViT [41] estimates the importance of tokens with an MLP [51] based predictor.
IA-RED 2 [36] introduces a multi-head interpreter to drop uninformative tokens.
SP-ViT [25] softly prunes tokens with token selector modules and packages the redundant tokens into one.
Different from previous methods, this paper compresses ViTs from a novel prospect, frequency domain, to prune both parameters and tokens in a unified framework.
Frequency domain analysis for ViT and CNN.
The recent explorations [3, 55, 42, 38, 52] of ViTs have indicated that ViTs behave in an opposite way to CNNs in frequency domain.
[3] finds out that
ViTs perform better than CNNs when only low-frequency components of images are fed into the models, and proposes the HAT method to enhance the capability of ViTs in capturing high-frequency information.
[55] analyzes ViT features from the Fourier spectrum domain and shows that the selfattention module amounts to a low-pass filter.
[38] also demonstrates that multi-head self-attentions exhibit opposite behaviors to convolutions, and take advantage of both mechanisms to design a novel
AlterNet.
To summarize, all these studies point out that the low-frequency components play an important role in information extraction of ViT.
Methodology 3.1 Preliminary
The necessary notations are defined as below.
As shown in Figure 2, a transformer block contains a multi-head self-attention (MHSA) module with multiple heads and a feed-forward network (FFN) module with two fully-connection layers.
The input images in a mini-batch are denoted as X ∈
R B × 3 × H × W , where B , W and H are the batch size, width and height of images, respectively.
The inputs of MHSA and FFN in the l -th block are denoted as X l, 1 ∈ R B × N l × D and X l, 2 ∈ R B × N l × D , respectively.
N l is the number of tokens, and D is the dimension of a token.
In the l -th block, the linear projection matrices W l,hq , W l,hk , and W l,hv are used to calculate Q l,h (query), K l,h (key), and
V l,h (value) for the h -th attention head.
The parameters of the linear projection module in MHSA are denoted as W lproj , and two linear projection matrices in FFN are W lfc 1 and W lfc 2 .
Our goal is to reduce the channel number of linear projection matrices and the token number N l .
𝑊 !",$ 𝑊 %",$ 𝑊 &",$ 𝑄 ",$ 𝐾 ",$ 𝑉 ",$ 𝑊 !",$ 𝑊 %",$ 𝑊 &",$ 𝑄 ",$ 𝐾 ",$ 𝑉 ",$
L a y e r
N o r m 𝑊 !",$ 𝑊 %",$ 𝑊 &",$ 𝑄 ",$ 𝐾 ",$ 𝑉 ",$ 𝑊 ’()*"
L a y e r
N o r m
C on ca t
H heads 𝑋 ", [Figure]
Multi
[Figure]
Self Attention (MHSA)
Feed-Forward Network (FFN) 𝑋 ",,
[Figure]
[Figure]
ViT Block
[Figure] [Figure]
3.2 Channel Pruning based on Low-Frequency Sensitivity
As previously introduced, low-frequency components in images are more valuable for the feature representation in ViT models, i.e. channels less effective in encoding low-frequency components will contribute less to the feature representation.
Therefore, the key goal is to estimate the sensitivity of a channel to low-frequency components in images.
To achieve this, we propose an evaluation policy named low-frequency sensitivity (LFS) that estimates the importance scores of model parameters by taking more low-frequency components in images into account.
Assume redundant channels have less influence on model outputs, removing redundant channels should hardly change the value of loss when feeding a set of training images into the model for loss computation.
Thus, the importance of a channel can be quantified by the difference in loss induced by removing this channel.
Given a number of images X ∈ R B × 3 × H × W randomly sampled from the training dataset D , the importance score I j of a weight w j is formulated as:
[Equation]
where Y ∈ R B × 1 is the label set of data X , L ( · ) denotes the loss function (cross-entropy loss in this paper), M ( X, W ) is the model output, and W indicates all model weights.
However, the score in Equation 1 can only reflect the importance on the original whole images.
To separate low-frequency components from images, low-pass filtering is applied on images in the
Fourier spectrum domain before feeding them into ViTs.
The low-frequency components in images ˜ X are formulated as:
[Equation]
where F ( · ) and F − 1 ( · ) denote the fast Fourier transformation (FFT) [40] and the inverse fast Fourier transform (IFFT), respectively, ⊙ is the Hadamard product, G ( · ) is the low-pass filter, and σ c ∈ (0 , 1) determines the cutoff frequency of the low-pass filter which is similar to the radial averaging of the 2D Fourier spectrum as in [16, 15, 5, 46].
Considering that a binary filter will cause the Ringing effect when the image is transformed back to the spatial domain, Gaussian filter is chosen for G ( · ) .
In addition to the task-specific loss, the pruned model shall also provide robust feature representation as the original model.
In other words, the feature representation of the low-frequency images ˜ X shall be as close to that of the original images X as possible.
Hence, apart from the cross-entropy loss L for the classification task, a knowledge-distillation loss is also taken into account.
Kullback–Leibler (KL) divergence loss KL ( · ) is used to measure the error between the CLS tokens corresponding to the lowfrequency image and nature image, respectively.
Denote the two CLS tokens as ˜ T (from low-frequency images) and T (from nature images), and simplify the cross-entropy loss L ( M ( X, W ) , Y ) to L ( X ) .
Then, the final importance score s j of weight w j , named Low-Frequency Sensitivity (LFS), is formulated as:
[Equation]
where λ is the hyper-parameter for the balance of two loss functions.
Calculating the LFS for each parameter with Equation 3 is infeasible for models with millions of parameters.
Fortunately, the score can be approximated with the first-order Taylor expansion [34, 60].
Therefore, the approximated version of LFS is represented as below:
[Equation]
where the gradient terms can be easily obtained in the backward procedure of the model.
The channel importance score can then be approximated by summing over LFS scores of all parameters in the channel, i.e. the LFS of a channel is computed by the sum of ˆ s j :
[Equation]
where J means the index set of weights in a channel.
3.3 Token Pruning based on Low-Frequency Energy
Token redundancy is another major issue in the ViT compression, and several methods [47, 28, 25] sample informative tokens via analyzing the relationship or attention scores between tokens.
Such a solution is sub-optimal because the selected tokens tend to be similar, and the information included in the token itself has been neglected to some extent.
To address this problem, the Low-Frequency
Energy (LFE) is proposed to make use of the low-frequency preference of ViT for token pruning.
Following other works [47, 28, 25], as shown in Figure 2, the selector is located between the multihead self-attention module and the feed-forward network module.
Inspired by [55], we evaluate the low-frequency ratio of the token after transforming tokens X l, 2 into the frequency domain by applying FFT on each channel of tokens, denoted as X l, 2 b, : ,j = F (cid:16) X l, 2 b, : ,j (cid:17) .
We then quantify the low-frequency information contained in a token by calculating the ratio of its remaining energy to the total energy after low-pass filtering.
Given filter G with cutoff factor σ t , the LFE is formulated as:
[Equation]
(cid:13)(cid:13)(cid:13) X l, 2 b, : , : (cid:13)(cid:13)(cid:13), (6) where DC [ · ] and LC [ · ] denote the direct-current component and the low-frequency component, respectively.
Intuitively, a token with more low-frequency components will achieve a larger η l,i .
Similar to EViT [28], the attention scores in the spatial domain is also included to evaluate the final importance scores of tokens.
For the h -th head in ViT, the attention value is calculated as:
[Equation]
where A l,h is the attention score matrix and d l,h is the output dimension.
The CLS token plays a more significant role than other tokens because it is the final output feature which collects information from all tokens.
Moreover, the head with denser and larger attention values is more important, i.e. , with a larger proportion.
Thus, our proposed modified attention score is formulated as:
[Equation]
where θ h, 0 = (cid:80) N − 1 j =
A l,h 0 ,j and θ h, 1 =
A l,h 0 , 0 are the head-weights of the CLS attention value
A l,hi, 0 and the other attention value
A l,hi,j , respectively.
To estimate the importance score of tokens from multiple and diverse aspects, we consider to combine the LFE η l,i and attention score ˆ T l,i to get the final importance score of a token as:
[Equation]
Note: the CLS token is the final output of the ViT model, and is not involved in the token pruning.
3.4 Bottom-up Cascade Pruning
BlockBlock-
Block-l
Block-L … … …
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
Information in frequency domain
Filtering in frequency domain
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
Information in frequency domain
Filtering in frequency domain
BlockBlock-
Block-l
Block-L … …
BlockBlock-
Block-l
Block-L … …
Information in frequency domain
Filtering in frequency domain
BlockBlock- Block-l Block-L …
[Figure]
In our method, the value of LFS and LFE in each block is related to the outputs from previous blocks.
Hence, the compression in one block will influence the LFS and LFE in its subsequent blocks.
It is sub-optimal to independently determine the pruning ratios and indices of channels and tokens for all blocks at once and ignore their inter-relationship.
Therefore, we design a Bottom-up Cascade
Pruning (BCP) process (Figure 3), which promotes pruning from the first block to the last block.
A hyper-parameter, named global allowable drop ε , is set to control the final performance drop after pruning.
During compression of each block, the number of tokens is gradually reduced until the performance drop reaches ε t = ρ · ε/L , where ρ is a hyper-parameter to control the accuracy drop caused by token pruning.
Once token pruning is done, channels will be compressed with a similar procedure.
When the performance drop caused by pruning reaches ε/L , the compression for a block is considered as completed.
More details are described in Algorithm 1 in Appendix A.1.
Experiments
In this section, the proposed method is evaluated on the benchmark ImageNet (ILSVRC2012) [43], which is a large dataset containing 1.2M training images and 50k validation images of 1000 classes.
All the experiments are deployed with Pytorch [39] on NVIDIA V100 GPUs.
The code is modified based on the previous study DeiT 2 .
The float operations (FLOPs) of models are evaluated by fvcore 3 .
4.1 Experiments on ImageNet
Implementation details.
The proposed method is applied to popular ViT models of three different sizes, DeiT-Tiny, DeiT-Small, and DeiT-Base.
The latest state-of-the-art (SOTA) methods are compared, including SCOP [49], CP-ViT [47], PoWER [18], HVT [37], IA-RED 2 [36], S 2 ViTE [9], EViT [28], and SPViT [21].
In the pruning procedure, the number of training samples used for evaluating the performance drop in BCP is 5000 (randomly sampling 5 training samples from each category), the number of training samples for calculating LFS is 2000, and the cutoff factors σ c and σ t are 0 . 1 and 0 . 85 .
For three models, DeiT-Tiny, DeiT-Small, and DeiT-Base, the global allowable drop ε are 9 . 5 , 14 , and 14 , and the ratio ρ for the allowable drop is 0 .
55 , 0 . 35 , and 0 .
3 respectively.
The removed channels involve the columns (output channels) of W l,hq , W l,hq , W l,hq , and W lfc 1 and rows (input channels) of W lproj , and W lfc 2 .
After pruning, the compressed models are fine-tuned with hard distillation [50] of their corresponding original models.
The base learning rate is set to 0.0001, and most of the other hyper-parameters follow the settings in [9].
We fine-tune the pruned
DeiT-Tiny/DeiT-Small/DeiT-Base models for 300/150/150 epochs.
More detailed settings and results of different epochs are listed in Appendix A.2.
Results and analysis.
The comparison with state-of-the-art methods is shown in Table 1, in which the top-1 accuracy and the reduction ratios of FLOPs are reported.
In all three models, the proposed method achieves the highest reduction ratio in FLOPs than previous methods with less than 1% performance drop.
Compared to the pure token pruning methods like SCOP [49], CP-ViT [47],
PoWER [18], HVT [37], and IA-RED 2 [36], our method achieves not only better performance and more reduction in FLOPs, but also less parameters, which demonstrates the superiority of pruning with frequency domain.
Although EViT [28], SPViT [21], and S 2 ViTE [9] achieves better accuracy on DeiT-Base, their reduction in FLOPs (EViT 34.1%, SPViT 33.1%, S 2 ViTE 33.1% vs Our 57.6%) is much lower than ours.
To evaluate the acceleration on inference speed of our pruning technique, the throughput is assessed on a single V100 GPU with batch size 256 in Table 2.
The DeiT-Tiny/Small/Base model achieves 69.7%/97.0%/107.1% speed up after pruning, which demonstrates the practicability of the proposed compression approach.
[Table] [Table]
[Table]
[Figure]
4.2 Ablation Study
Effectiveness of LFS and LFE.
For fair comparison to analyze the effectiveness of the proposed
LFS (for channel pruning) and LFE (for token pruning), we conduct experiments without using
BCP framework in Table 3.
Two state-of-the-art methods NViT [60] and EViT [28] are selected as baselines for channel pruning and token pruning, respectively.
NViT identifies channels based on the Taylor score and EViT selects tokens according to the attention score.
The channels are pruned globally using the manual pruning rate as NViT, and the number of tokens is determined following the same ratio as EViT.
The main difference between other methods and ours is whether to leverage the characteristics of ViT on low-frequency components.
Table 3 shows the results of only compressing channels, tokens, and both.
For channel pruning, it can be found that LFS outperforms
NViT ∗ with the same FLOPs reduction.
For token pruning, LFE achieves an even higher accuracy than the original DeiT-Small.
Both comparison results demonstrate the superiority of pruning based on information in the frequency domain over using only information in the space domain.
The results of different pruning ratios for channels and tokens are displayed in Figure 4(a) and 4(b), in which the proposed method consistently outperform the other methods under all ratios. The distributions of LFS and LFE for models are displayed in Appendix A.10 Fig.
The influence of the cutoff factor, which determines the ratio of saved low-frequency components to all frequency, is also analyzed.
As shown in Figure 4(c), σ c = 0 .
1 is the sweet spot, which further proves the preference of ViT on low-frequency information.
For σ t , 0 . 85 is the best choice. [Table]
Factor analysis for BCP.
The pruning ratios for channels and tokens are determined by two hyperparameters ε and ρ .
As shown in Table 4, when maintaining ρ = 0 .
35 and increasing ε from 10 to 18, the model parameter size will be reduced from 18.7M to 16.8M
[Table]
and the performance ranges from 79.8% to 78.9%.
Similarly, when maintaining ε = 14 and increasing ρ from 0.15 to 0.55, both the model size and the FLOPS reduction ratio are increased, while the performance is reduced from 79.9% to 78.7%.
Compared with the result (78.7% accuracy and 55.0% Flops reduction ratio) without
BCP in the last row of Table [Table] we can observe that BCP improves the accuracy or compression ratio of the model, which verifies the effectiveness of the proposed BCP strategy.
After comprehensively considering the model parameters size, the FLOPS reduction ratio and model accuracy, we set the ε = 14 and ρ = 0 .
35 in this paper.
These two parameters can be adjusted according to specific requirements in real-world applications.
Influence of each module.
To further study how each component affects the proposed method, LFS,
LFE, and BCP are respectively removed from our scheme.
The modified versions are then applied on the same model.
Since BCP will automatically adjust pruning ratios of each block, it is necessary to introduce additional variables for controlled experiments.
For a fair comparison, we keep the same pruning ratios (determined by our VTC-LFC) of each block in all experiments.
As shown in Table 5, the proposed LFS/LFE/BCP improves the performance by 1.3%/0.6%/0.9% before fine-tuning and 0.1%/0.5%/0.2% after fine-tuning, respectively.
The experimental results show the effectiveness of the proposed modules.
[Table]
Influence of pruning sequence.
The main reason for pruning from the first block to the last one is that the pruning in previous blocks will change the inputs of their subsequent blocks, which may influence the value of our proposed LFS and LFE.
If the last block is pruned first, the selected channels and tokens in this block will need to be re-adjusted when former blocks are compressed.
Instead, if the former blocks are pruned firstly, the inputs for the subsequent blocks are fixed.
Both pipelines are executed and compared on DeiT-Small, and the results shown in Table 6 demonstrate the advantage of the proposed buttom-up (from first to last) pipeline
[Table]
[Table]
Influence of hyper-parameter λ .
The hyper-parameter λ used to balance different terms in LFS during compression is analyzed here.
As listed in Table 7, the models pruned with different λ values are evaluated after pruning.
It can be found that the model achieves the best accuracy when λ is 0.1.
In addition, the proposed method is not very sensitive to the value of λ ( λ > 0 ).
[Table]
Compression on other ViT models.
In addition to DeiT models, the proposed method is also evaluated on LV-ViT and window-based attention model Swin, in which the pruned models are fine-tuned for 150 epochs, respectively.
Specifically, due to the token downsampling and the window shifting, token pruning is not adapted to Swin yet so that only channel pruning is adopted on Swin.
To simplify, the token labels in the original LV-ViT are not used during fine-tuning.
For LV-ViT, our method obtains 0.3% higher (81.8% vs 81.5%) performance and 0.1G lower FLOPs (3.2G vs 3.3G) than the combination of existing channel pruning (NViT) and token pruning (EViT) approaches.
On Swin, the proposed VTC-LFC achieves 0.2/3.1% higher accuracy than the previous SPViT [21]/STEP [27] with fewer FLOPs (3.3G vs 3.4G/3.5G) and parameters (17.1M vs 25.8M/23.6M).
The results shown in Table 8 demonstrate that proposed method can still achieve better performance as well as lower
FLOPs than previous approaches on other ViT architectures.
[Table]
Conclusion and Discussion
This paper reveals the disadvantages of pruning ViTs only in the spatial domain and takes advantage of the preference of ViTs for low-frequency information to conduct compression.
Two metrics, lowfrequency sensitivity and low-frequency energy are proposed to leverage knowledge in the frequency domain for better channel pruning and token pruning.
The comparison with the spatial domain pruning approaches proves that the proposed method can identify the informative channels and tokens more precisely, thus better maintaining the model accuracy.
Additionally, with the proposed bottomup cascade pruning strategy, both channels and tokens are automatically compressed in a unified framework.
Extensive experiments of different models on ImageNet demonstrate that more than half of computational costs are saved from ViTs, with significant improvements in inference efficiency.
The comparison with the latest compression methods shows the superiority (better performance and more FLOPs reduction) of the proposed approach.
As a preliminary study about pruning ViTs with the frequency domain, more efficient ways and deeper studies are expected to be explored based on it [Table]  Acknowledgement
This work was supported by Alibaba Group through Alibaba Research Intern Program.
A Appendix A.1 Detailed procedure of bottom-up cascade pruning
The procedure of the proposed bottom-up cascade pruning (BCP) is illustrated here.
As shown in
Algorithm 1, the BCP greedily prunes channels and tokens block-by-block until the performance drop reaches the default threshold (allowable drop ε ).
Algorithm 1 Bottom-up Cascade Pruning.
and set D l ; 17: remove p channels with the smallest sensitivity from current block; 18: get accuracy of pruned model: ψ l = Acc ( M prune , D test ) ; 19: update p with p = min (cid:0) ⌈ (2 − ( ψ best − ψ l ) /ε t ) · p ⌉ , ⌈ p + ( n lc − p ) / 4 ⌉ , n lc (cid:1) ; 20: end while 21: remove p channels with the smallest sensitivity from current block; 22: get accuracy of pruned model: ψ l =
Acc ( M prune , D test ) ; 23: update ψ best = ψ l 24: end for
Output:
The compressed ViT model, M prune ;
A.2 Distribution of selected tokens
[Figure]
In the introduction, we have indicated that selecting tokens only with attention values may maintain many similar tokens.
To give a better illustration, the distribution of selected tokens with attention- based approach (EViT) and our proposed LFE are visualized in Figure 5.
it is obviously that our method can obtain diverse tokens rather than similar tokens.
A.3 Details of pruning and fine-tuning
The detailed hyper-parameters of pruning and fine-tuning are reported in Table 9 and Table 10, so that our work can be reproduced by other researchers.
The results of fine-tuning with different epochs are listed in Table
[Table]
[Table]
[Table]
[Table]
DeiT-Small 78.6/94.
DeiT-Base 81.2/95.
A.4 Approximation of LFS
Here, we provide the detailed derivation for the approximated low-frequency sensitivity (LFS)
ˆ s j .
Given complete LFS s j formulated as:
[Equation]
By performing the Taylor expansion, the loss functions L ( ˜ X | w j = 0) and KL ( ˜ T, T | w j = 0) can be rewritten as:
[Equation]
where θ ( L ) and θ ( KL ) are the sum of high-order terms.
Considering that the first-order expansion has had high correlation to the original loss function and the calculation for the higher-order terms is inefficient, only the zero-order and first-order terms are maintained:
[Equation]
Substituting this expansion into the complete LFS, the approximated LFS ˆ s j is formulated as:
[Equation]
A.5 Resistance of pruned models to different noises
In addition to the salt-and-pepper noise shown in the introduction, the resistance of models to other noises (including Gaussian noise, exponential noise, and uniform noise) are also evaluated.
As shown in Figure 6, for all kinds of noises, the performance of our pruned models drops slower, which demonstrates the robustness of channels selected by our method.
[Figure]
A.6 Throughput of different models
[Table]
A
7 Performances on ImageNet-Real and CIFARApart from ImageNet-1k, the pruned models are also evaluated on ImageNet-Real and CIFAR-10.
For CIFAR-10, the compressed DeiT-Tiny/Small/Base is fine-tuned for 300/150/75 epochs.
The results shown in Table 13 demonstrates that the pruned models perform well on other datasets.
A.8 Spectrum of channels
The spectra of output features corresponding to saved channels and pruned channels are visualized in Figure 7, where the map with green box is saved while the map with red box is pruned.
The
[Table]
baseline method (NViT) without taking advantage of characteristic in frequency domain is selected for comparison.
Although selection of two methods in channels is similar, our method tends to save the channel with more low-frequency components (row 1, column 4 in Figure 7(b)) while NViT chooses the channel with uniform values (row 2, column 3 in Figure 7(a)).
The accumulation of such changes in selection finally result in our better performance, which further proves the importance of low-frequency information to ViTs.
[Equation]
[Figure]
[Equation]
[Figure]
A.9 Architecture of pruned models
To give a straightforward visualization for the pruned models, the ratios of saved channels and tokens in each block is presented in Figure 8.
It can be observed that more channels are pruned than tokens in shallow blocks ( e.g. block-1 and block-2), while more tokens are reduced in deep blocks.
This inspires that the architecture with thin channels in shallow layers and few tokens in deep layers may be more suitable to ViTs.
[Figure]
A.10 Distributions of importance scores
The distributions of the proposed importance scores for channels (LFS) and tokens (LFE × attentionscore) are displayed in Fig.
9. 2000 images from ImageNet-1k are sampled and fed into DeiT-Tiny to obtain the average LFS for each channel and scores for tokens.
Compared to the original model (Fig. 9(a)), the density of LFS near zero is lower in the pruned model.
Similarly, the token scores in the pruned model are denser than those in the original model (Fig. 9(b)).
A.11 Low-pass filtering on input images
In Figure 10, the images filtered with different cutoff factors are visualized in both the spatial domain and the Fourier domain.
[Figure]
[Figure]