N OT A LL P ATCHES ARE W HAT Y OU N EED : E XPEDITING V ISION T RANSFORMERS VIA T OKEN R EORGANIZATIONS
A BSTRACT
Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them.
Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA.
Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions.
In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training.
For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention.
Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations.
To this end, our method EViT improves ViTs from two perspectives.
First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efﬁcient inference.
For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classiﬁcation.
Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images.
An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classiﬁcation at the same computational cost of a vanilla DeiT-S.
Meanwhile, our method does not introduce more parameters to ViTs.
Experiments on the standard benchmarks show the effectiveness of our method.
The code is available at https://github.com/youweiliang/evit
I NTRODUCTION
Computer vision research has evolved into Transformers since ViTs (Dosovitskiy et al., 2021).
Equipped with global self-attention, ViTs have shown impressive capability upon local convolution (i.e., CNNs) on prevalent visual recognition scenarios, including image classiﬁcation (Dosovitskiy et al., 2021;
Touvron et al., 2021a; Jiang et al., 2021; Graham et al., 2021), object detection (Carion et al., 2020), and semantic segmentation (Xie et al., 2021; Liu et al., 2021; Wang et al., 2021a;c), with both supervised and unsupervised (self-supervised) training (Pan et al., 2021; Ge et al., 2021) conﬁgurations.
Based on the main spirit of ViTs (i.e., MHSA), there are wide investigations (Liu et al., 2021;
Chu et al., 2021; Wang et al., 2021a) to explore the network structure of ViT models for continuous recognition performance improvement.
Along with the development of ViT models, the computation burden is becoming an issue.
The global self-attention between image tokens and long-range dependency make the model converge slow compared to CNNs.
As illustrated in (Dosovitskiy et al., 2021), training a ViT from scratch Tencent
[Figure] typically requires larger datasets (e.g., ImageNet-21k (Deng et al., 2009) and JFT-300M (Sun et al., 2017)) than those of CNNs (e.g., CIFAR-10/100 (Krizhevsky et al., 2009) and ImageNet-1k).
Also, using more training iterations is a necessity for network convergence (Dosovitskiy et al., 2021; Touvron et al., 2021a; Wang et al., 2022).
Without such large scale training, the ViT models are not fully exploited and perform inferior on visual recognition scenarios.
These issues motivate us to expedite ViTs for practice usage.
The model acceleration of ViTs is important to reduce computational complexity.
However, there are few studies focused on ViT acceleration.
This is because the signiﬁcant model difference between
CNNs and ViTs prevents CNN model acceleration (e.g., pruning and distillation) from applying on
ViTs.
Nevertheless, we analyze ViT from another perspective.
We observe that not all image tokens in ViTs contribute positively to the ﬁnal predictions.
Fig. 1 shows some examples where part of the input image tokens are randomly dropped out.
In Fig. 1b, removing image tokens related to the visual content of the corresponding category makes the ViT predict incorrectly.
In comparison, removing unrelated image tokens does not affect ViT predictions, as shown in Fig. 1a.
On the other hand, we notice that ViTs divide images into non-overlapping tokens and perform self-attention (Vaswani et al., 2017) computation on these tokens.
A notable characteristic of self-attention is that it can process a varying number of tokens.
These observations motivate us to reorganize image tokens for ViT model accelerations.
In this work, we propose a token reorganization method to identify and fuse image tokens.
Given all the image tokens as input, we compute token attentiveness between these tokens and the class token for identiﬁcation.
Then, we preserve the attentive image tokens and fuse the inattentive tokens into one token to allow the gradient back-propagate through the inattentive tokens for better attentive token identiﬁcation.
In this way, we gradually reduce the number of image tokens as the network goes deeper to decrease computation cost.
Also, the capacity of the ViT backbone can be ﬂexibly controlled via the identiﬁcation process where no additional parameters are introduced.
We adopt our token reorganization method on representative ViT models (i.e., DeiT (Touvron et al., 2021a) and LV-ViT (Jiang et al., 2021)) for ImageNet classiﬁcation evaluation.
The experimental results show our advantages.
For instance, with the same amount of input image tokens, our method speeds up the DeiT-S model by 50%, while only sacriﬁcing 0.3% recognition accuracy on ImageNet classiﬁcation.
On the other hand, we extend our methods to boost the ViT model recognition performance under the same computational cost.
By increasing the input image resolution, our method facilitates
Vision Transformers in taking more image tokens to achieve higher classiﬁcation accuracy.
Numerically, we improve the ImageNet classiﬁcation accuracy of the DeiT-S model by 1% under the same computational cost.
Moreover, by using an oracle ViT to guide the token reorganization process, our method can increase the accuracy of the original DeiT-S from 79.8% to 80.7% while reducing its computation cost by 36% under the multiply-accumulate computation (MAC) metric.
R ELATED WORK 2. V ISION T RANSFORMERS
Transformers (Vaswani et al., 2017) have drawn much attention to computer vision recently due to its strong capability of modeling long-range relation.
A few attempts have been made to add self- attention layers or Transformers on top of CNNs in image classiﬁcation (Hu et al., 2019), object detection (Carion et al., 2020), segmentation (Wang et al., 2021c), image retrieval (Lu et al., 2019) and even video understanding (Sun et al., 2019; Girdhar et al., 2019).
Vision Transformer (ViT)
(Dosovitskiy et al., 2021)
ﬁrst introduced a set of pure Transformer backbones for image classiﬁcation and its follow-ups modify the ViT architecture for not only better visual recognition (Touvron et al., 2021a; Yuan et al., 2021;
Zhou et al., 2021) but many other high-level vision tasks, such as object detection (Carion et al., 2020; Zhu et al., 2020; Liu et al., 2021), semantic segmentation (Wang et al., 2021a; Xie et al., 2021; Chu et al., 2021), and video understanding (Bertasius et al., 2021; Fan et al., 2021).
Vision Transformers have shown its strong potential as an alternative to the previously dominant CNNs.
2. M ODEL ACCELERATION
Neural networks are typically overparameterized (Allen-Zhu et al., 2019), which results in signiﬁcant redundancy in computation in deep learning models.
To deploy deep neural networks on mobile devices, we must reduce the storage and computational overhead of the networks.
Many adaptive computation methods are explored (Bengio et al., 2015;
Wang et al., 2018; Graves, 2016;
Hu et al., 2020; Wang et al., 2020b; Han et al., 2021b) to alleviate the computation burden.
Parameter pruning (Srinivas & Babu, 2015; Han et al., 2015; Chen et al., 2015b) reduces redundant parameters which are not sensitive to the ﬁnal performance.
Some other methods leverage knowledge distillation (Hinton et al., 2015; Romero et al., 2014; Luo et al., 2016; Chen et al., 2015a) to obtain a small and compact model with distilled knowledge of a larger one.
These model acceleration strategies are limited to convolutional neural networks.
There are also some attempts to accelerate the computation of the Transformer model, including proposing more efﬁcient attention mechanisms (Wang et al., 2020a; Kitaev et al., 2020; Choromanski et al., 2020) and the compressed Transformer structures (Liu et al., 2021; Heo et al., 2021;
Wang et al., 2021a).
These methods mainly focus on reducing the complexity of the network architecture through artiﬁcially designed modules.
Another approach to ViT acceleration is reducing the number of tokens involved in the inference of ViTs.
Notably, Wang et al.
(2021b) proposed a method to dynamically determine the number of patches to divide on an image.
The ViT will stop inference for an input image if it has sufﬁcient conﬁdence in the prediction of the intermediate outputs.
Remarkably, Ryoo et al.
(2021) proposed TokenLearner to expedite ViTs, where a relatively small amount of tokens are learned by aggregating the entire feature map weighted by a dynamic attention map conditioned on the feature map.
This can be seen as a sophisticated method for tokenizing the input images.
Different from TokenLearner, our work focuses on the progressive selection of informative tokens during training.
Another related work is DynamicViT
(Rao et al., 2021), which introduces a method to reduce token for a fully trained ViT, where an extra learnable neural network is added to ViT to select a subset of tokens.
Our work provides a novel perspective for reducing the computational overhead of inference by proposing a token reorganization method to progressively reduce and reorganize image tokens.
Unlike DynamicViT, our method does not need a fully trained ViT to help the training and brings no additional parameters into ViT.
T OKEN REORGANIZATIONS
Our method EViT is built upon ViT (Dosovitskiy et al., 2021) and its variants for visual recognition.
We ﬁrst review ViT and then present how to incorporate our method into the ViT training procedure.
Each component of EViT, including attentive token identiﬁcation and inattentive token fusion, will be elaborated.
Furthermore, we analyze the effectiveness of our method by visualizing the attentive tokens at different layers and discuss training on higher resolution images with EViT.
3. V I T OVERVIEW
Vision Transformers (ViTs) are ﬁrst introduced by Dosovitskiy et al.
(2021) into visual recognition.
They perform tokenization by dividing an input image into patches and projecting each patch to a token embedding.
An extra class token
[CLS] is added to the set of image tokens and is responsible for aggregating global image information and ﬁnal classiﬁcation.
All of the tokens are added by a learnable vector (i.e., positional encoding) and fed into the sequentially-stacked Transformer
[Table]
encoders consisting of a multi-head self-attention (MHSA) layer and a feed-forward network (FFN).
In MHSA, the tokens are linearly mapped and further packed into three matrices, namely Q , K , and
V .
The attention operation is conducted as follows.
[Equation]
where d is the length of the query vector.
The result of Softmax( QK (cid:62) / √ d ) is a square matrix which is called the attention map.
The ﬁrst row of attention map represents the attention from [CLS] to all tokens and will be used to determine the attentiveness (importance) of each token (detailed in the next subsection).
The output tokens of MHSA are sent to FFN, consisting of two fully connected layers with a GELU activation layer (Hendrycks & Gimpel, 2016) in between.
At the ﬁnal Transformer encoder layer, the [CLS] token is extracted and utilized for object category prediction.
More details of Transformers can be found in Vaswani et al.
3. A TTENTIVE TOKEN IDENTIFICATION
Let n denote the number of input tokens to a ViT encoder.
In the last encoder of ViT, the [CLS] token is taken out for classiﬁcation.
The interactions between [CLS] and other tokens are performed via the attention mechanism (Vaswani et al., 2017) in the ViT encoders:
[Equation]
where q class , K , and V denote the query vector of [CLS] , the key matrix, and the value matrix, respectively, in an attention head.
In other words, the output of the [CLS] token x class is a linear combination of the value vectors V
=
[ v 1 , v 2 , . .
. , v n ] (cid:62) , with the combination coefﬁcients (denoted by a in Eq. 2) being the attention values from [CLS] with respect to all tokens.
Since v i comes from the i -th token, the attention value a i (i.e., the i -th entry in a ) determines how much information of the i -th token is fused into the output of [CLS] (i.e., x class ) through the linear combination.
It is thus natural to assume that the attention value a i indicates the importance of the i -th token.
Moreover, Caron et al.
(2021) also showed that the [CLS] token in ViTs pays more attention (i.e., having a larger attention value) to class-speciﬁc tokens than to the tokens on the non-object regions.
To this end, we propose to use the attentiveness of the [CLS] token with respect to other tokens to identify the most important tokens.
Based on these arguments, a simple method to reduce computation in ViT is to remove the tokens with the smallest attention values.
However, we ﬁnd that directly removing those tokens severely deteriorates the classiﬁcation accuracy, as shown in Table 1.
Therefore, we propose to incorporate image token reorganization during the ViT training process.
In multi-head self-attention layer, there are multiple heads performing the computation of Eq.
1 in parallel.
Thus, there are multiple [CLS] attention vectors a ( h ) , h = [1 , . . .
, H ] , with H being the total number of attention heads (Vaswani et al., 2017).
We compute the average attentiveness value of all heads by ¯ a = (cid:80) Hh =1 a ( h ) /H .
As shown in Figure 2, we identify and preserve the tokens corresponding to the k largest (top-k) elements in ¯ a ( k is a hyperparameter), which we call the attentive tokens, and further fuse the other tokens (which we call the inattentive tokens) into a new token.
The fusion of tokens is detailed in the following paragraph.
We deﬁne the token keeping rate as κ = k/n .
3. I NATTENTIVE TOKEN FUSION
Although the tokens on the backgrounds of images are less informative and can be discarded without signiﬁcantly inﬂuencing the performance of the ViT model, they may still be able to contribute to
[Figure]
CLS Token
M u l t iH e a d
S e l fA tt e n t i o n
A dd & N o r m
Token Reorganization
AttentiveTokenIdentification
Inattentive
TokenFusion
Calculated CLS Attentiveness
F ee d F o r w a r d
N e t w o r k
A dd & N o r m …
CLS Token the prediction results.
On the other hand, some images may have large object parts covering a large proportion of the images.
Thus, it is possible to remove some tokens corresponding to those object parts when we select a ﬁxed number of tokens to keep in a ViT encoder, which may negatively affect the image recognition performance.
To mitigate these problems, we propose to fuse the inattentive tokens at the current stage to supplement attentive ones, as illustrated in Figure 2.
The inattentive token fusion beneﬁts our method to preserve more information provided by the inattentive tokens.
Speciﬁcally, we denote the indices set of the inattentive tokens as N .
The proposed inattentive token fusion is a weighted average operation, which can be written as x fused =
(cid:80) i ∈N a i x i .
The fused token x fused is appended to the attentive tokens and sent to the subsequent layers.
The computation cost of token fusion is negligible compared to the bulk computation of ViT.
3. A NALYSIS
Training with higher resolution images.
Since our approach is efﬁcient in processing image tokens, we are able to feed more tokens into an EViT while maintaining the computational cost at the same level as a vanilla ViT.
A straightforward method to get more tokens is resizing the input images to a higher resolution and keeping the patch size unchanged.
Note that these higher resolution images do not necessarily come from raw images with a higher resolution.
In our vision recognition experiments on ImageNet, we simply resize the standard input images of size 224 × 224 to a larger spatial size (e.g., 256 × 256 ) via bicubic interpolation to obtain the higher resolution images, which are further divided into more tokens.
Therefore, compared to a vanilla ViT, EViT uses no additional information from the images to obtain the prediction results in both training and inference.
The experimental results in Table 5 validate the effectiveness of the proposed method.
Visualization.
The proposed EViT accelerates ViTs by identifying the attentive tokens and discarding the redundant calculation on inattentive image tokens.
To further investigate the interpretability of EViT, we visualize the attentive token identiﬁcation procedure in Figure 3.
We present the original images and the attentive token identiﬁcation results at different layers (e.g., the 4 th , 7 th and 10 th layer).
It can be seen that as the network deepens, the inattentive tokens are gradually removed or fused, while the most informative/attentive tokens are identiﬁed and preserved.
In this way, the proposed method facilitates the ViTs in focusing on class-speciﬁc tokens in images.
The visualization results also validate that our EViT is effective in dealing with images with either simple backgrounds or complex backgrounds.
input
[Figure]
[Table]
E XPERIMENTS
Implementation details.
We train all of the models on the ImageNet (Deng et al., 2009) training set with approximately 1.2 million images and report the accuracy on the 50k images in the test set.
By default, the token identiﬁcation module is incorporated into the 4 th , 7 th and 10 th layer of DeiT-S and DeiT-B (with 12 layers in total) and incorporated into the 5 th , 9 th and 13 th layer of LV-ViT-S (with 16 layers in total).
The image resolution in training and testing is 224 ×unless otherwise speciﬁed.
For the training strategies and optimization methods, we simply follow those in the original papers of DeiT (Touvron et al., 2021a) and LV-ViT (Jiang et al., 2021).
Since our method can be easily incorporated into these models without making substantial modiﬁcations to them, the original training strategies work well with our method.
Besides, we adopt a warmup strategy for attentive token identiﬁcation.
Speciﬁcally, the keep rate of attentive tokens is gradually reduced from 1 to the target value with a cosine schedule.
Unlike DynamicViT
(Rao et al., 2021), we do not use a pretrained ViT to initialize our models in most experiments, except in the experiments with an oracle ViT (see the following paragraphs).
We train the models with EViT from scratch for 300 epochs on 16 NVIDIA A100 GPUs and measure the throughput of the models on a single A
GPU with a batch size of 128 unless otherwise speciﬁed.
The multiply-accumulate computations (MACs) metric is measured by torchprofile (Liu, 2021).
[Table] [Table]
We report the main results of EViT on Tables 2 and 3.
On both DeiT and LV-ViT, the proposed
EViT achieves signiﬁcant speedup while restricting the accuracy drop in a relatively small range.
For example, DeiT-S trained with EViT with a keep rate of 0.7 increase the inference throughput by 50% while maintaining the Top-1 accuracy reduction within 0.3% on ImageNet.
Inattentive token fusion.
To mitigate the problem of information loss in token removal, we propose inattentive token fusion, which fuses the non-topk tokens according to the attentiveness from the [CLS] token.
We experimentally compare the proposed token reorganization method with and without inattentive token fusion.
As shown in Table [Table] token reorganization with inattentive token fusion typically outperforms the one without inattentive token fusion.
Although the improvement is small, there is no additional computational overhead introduced.
From another perspective, it indicates the effectiveness of our attentive token identiﬁcation because the majority of effective tokens are well preserved.
Moreover, the accuracy ﬂuctuation in EViT with inattentive token fusion is smaller than the vanilla inattentive token removal, as shown by the standard deviation in Table 2.
Inattentive token fusion also beneﬁts EViT on high resolution images, as shown in Table 9.
Epochs of training.
Since ViTs do not have inductive bias such as translational invariance processed by CNNs, they typically require more training data and/or training epochs to reach a comparable generalization performance as CNNs (Dosovitskiy et al., 2021; Touvron et al., 2021a).
We ﬁnd that training longer epochs continues to beneﬁt ViTs in the efﬁcient computation regime.
We train the DeiT-S model with 0.7 keep rate for longer epochs of 450 and 600, respectively.
As shown in
Table 4, their performance steadily improves with training epochs.
Training/
Finetuning on higher resolution images.
By fusing the inattentive tokens, We are able to feed EViT with more tokens under the same computational cost.
Therefore, we train and/or ﬁnetune EViT on resized images with higher resolutions than the standard resolution of 224 2 , and we report the results on Table 5.
We can see that EViT in most tested cases performs favourably against a vanilla DeiT/LV-ViT, while having comparable or higher inference throughput than the baselines.
Notably, training EViT-LV-ViT-S on images of resolution 224 2 and further ﬁnetuning it on a higher resolution of 448 2 for another 100 epochs gives a very competitive Vision Transformer model, achieving an ImageNet top-1 accuracy of 84.7%, which is 0.3% higher than LV-ViT-S@ with basically the same throughput and number of parameters.
The experimental results validate our hypothesis that images typically contain tokens that are less informative and contribute little to the recognition task.
Since ViTs perform global self-attention among all tokens in as early as
The numbers before and after ↑ indicate the image size in the 300-epoch training and 100-epoch ﬁnetuning, respectively.
[Table] [Table]
the ﬁrst layer, the early interaction and information exchange between tokens makes it possible to discard/fuse some of the least informative tokens in intermediate ViT layers since they have been “seen” by other tokens (including the [CLS] token).
In comparison, the receptive ﬁeld of a CNN at the shallow layers is relatively small due to the locality property of convolution, which makes it difﬁcult to reduce computation at early stages.
Training with an oracle ViT. In EViT, the criterion of selecting tokens is the attention between the [CLS] token and other tokens.
Therefore, it would be very helpful if we know the importance of each token to the prediction tasks in advance.
To this end, we introduce an oracle ViT to guide the [CLS] through the token selection process.
A good oracle knows which tokens are important and which are not.
For this purpose, we use a fully trained DeiT-S/B as an oracle and initialize EViTDeiT-S/B with the parameters of the oracle ViT, such that the EViT know which tokens contribute more to the prediction result.
We train EViT equipped with an oracle using the same training setup as training a vanilla EViT.
As shown in Table 6, training with an oracle signiﬁcantly improves the recognition accuracy of EViT.
[Table]
Comparison with DynamicViT. We compare the proposed EViT with DynamicViT
(Rao et al., 2021).
Note that the authors of DynamicViT used a ﬁnetuning strategy.
Therefore, we ﬁrst compare the two methods in ﬁnetuning, and then we also train DynamicViT from scratch for 300 epochs to compare it with our baselines.
In the case of training from scratch, no pretrained ViT can be used, and thus the distillation loss and KL loss in DynamicViT are discarded.
The results in Tableshow that EViT outperforms DynamicViT in recognition accuracy under the same computational cost while EViT uses fewer parameters.
Moreover, the accuracy of EViT continues to improve in longer-epoch training.
When training from scratch, a signiﬁcant accuracy reduction is observed in
DynamicViT, which suggests that DynamicViT requires a teacher ViT (i.e., the parameter initialization, the distillation loss, and the prediction (KL) loss from a pretrained ViT (Rao et al., 2021)) to obtain a good performance.
Comparison with other Vision Transformers.
We plot the accuracy and throughput of various
ViTs in Figure 4, which shows EViT is very competitive among the vision models in terms of computation-accuracy trade-off.
For EViT, we only plot the EViT models that are trained from scratch for fair comparisons, except EViT/0.7-LV-ViT-S@448, which is ﬁnetuned for 100 epochs.
The comparing methods include DeiT (Touvron et al., 2021a), CaiT (Touvron et al., 2021b), LVViT (Jiang et al., 2021), CoaT (Xu et al., 2021), Swin (Liu et al., 2021),
Twins (Chu et al., 2021),
Visformer (Chen et al., 2021), ConViT (d’Ascoli et al., 2021), TNT (Han et al., 2021a), EfﬁcientNet (Tan & Le, 2019), and PVT (Wang et al., 2021a).
These ViTs variants focus on modifying the
ViT architectures or the interaction methods between image tokens to achieve improvement over the vanilla ViT (Dosovitskiy et al., 2021).
Thus, their methods and ours are complementary to each other, which makes it possible to incorporate our method into some of the ViT variants.
C ONCLUSION
In this paper, we present a token reorganization method.
By identifying the tokens with the largest attention from the class token, the proposed EViT reaches a better trade-off between accuracy and efﬁciency than various Vision Transformer models.
Moreover, we propose inattentive token fusion, which fuses the information from less informative tokens to a new token.
Inattentive token fusion improves both the recognition accuracy and training stability.
Experimentally, we apply the proposed token reorganization method to two variants of Vision Transformer, namely, DeiT (Touvron et al., 2021a) and LV-ViT (Jiang et al., 2021).
In both variants, EViT achieves a signiﬁcant speedup in inference while the reduction in recognition accuracy is relatively small.
Moreover, when training on higher resolution images, EViT improves the accuracy to various extents while maintaining a similar or smaller computation cost as the original DeiT/LV-ViT models.
Besides, when equipped with an oracle ViT which knows which tokens are more important, EViT can achieve further improvement on the trade-off between accuracy and efﬁciency.
The proposed EViT can be easily adapted in
ViTs and brings no additional parameters, nor does it require sophisticated training strategies.
The proposed token reorganization method can serve as an effective acceleration approach for Vision
Transformers.
A V ISUALIZATION
In this part, we present more visualization results in Figure 5 to show the attentive token identiﬁcation.
The input images are randomly selected from the ImageNet dataset.
The results validate that our EViT is able to deal with different images from various categories.
[Figure]
[Table] [Table]
[Table]
B E XTENDED E XPERIMENTS
More results of EViT on DeiT.
We provide the performance of EViT on DeiT-B (Touvron et al., 2021a) in Table 8.
Again, the resulting models of EViT-DeiT-B achieve signiﬁcant speedup while restricting the accuracy drop in a relatively small range.
For example, EViT-DeiT-B with a keep rate of 0.8 increases inference throughput by 26% while maintaining the Top-1 accuracy reduction within 0.1% on ImageNet.
More comparisons of vanilla inattentive token removal and inattentive token fusion.
We compare the performance of the two variants of EViT, namely, vanilla inattentive token removal and inattentive token fusion, on the training of higher resolution images.
In this scenario, EViT with inattentive token fusion also outperforms the EViT with vanilla inattentive token removal, especially in the Top-5 accuracy, as shown in Table 9.
Although the improvement is relatively small, it is practical as inattentive token fusion brings no additional computational overhead compared to vanilla inattentive token removal.
Using tokens-to-tokens attentive scores for inattentive tokens identiﬁcation.
As discussed in
Section 3.2, we use the attention from the [CLS] token to other tokens as the attentive scores to identify the class-related tokens that are important to visual recognition.
This attentive/inattentive token identiﬁcation approach is motivated by the fact that the ﬁnal prediction of ViTs is determined by the representation of the [CLS] token in the last layer.
However, we also ﬁnd that it is possible to use a tokens-to-tokens attention approach to identify the attentive tokens, where the attention scores are computed as the average of the attention from all tokens to all tokens.
Speciﬁcally, recall the attention map A is computed by:
[Equation]
The attentive score is computed by
[Equation]
where n is the number of tokens and A i, : is the i -th row of the attention map A .
When there are multiple self-attention heads, the ﬁnal attentive score is averaged over all heads: ¯ a = (cid:80)
H h
=1 a ( h ) /H , where H is the number of heads.
In a nutshell, in the tokens-to-tokens strategy, each token can “vote” to decide which tokens are attentive, while in the [CLS] -to-tokens strategy, only the [CLS] token can “vote”.
As shown in Table 10, EViT with the tokens-to-tokens attention strategy performs comparably with the [CLS] -to-tokens attention strategy but is less efﬁcient.
Nevertheless, this experimental observation opens the door for incorporating the proposed EViT into the ViT variants with no [CLS] tokens (e.g., PVT (Wang et al., 2021a)).
[Table] [Table]
Training time of EViT.
We provide a comparison on the training time of DeiT-S (Touvron et al., 2021a) and EViT-DeiT-S in Table 11.
The results show that EViT requires considerably smaller
GPU × hours to train when compared to a vanilla DeiT under the same number of epochs.
Since
EViT continues to improve in recognition accuracy with larger training epoch (as shown in Table 4), the performance gap between a vanilla ViT and an EViT can be narrowed down and even closed when the EViT is trained for the same amount of time as the ViT.
Finetuning of EViT on high resolution images.
We provide results of ﬁnetuning EViT on highresolution images in the last three rows in Table 12.
To ﬁnetune EViT-DeiT-S from low-resolution images to high-resolution images, we need to resolve the discrepancy between the positional embedding (Touvron et al., 2021a) of two different resolutions.
To this end, we interpolate the positional embedding of lower resolution to the desired higher resolution.
This method works quite well in our experiments, as shown in the last three rows in Table 12.
However, directly training on higher resolution images seems a better way to train EViT, as it requires less training time (because no ﬁnetuning is performed) and typically obtains higher accuracy than ﬁnetuning.
Token reorganization locations.
It is possible that different combinations of keep rates and token reorganization locations can reach the same computational efﬁciency.
For example, we can a) move the reorganization modules into shallower layers and increase the keep rates, or b) move the reorganization modules into deeper layers and decrease the keep rates, to keep the computation cost (approximately) unchanged.
To get a deeper understanding of this aspect, we train EViT-DeiT-S models with different token reorganization locations and keep rates, each of which has a similar computational cost as the baseline.
The results in Table 13 reveal two perspectives.
First, moving the reorganization modules into shallower layers deteriorates the accuracy.
For example, when the token reorganization module is placed before the third layer (i.e., at the ﬁrst or second layer), the recognition accuracy drops considerably even though the computational cost is the same.
This suggests that ViT cannot identify the important tokens at the early stages, which is quite reasonable as the processing of input tokens is just started at shallow layers, where the attention maps are unreliable for token removal.
Second, placing the reorganization modules in different deeper layers has only marginal inﬂuence on the accuracy.
For instance, when the token reorganization modules are placed behind the third layer, the resulting models have basically the same accuracy.
This suggests that EViT has stable performance as long as the reorganization locations are not in shallow layers.
Because of this reason, we refrain from searching for a (slightly) better conﬁguration of the reorga-
[Table]
[Table]
[Table]
[Table]
nization locations and keep rates.
We adopt a simple strategy to decide the reorganization locations in our experiments, where the token reorganization locations cut the ViT into blocks with the same number of layers.
Speciﬁcally, for a ViT with L layers and t token reorganization layers in total, we ﬁrst obtain the separating length s = L/
( t + 1) .
Then, the layer indices of the reorganization layers are [ s + 1 , 2 s + 1 , . .
, ts + 1] , which cuts the ViT evenly.
For the keep rates, we simply set them to the same value for each token reorganization module in the EViT.
Adopting the DINO attention map for inattentive token identiﬁcation.
In DINO (Caron et al., 2021), the authors showed that the [CLS] attention produced by a self-supervised trained ViT is naturally focused on the object parts in images.
Therefore, one may wonder if the good DINO attention can be used in EViT for inattentive token identiﬁcation.
To answer this question, we use the [CLS] attention produced by the last layer of DINO to guide the token selection process in
EViT. Instead of starting the token reorganization process at the 4 th layer of EViT-DeiT-S as we have done in previous experiments, we start the token reorganization at the ﬁrst layer.
Speciﬁcally, we select the top 50% tokens that correspond to the positions with the strongest DINO attention (i.e., corresponding to the top 50% attention values in the DINO attention).
Experimental results in Table 14 show that EViT works well with the DINO attention.
As a comparison, when the reorganization layer is placed in the ﬁrst layer in a vanilla EViT, as shown in Table 13, a signiﬁcant accuracy reduction is observed, while EViT equipped with DINO can maintain its recognition ability even if 50% of tokens are removed as early as the ﬁrst layer.
However, this is not an efﬁcient approach in practice because obtaining the DINO attention requires forwarding the input through
[Table] [Table]
the DINO model (i.e., a DeiT model) and bringing extra computational burden, as shown by the “Combined MACs” in Table 14.
The correlation between the quality of the attention masks and the performance of EViT. From
Figures 3 and 5, we can see that the masks produced by EViT are quite intuitive and of good quality, which means that the attention values of the CLS token to other tokens correlate well with the semantic segmentation of images.
Therefore, we can deliberately use low quality masks in EViT to see how its performance changes with the quality of the masks.
Note that we use the largest [CLS] -to-tokens attention values to obtain the mask for the keeping tokens (we call this method
EViT-topk).
Thus, the worse possible mask one can obtain is probably using the smallest [CLS] - to-tokens attention values (we call this method EViT-mink), and a not-so-bad mask can be obtained by randomly generating a mask without considering the [CLS] -to-tokens attention (we call this method EViT-random).
We perform experiments to see the classiﬁcation accuracy of EViT under these circumstances.
As shown in Table 15, the classiﬁcation accuracy of these three methods clearly correlates with the quality of the masks.
From the experimental results, we conclude that masks of better quality help improve the accuracy.
Therefore, we believe it is a potential direction for future research as to how to (efﬁciently) obtain a better mask to improve the performance of EViT.
The attention scores of the inattentive tokens w.r.t.
the layer depth.
In order to ﬁnd how the attentive scores of the inattentive tokens evolve, we randomly sample several images and plot the change of the [CLS] -to-tokens attention values of n inattentive tokens ( n = 25 ) w.r.t.
the depth of layers in Fig.
Speciﬁcally, we use a pre-trained DeiT-S and identify the n tokens with the smallest [CLS] -to-tokens attention in the 4 th layer.
Then we keep track of these selected tokens in the subsequent layers and draw the box plot to see how the statistics (e.g., the mean, standard deviation, and maximum and minimum values) change with the layer depth.
The results suggest that the tokens that are identiﬁed as inattentive tokens in the shallow layer tend to be less preferred in deeper layers.
The evolution of the inattentive tokens during EViT training process.
To prove the effectiveness of incorporating EViT into the training process of ViT, we present the evolution of the inattentive tokens during the training of EViT-DeiT-S with a keep rate of 0.7 in Figure 7.
It can be observed that at the initial training stage, the EViT model tends to misidentify the inattentive tokens (e.g., the bulk of the car in the second column of the ﬁrst three rows).
However, the inattentive tokens gradually converge to the less informative regions of images during the training process.
For example, in the last column in Figure 7, the masked regions mainly focus on the background (e.g., the grass).
This observation validates the effectiveness of our proposed token identiﬁcation methods.
[Figure]
Input image
Epoch Epoch
Epoch Epoch
Epoch
Epochl a y e r l a y e r l a y e r
Input image
Epoch Epoch
Epoch Epoch
Epoch
Epochl a y e r l a y e r l a y e r
Input image
Epoch Epoch
Epoch Epoch
Epoch
Epochl a y e r l a y e r l a y e r