R EGION V I T: R EGIONAL - TO -L OCAL A TTENTION FOR V ISION T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classiﬁcation.
However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications.
Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ novel regional-to-local attention rather than global self-attention in vision transformers.
More speciﬁcally, our model ﬁrst generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location.
The regional-to-local attention includes two steps: ﬁrst, the regional self-attention extracts global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention.
Therefore, even though local self-attention conﬁnes the scope in a local region but it can still receive global information.
Extensive experiments on four vision tasks, including image classiﬁcation, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works.
Our source codes and models will be publicly available.
I NTRODUCTION
Transformers (Vaswani et al., 2017) based on self-attention come naturally with the ability to learn long-range dependencies in sequential data.
As important as it is to language modeling (Devlin et al., 2019), such ability is also highly desired for many vision tasks where contextual modeling plays a signiﬁcant role.
For this reason, self-attention and transformers have been receiving an increasing attention in the vision community (Bello, 2021; Srinivas et al., 2021;
Zhao et al., 2020; Ramachandran et al., 2019a; Bello et al., 2019; Hu et al., 2019; Ramachandran et al., 2019b;
Wang et al., 2018).
Especially, the recent Vision Transformers (ViT) (Dosovitskiy et al., 2021) demonstrates comparable image classiﬁcation results against the ﬁrmly established and prevalent CNNs in computer vision
(He et al., 2016; Tan & Le, 2019; Brock et al., 2021), albeit relying on a huge amount of training data.
It has since then led to an explosion of interest in further investigating its potential for a wide variety of vision applications (Wu et al., 2021;
Wang et al., 2021; Heo et al., 2021; Zhang et al., 2021a; Li et al., 2021;
Graham et al., 2021; Liu et al., 2021; Chu et al., 2021a; Yan et al., 2021; Chen et al., 2021b).
The ViT inherits the entire architecture from the vanilla transformer (Vaswani et al., 2017), which is designed for natural language processing tasks, and some of those designs thus may not meet the needs of vision tasks.
For example, the transformer has an isotropic network structure with a ﬁxed number of tokens and unchanged embedding size, which loses the capability to model the context with different scales and allocates computations at different scales.
As opposed to this, a majority of
CNNs adopt a popular pyramid architecture to compute multi-scale features efﬁciently.
Recent vision transformers such as PVT (Wang et al., 2021) and PiT (Heo et al., 2021) also follow a similar pyramid structure as CNNs, showing improvement on both computation and memory efﬁciency as well as on model accuracy.
Another critical bottleneck of the transformer is that the self-attention module has a quadratic cost in memory and computation with regard to the sequence length (i.e., the number of tokens).
This issue is even worse in ViT as images are 2-D, suggesting a quadratic relationship between the number of tokens and image resolution.
As a result, ViT indicates a quadruple complexity w.r.t image resolution.
The highly compute- and memory-intensive self-attention makes it challenging to train vision transformer models at ﬁne-grained patch sizes.
It also signiﬁcantly undermines the applications of these models to tasks such as object detection and semantic segmentation, which beneﬁt from or require ﬁne feature information computed from high-resolution images.
Other arrows are ignored for simplicity.
(a) ViT (b) PVT (c) RegionViT
(Ours) 16 ⨉16 ⨉16 ⨉32 ⨉16 ⨉8 ⨉32 ⨉16 ⨉8 ⨉64 ⨉32 ⨉16 ⨉
To address the aforementioned computational limitations of vision transformers, in this work we develop a memory-friendly and efﬁcient self-attention method for transformer models to reach their promising potential for vision applications.
We propose a novel coarse-to-ﬁne mechanism to compute self-attention in a hierarchical way.
Speciﬁcally, our approach ﬁrst divides the input image into a group of nonoverlapping patches of large size (e g., 28 × 28), on which regional tokens are computed via linear projection.
Similarly, local tokens are created for each region using a smaller patch size (e.g., 4 × 4).
We then use a standard transformer to process regional and local tokens separately.
To enable communication between the two types of tokens, we ﬁrst perform self-attention on regional tokens ( regional attention ) and then jointly attend to the local tokens of each region including their associated regional token ( local attention ).
By doing so, regional tokens pass global contextual information to local tokens efﬁciently while being able to effectively learn from local tokens themselves.
For clarity, we represent this two-stage attention mechanism as Regional-to-Local Attention , or R2L attention for short (see Figure 1 for an illustration).
Since both regional and local attention involve much fewer tokens, our R2L attention requires substantially less memory than regular global self-attention used in vision transformers.
For example, in our default setting, the memory saving using R2L attention can be up to as much as 73%.
We demonstrate the effectiveness of our approach on image classiﬁcation and several downstream vision tasks including object detection and action recognition.
To summarize, our key contributions in this work are as follows:
We propose a new vision transformer (RegionViT) based on regional-to-local attention to learn both local and global features.
Our proposed regional-to-local attention alleviates the overhead of standard global attention (too many tokens) and the weakness of pure local attention (no interaction between regions) used in existing vision transformers.
Our regional-to-local attention reduces the memory complexity signiﬁcantly as compared to standard self-attention, leading to a savings in memory complexity by about O ( N/M 2 ) , where M is the window size of a region and N is the total number of tokens.
This effectively allows us to train a more deep network for better performance while with comparable complexity.
Our models outperform or on par with several concurrent works on vision transformer that exploit pyramid structure for image classiﬁcation.
Experiments also demonstrate that our models work well on several downstream classiﬁcation tasks, including object detection and action recognition.
R ELATED W ORK
CNNs with Attention.
Attention has been widely used for enhancing the features in CNNs, e.g.,
SENet (Hu et al., 2018) and CBAM (Woo et al., 2018).
Various methods have been proposed that combine self-attention with CNNs (Bello, 2021; Srinivas et al., 2021;
Zhao et al., 2020; Ramachandran et al., 2019a; Bello et al., 2019; Hu et al., 2019; Ramachandran et al., 2019b;
Wang et al., 2018).
E.g., SAN (Zhao et al., 2020) and SASA (Ramachandran et al., 2019a) attempts to replace all convolutional layers with local self-attention network.
While these works show promising results, their complexity is relatively high due to the self-attention.
On the other hand, BoTNet (Srinivas et al., 2021) replaces few convolutional layers with a slightly-modiﬁed self-attention to balance computation and accuracy.
[Table]
LambdaNet work (Bello, 2021) uses the approximated self-attention to reduce the overhead of selfattention and make the network efﬁcient. Few works use attention approach to deﬁne the regional of interest for the ﬁne-grained visual recognition (Zheng et al., 2020; Ding et al., 2021; Wharton et al., 2021).
In contrast, our model utilizes regional-to-local attention to alleviate the workload of global self-attention by local attention while still keeping the global information via regional attention.
Vision Transformer.
ViT (Dosovitskiy et al., 2021) has recently achieved comparable results to CNNs when using a huge amount of data, e [Table] g., JFT300M (Sun et al., 2017).
DeiT (Touvron et al., 2020; Wightman, 2019) subsequently proposes an efﬁcient training scheme that allows vision transformer to work on par with CNNs while training only on ImageNet1K (Deng et al., 2009).
Despite promising performance of ViT, the architecture is directly borrowed from natural language processing, which might not be suitable for vision applications.
Motivated by this, two main line of research works have been developed to improve ViT. One is to enhance different components of the vision transformer (Yuan et al., 2021; Han et al., 2021; Touvron et al., 2021; Jiang et al., 2021) while still using isotropic structure (i.e., ﬁxed token numbers and channel dimension) like ViT, e.g., while
T2T-ViT (Yuan et al., 2021) introduces a Tokens-to-Token (T2T) transformation to encode local structure for each token, CrossViT propose a dual-path architecture, each with different scales, to learn multi-scale features (Chen et al., 2021a).
CaiT (Touvron et al., 2021) proposes a layer-scalar to train a deeper network for better performance and LV-ViT (Jiang et al., 2021)
modiﬁed how the model is trained when CutMix (Yun et al., 2019) augmentation is applied on ViT. Another parallel thread for improving vision transformer is in incorporating CNN-like pyramid structure into ViT (Wu et al., 2021;
Wang et al., 2021; Heo et al., 2021; Zhang et al., 2021a; Li et al., 2021;
Graham et al., 2021;
Liu et al., 2021;
Chu et al., 2021a; Yan et al., 2021; Chen et al., 2021b).
PVT (Wang et al., 2021) and
PiT (Heo et al., 2021) introduce the pyramid structure into ViT, which makes them more suitable for objection detection as it can provide multi-scale features.
LocalViT (Li et al., 2021) and ConT (Yan et al., 2021) mix convolutions with self-attention to encode locality information.
Swin (Liu et al., 2021), ViL (Zhang et al., 2021a) and Twins (Chu et al., 2021a) limited the self-attention into a local region and then propose different methods to allow the interaction among each local region.
Our model also utilizes pyramid structure and limits the self-attention in a local region; while we propose the regional-to-local attention to exchange the information among each region efﬁciently.
Difference from PVT, Swin, ViL and Twins.
Table 1 summarizes the difference between our proposed approach, RegionViT with the closely related works.
PVT (Wang et al., 2021) uses pyramid structure but the scope of self-attention is still global.
While ViL (Zhang et al., 2021a) limits self-attention into a local region, they adopt overlapping windows as convolutions to process all the tokens.
On the other hand, Swin (Liu et al., 2021), Twins (Chu et al., 2021a) and ours adopt non-overlapped windows.
While Swin shifts the position of windows alternatively in the consecutive transformer encoder to allow the interaction between regions, Twins subsamples local tokens as the global tokens, and then use the global tokens as the keys for self-attention to achieve the interaction between regions.
By contrast, our proposed method utilizes a extra set of regional tokens with regional-to-local attention to perform self-attention on regional tokens only for the global information and then each regional token is sent to the associated local tokens to pass the global information.
One major difference from others is that the local self-attention in our approach includes one extra regional token to exchange the global information with local tokens.
M ETHOD
Our method is built on top of a vision transformer, so we ﬁrst present a brief background of ViT and then describe our method (RegionViT) on regional-to-local attention for vision transformers.
R2L Transformer
Encoder
C ⨉ H/4 ⨉ W/
[Figure]
Tokenization Tokenization
C ⨉ H/28 ⨉ W/
C ⨉ H/4 ⨉ W/
C ⨉ H/28 ⨉ W/
D o w n s a m p li n g
R2L Transformer
Encoder 2C ⨉ H/8 ⨉ W/ 2C ⨉ H/56 ⨉ W/
D o w n s a m p li n g
R2L Transformer
Encoder 4C ⨉ H/16 ⨉ W/ 4C ⨉ H/112 ⨉ W/
D o w n s a m p li n g
R2L Transformer
Encoder 8C ⨉ H/32 ⨉ W/ 8C ⨉ H/224 ⨉ W/
P r e d i c t i o n
F C
[Figure]
Vision Transformer.
As opposed to CNN-based approaches for image classiﬁcation, ViT is a purely attention-based counterpart, borrowed from NLP.
It consists of stacked transformer encoders, each of which contains a multihead self-attention (MSA) and a feed-forward network (FFN) with layer normalization and residual shortcut.
To classify an image, ViT ﬁrst splits it into patches of ﬁxed size, e.g. 16 × 16, and then transforms them into tokens by linear projection.
A class token is additionally prepended to the patch tokens to form the input sequence.
Before the token are fed into transformer encoders, a learnable absolute positional embedding is added to each token to learn the position information.
At the end of the network, the class token is used as the ﬁnal feature representation for classiﬁcation.
Mathematically, ViT can be expressed as
[Equation]
where x cls ∈ R 1 × C and x patch ∈ R N × C are the class token and patch tokens respectively and
x pos ∈ R (1+ N ) × C is the position embedding, and [ || ] denotes the tensor concatenation.
k , N and C are the layer index, the number of patch tokens and dimension of the embedding, respectively.
While the vanilla ViT is ideally capable of learning global interaction among all the patch tokens, the memory complexity of self-attention becomes high when there are many tokens as the complexity is quadratically linear to the length of the input sequence.
Moreover, the use of isotropic structure limits the capability of extending the vanilla ViT model to many vision applications that require high-resolution details, e.g., object detection.
To address these issues, we propose regional-to-local attention for vision transformer, RegionViT for short.
An Overview of Our Proposed Approach (RegionViT).
Figure 2 illustrates the architecture of the proposed vision transformer, which consists of two tokenization processes that convert an image into regional (upper path) and local tokens (lower path).
Each tokenization is a convolution with different patch sizes, e.g., in Figure 2, the patch size of regional tokens is 28 2 while 4 2 is used for local tokens with dimensions projected to C , which means that one regional token covers 7 2 local tokens based on the spatial locality, leading to the window size of a local region to 7 2 .
At stage 1, two sets of tokens are passed through the proposed regional-to-local transformer encoders.
However, for the later stages, to balance the computational load and to have feature maps at different resolutions, we deploy a downsampling process to halve the spatial resolution while doubling the channel dimension like CNN on both regional and local tokens before going to the next stage.
Finally, at the end of the network, we average the remaining regional tokens as the ﬁnal embedding for the classiﬁcation while the detection uses all local tokens at each stage since it provides more ﬁne-grained location information.
By having the pyramid structure, the ViT can generate multi-scale features and hence it could be easily extended to more vision applications, e.g., object detection, rather than image classiﬁcation only.
We explain the main components of the regional-to-local transformer encoder in the next section.
Regional-to-Local Transformer Encoder.
Figure 3 shows the proposed regional-to-local (R2L) transformer encoder which includes R2L attention and feed-forward network ( FFN ).
Speciﬁcally, within R2L attention, the regional self-attention (RSA) ﬁrst involves all regional tokens to learn the global information efﬁciently as the number of regional tokens is few and then local self-attention (LSA) takes the local tokens with the associated regional token to learn local feature while the involved regional token could provide global information at the same time.
Both RSA and LSA are multihead self-attention (MSA)
but with different input tokens.
Finally, the FFN is applied to enhance the features.
We also add layer normalization ( LN ) and residual shortcuts as in standard transformer encoders.
Mathematically, given the regional and local tokens, x d − 1 r and x d − 1 l as the inputs at layer d , the R2L transformer encoder can be expressed as:
[Equation] (2) where i, j are the spatial index with respect to regional tokens while m, n are the index of local token in the window size M 2 .
Note that the input of LSA, y di,j , includes one regional token and corresponding local tokens, and thus, the information between local and regional tokens are exchanged; on the other hand, the outputs, x dr and x dl , can be extracted from x di,j like the top-right equation.
[Figure]
The RSA exchanges information among all tokens, which covers the context of the whole image; while the
LSA combines the features among the tokens belonging to the spatial region, including both regional and local tokens.
Since the regions are divided by non-overlapped windows, the RSA is also designed to exchange the information among regions where the LSA takes one regional token and then combines with it the local tokens in the same region.
In such a case, all local tokens are still capable of getting global information while being more focused on local neighbors.
It is worth noting that the weights are shared between RSA and LSA except for the layer normalization; therefore, the number of parameters won’t increase signiﬁcantly when compared to the standard transformer encoder.
With these two attentions, the R2L can effectively and efﬁciently exchange information among all regional and local tokens.
In particular, the self-attention on regional tokens aims to extract high-level information and act as a bridge to pass information of local tokens from one region to other regions.
On the other hand, the
R2L attention focus on local contextual information with one regional token.
Moreover, the R2L transformer encoder can further reduce the memory complexity signiﬁcantly.
The memory complexity of a self-attention is O ( N 2 ) , where N is the number of local tokens.
When a region contains M local tokens, and there are N/M regions, the memory complexity is O (( M + 1) 2 × ( N/M )) ; while the complexity from the attention on regional tokens is O (( N/M ) 2 ) .
Hence, the overall complexity becomes O (( M +1) 2 × ( N/M ) +( N/M ) 2 ) .
E.g., the main component of RegionViT is stage 3, where M is 49 and N is 196; therefore, the memory complexity saving is ∼ 73% . The saving of each model and each stage is varied based on the model conﬁguration.
Relative Position Bias.
The locality is an important clue for understanding visual content; therefore, instead of adding absolute position embedding used by the vanilla ViT, we introduce the relative position bias into the attention map of R2L attention since relative position between patches (or pixels) are more important than the absolution position as objects in the images can be placed in an arbitrary way (Ramachandran et al., 2019a; Liu et al., 2021).
We only add this bias to the attention between local tokens and not the attention between regional tokens and local tokens.
Speciﬁcally, for a given pair of local tokens at location, ( x m , y m ) , ( x n , y n ) , the attention value a can be expressed as
[Equation]
where b ( x m − x n ,y m − y n ) is taken from a learnable parameter B ∈ R 2 M − 1 × 2 M − 1 , where M is the window size of a region.
The ﬁrst term q ( x m ,y m ) k T ( x n ,y n ) is the attention value based on their content.
With relative position bias, our model does not require the absolute positional embedding as vanilla
ViT since this relative position bias helps to encode the position information.
Downsampling.
The spatial and channel dimension are changed along different stages of our network by simply applying 3 × 3 depth-wise convolution with stride 2 for halving the resolution and doubling the channel dimension for both regional and local tokens (weights are shared)
(Heo et al., 2021).
We also test with regular convolutions but it does not improve the accuracy with increased complexity.
[Table]
Input Tokenization.
The tokenization for local tokens can be implemented by using a 4 × 4 convolution with channel size C and stride 4 (i
[Table]
e. non-overlapped).
However, as pointed out in T2T (Yuan et al., 2021), CrossViT (Chen et al., 2021a)
[Table]
and PiT (Heo et al., 2021), using a stronger but still simple subnetwork for the tokenization could further improve the performance, especially for the smaller models.
Thus, we adopt two input tokenizations in our model, one still contains only one convolutional layer and another one contains three convolutional layers (see Table 2 for details).
Regional Tokens.
We adopt a simple approach to generate regional tokens, that is, using linear projection with larger patch sizes in comparison to the patch size for the local tokens.
E.g., as shown in Figure 2, the patch size for local tokens is 4 × 4, we simply use patch size 28 × 28 to split the image and then project each patch linearly to generate regional tokens.
Table 2 shows our RegionViT models with different conﬁgurations.
By default, the window size iswhile we also experiment with a larger window size (14), those models are annotated with +.
E XPERIMENTS 4. I MAGE C LASSIFICATION
Datasets.
We use ImageNet1K (Deng et al., 2009)
(IN1K) and ImageNet21K (Deng et al., 2009) (IN21K) to validate our method.
ImageNet1K contains 1.28 million training images and 50k validation images over 1k classes, and ImageNet21K is a large-scale dataset that consists of around 14 million images over 21,841 classes.
We use all images for training and then ﬁnetune the model on ImageNet1K.
Moreover, we also perform the transfer learning from ImageNet1K to ﬁve downstream datasets, including CIFAR10 (Krizhevsky et al., 2009), CIFAR100 (Krizhevsky et al., 2009),
IIIPets (Parkhi et al., 2012), StandfordCars (Krause et al., 2013) and ChestXRay8 (Wang et al., 2017).
Training and Evaluation.
We follow DeiT (Touvron et al., 2020) to train our models on IN1K except that we use batch size 4,096 with a base learning rate 0.004 and the warm-up epochs is 50.
We adopt the AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate scheduler (Loshchilov & Hutter, 2017).
We apply Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019), RandomErasing (Zhong et al., 2020), label smoothing (Szegedy et al., 2016), RandAugment (Cubuk et al., 2020) and instance repetition (Hoffer et al., 2020).
During training, we random cropped a 224 × 224 region and take a 224 × 224 center crop after resizing the shorter side to 256 for evaluation.
We used a similar setting for IN21K and transfer learning, and more details can be found in Section A.
[Table] [Table]
[Table] [Table]
Results on ImageNet1K. Table 3 shows the results on ImageNet1K where the methods listed all adopt CNN-like pyramid structure into the ViT and are all very recent and concurrent works.
For the smaller models (Ti, S), RegionViT achieves a better trade-off between accuracy and complexity (parameters or FLOPs); on the other hand, for the larger models (M and B), it obtains better accuracy while having fewer FLOPs and parameters.
The efﬁciency of RegionViT comes from the proposed
R2L transformer encoder, and hence with such efﬁciency, it enables the network to be wide and deep for better accuracy with comparable complexity.
Results on ImageNet21K. Table 4a shows that our model achieves a good accuracy-efﬁciency tradeoff with ImageNet21K for pretraining.
RegionViT-B+ only needs 25% parameters and FLOPs compared to CvT-W24 (Wu et al., 2021) and half parameters and FLOPs to Swin-L (Liu et al., 2021).
Results on Transfer Learning.
Table 4b shows the transfer learning results with ImageNet1K pretrained weights.
Our model outperforms DeiT (Touvron et al., 2020) by 2 ∼ 3% on ChestXRay8, which has a larger domain gap from ImageNet1K than other four datasets.
We think this is because the hierarchical feature models could provide better generalization for the domain gap compared to DeiT which uses isotropic spatial resolution throughout the whole network.
4. O BJECT D ETECTION AND S EMANTIC S EGMENTATION
Dataset.
We use MS COCO 2017 to validate our models (Lin et al., 2014) on object detection.
COCO 2017 dataset contains 118K images for training and 5K images for validation across 80 categories.
We also test our model on keypoint detection and results are shown in Sec.
B. On the other hand, we validate our model with semantic segmentation on ADE20K
(Zhou et al., 2017).
The ADE20K dataset contains 20k images in the training set, 2k images for validation.
Training and Evaluation.
For object detection, we adopt RetinaNet (Lin et al., 2017) and MaskRCNN (He et al., 2017) as our detection framework.
We simply replace the backbone with RegionViT, and then output the local tokens at each stages as multi-scale features for the detector.
The regional tokens are not used in the detection.
Before sending the features to the detector framework, we add new layer normalization layers to normalize the features.
We use RegionViT-S and RegionViT-B as the backbone and the weights are initialized from ImageNet1K pretraining.
The shorter side and longer side of the input image is resized to 672 and 1,120, respectively.
We train our models based on the settings in 1 × and 3 × schedule in Detectron2 (Wu et al., 2019) for object detection.
More training details could be found in A.2.
For semantic segmentation, we adopt Semantic FPN as the framework (Kirillov et al., 2019) and use RegionViT as the backbone.
We initialize models with
ImageNet1K weights, and mostly follow the training receipt in Twins’ paper (Chu et al., 2021a). More training details can be found in Sec. A.3. Results on Object Detection.
Table 5 compares RegionViT with other methods.
For the smaller models, RegionViT-S and RegionViT-S+ achieve similar accuracy while being moderately efﬁcient in terms of FLOPs.
We ﬁnd that the position encoding generator (PEG) proposed by (Chu et al., 2021b) could help the detection accuracy substantially, so we also provide the results with PEG, which are also used in Twins (Chu et al., 2021a).
For the middle size models with RetinaNet, RegionViT-B are slightly worse than Twins with 1 × schedule; however, our model catches up the performance with 3 × schedule.
When comparing the large models under similar FLOPs (RegionViT-B+ w/ PEG † ), we further train our model with the similar resolution used by other works, and observe that our models outperform all others while being more parameter efﬁcient (models marked with † in Table 5).
[Table]
Results on Semantic Segmentation.
Table 6 compares ours to Swin (Liu et al., 2021) and Twins.
As PEG is useful for object detection, we show the results with PEG.
Our models achieved signiﬁcant improvement with similar FLOPs for both models.
This suggests that the proposed R2L attention can effectively model global context [Table]
4. A CTION R ECOGNITION
Datasets and Setup.
We validate our approach on Kinetics400 (K400) (Kay et al., 2017) and Something-Something V2 (SSV2) (Goyal et al., 2017).
We adopt the divided-space-time attention in TimeSformer (Bertasius et al., 2021) as the temporal modeling to perform action recognition experiments.
More details could be found in Sec.
A.4.
[Table]
Results.
Table 7 shows that with RegionViT as the backbone, the model can be much more efﬁcient with competitive accuracy to TimeSformer, which uses vanilla ViT as the backbone.
RegionViT-M could reduce more than 50% FLOPs and parameters than the TimeSformer.
This not only shows the importance of spatial modeling in action recognition but also validates that our proposed model can also be extended for efﬁcient action recognition.
4. A BLATION S TUDY
We perform the following experiments to verify the effectiveness of different components in RegionViT. All experiments are conducted based on RegionViT-S.
[Table]
Regional Tokens.
Table 8 shows the results with and without regional tokens on three tasks, and Table A5 includes FLOPs and parameters comparison.
For image classiﬁcation, the regional tokens provide around 0.4% improvement with negligible overhead in both computations (6%) and parameters (0.7%).
On the other hand, the regional tokens clearly improve the performance of object detection and semantic segmentation with negligible overhead ( < 2% on both
FLOPs and parameters).
This is because dense prediction tasks require more multi-scale features with global contextual information (provided by the regional tokens) than image classiﬁcation.
[Table]
Downsampling.
Table 9 shows different approaches for downsampling the patches.
3 ×kernel size achieves better results than 2 × 2 convolution, because the kernel is non-overlapped with 2 × 2 convolution, which limits the interaction among local tokens.
Moreover, using regular convolution does not improve the performance but increases computations and parameters.
Thus, we use 3 × 3 depthwise convolution for downsampling.
[Table]
Weight sharing between RSA and LSA as well as downsampling.
Table 10 shows the results with and without weight sharing.
As seen from the table, using separated weights for RSA and LSA only slightly improves the accuracy but signiﬁcantly increases the model size.
This suggests sharing parameters between RSA and LSA sufﬁces for learning local and global information in our approach.
[Table]
Keys in LSA.
Table 11 studies the number of regional tokens in LSA.
When using all regional tokens in the LSA, it provides the local tokens the possibility to explore all global information.
Nonetheless, this model only improves 0.1% but with 13% more computations, which suggests that when considering the regional tokens, the one associated with the current region is sufﬁcient to achieve good performance.
[Table]
Position information.
Table 12 compares performance of different combinations of absolute position embedding (APE) and relative position bias (RPB).
The models with RPB achieve better accuracy with similar FLOPs and parameters.
Although the model with APE could slightly improve the performance, it limits the model to run at a ﬁxed resolution, which is not suitable for vision tasks where image size could be varied.
Thus, we only adopt the relative position bias in our models.
[Table]
Overlapped windows.
Table 13 compares the results with and without overlapped windows.
The overlapping ratio of neighboring windows is 50% and hence this model allows more interactions among local tokens.
It improves
RegionViT-S by 0.2% but increases the FLOPs by 3.5 × .
This suggests that the information exchange between the border of windows is sufﬁciently covered by the proposed R2L attention.
[Table]
Comparison between R2L attention and shifted-window attention.
Table 14 shows the comparison between shifted-window attention (Liu et al., 2021) and proposed R2L attention.
R2L attention outperforms shifted-window attention by 0.7%, which shows that our proposed R2L attention can effectively model global information to achieve good performance.
C ONCLUSION
In this paper, we propose a new ViT architecture that exploits the pyramid structure used in most of
CNNs to provide multi-scale features and hence, the models can be more easily extended to different vision applications, like object detection.
Moreover, the proposed regional-to-local attention relaxes the memory overhead of performing self-attention on the ﬁne-grained image tokens by limiting the scope of attention but still keeping the capability to explore global information.
Extensive experiments on several standard benchmark datasets well demonstrate that our proposed models outperform or are on par with many concurrent ViT variants on four vision applications, including image classiﬁcation, object and keypoint detection, semantic segmentation and action recognition.
E THICS S TATEMENT
Our work introduces a memory-friendly and efﬁcient self-attention method for transformer models, which have wide-ranging applications to image classiﬁcation, object detection, human activity recognition, etc.
By improving efﬁciency, our work can have a positive effect on these applications in society, allowing faster processing.
E.g., this could enable faster responses to detected events such as medical emergencies or detecting defects in manufacturing parts.
Lower computation costs could also save energy and therefore have a positive impact on the environment.
Negative impacts of our research are difﬁcult to predict, however, it shares many of the pitfalls associated with deep classiﬁcation models.
E.g., Image and video classiﬁcation systems have negative implications on privacy and could be used by malicious actors or governments to infringe on the privacy of citizens.
Future research into private and ethical aspects of visual recognition is an important direction.
R EPRODUCIBILITY S TATEMENT
In order to reproduce our results, we describe the details of the hyperparameters used in our training in Appendix A for all the tasks and we also attach our codes for image classiﬁcation as part of the supplemental material.
We will publicly release all the codes and models after acceptance.
In the attached codes, we set all the hyperparameters as their default value, so that users can re-run our experiments with exactly the same hyperparameters.
The README ﬁle also describes the requirement to build necessary Python environment for running the codes.
A PPENDIX
Summary This appendix contains the following additional details.
First, in Sec.
A, we describe the training details on image classiﬁcation, object and keypoint detection, semantic segmentation and action recognition separately.
Then, we provide detailed results on object detection and ablation studies
[Table]
The supplementary materials include our codes to reproduce our results on image classiﬁcation and the README ﬁle in the attached codes ( RegionViT Code.zip ) also provides the detailed instructions to train and evaluate the networks.
A T RAINING D ETAILS A. I MAGE C LASSIFICATION
We follow DeiT (Touvron et al., 2020) to train our models on ImageNet1K (Deng et al., 2009) except that we use batch size 4,096 with base learning rate 0.004 and the warm-up epochs is 50.
We adopt the AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate scheduler (Loshchilov & Hutter, 2017).
We apply Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019), RandomErasing (Zhong et al., 2020), label smoothing (Szegedy et al., 2016), RandAugment (Cubuk et al., 2020) and instance repetition (Hoffer et al., 2020).
During training, we randomly crop a 224 × 224 region and take a 224 × 224 center crop after resizing the shorter side to 256 for evaluation.
While pretraining on ImageNet21K (Deng et al., 2009), we use similar settings with slight modiﬁcations.
We train the model using 120 epochs with 5 epochs for warmup, weight-decay of 0.01 and base learning rate of 0.001.
For transfer learning experiments, we ﬁnetune the ImageNet1K pretrained models with 1, epochs, batch size of 768, learning rate of 0.01, SGD optimizer, weight decay of 0.0001, and using the same data augmentation in training on ImageNet1K. During evaluation, we resize the shorter side of an image to 256 and then take a 224 × 224 region at the center and report top-1 accuracy.
For ﬁnetuning experiments from ImageNet21K, we ﬁnetune the models with higher resolution of 384 × 384, for 30 epochs with cosine learning rate scheduler, base linear rate of 0.002, weight-decay of 1e-8.
Moreover, as it is trained on larger resolution, we adjust the window size M to have the same number of regional tokens at the ﬁrst stage, e.g., we change the window size to 12 for the the models trained at 224 × 224 with window size 7 .
We adopt the bicubic interpolation to upsample the weights of tokenizations and relative position bias.
During evaluation, we directly resize the shorter side to 384 and then take center 384 × 384 crop.
We trained the models with 32 GPUs.
[Table]
A. O BJECT D ETECTION AND K EYPOINT D ETECTION
We adopt RetinaNet (Lin et al., 2017) and MaskRCNN (He et al., 2017) as our detection framework and use KeypoinyRCNN for keypoint detection, and we simply replace the backbone with RegionViT, and then output the local tokens at each stages as multi-scale features for the detector.
The regional tokens are not used in the detection.
Before sending the features to the detector framework, we add new layer normalization layers to normalize the features.
We use RegionViT-S and RegionViT-B as the backbone and the weights are initialized from ImageNet1K pretraining.
We train our models based on the settings in 1 × schedule in Detectron2 (Wu et al., 2019), i.e., the batch size is 16 and the learning rate is set to 0.0001 which drops 10 × at the 60,000-th and 80,000-th iteration.
We use
AdamW optimizer with weight-decay of 0.05, and resize the shorter and longer side of an image to 672 and 1,120, respectively.
We also adopt drop-path when ﬁnetuning the detector, with a rate set to 0.2 for both RetinaNet and MaskRCNN.
We trained the models with 8 GPUs.
When training with longer schedule (3 × ), we adopt stronger data augmentation (multi-scale training) used in Swin (Liu et al., 2021) or Twins (Chu et al., 2021a) rather than ﬁxed-size training used in 1 × schedule.
A. S EMANTIC S EGMENTATION
We adopt Semantic FPN as our semantic segmentation framework (Kirillov et al., 2019) by replacing the backbone network with RegionViT like the object detection task.
We train the models with
ImageNet1K pretrained weights.
We mostly follow the training receipt in Twins (Chu et al., 2021a)
The shorter side of the image is resized to 448 and the longer side won’t exceed 1792, and then we crop a 448 × 448 region for training.
During the evaluation, we take the whole image after resizing as illustrated above.
We train the model with AdamW optimizer with learning rate 1e-4 for 80k iterations, and the learning rate is decayed with poly schedule (power is 0.9).
The weight decay and drop-path rate is 0.05 and 0.2, respectively.
We trained all models with 8 GPUs.
A. A CTION R ECOGNITION
We adopt the divided-space-time attention in TimeSformer (Bertasius et al., 2021) as the temporal modeling to perform action recognition experiments.
More speciﬁcally, we plugin the temporal attention before every R2L transformer encoder and ﬁnetune the model pretrained with ImageNet1K (instead of ImageNet21K).
We use the AdamW optimizer (Loshchilov & Hutter, 2019) with base linear rate of 2e-4 and weight-decay of 0.0001.
We apply the same Mixup (Zhang et al., 2018),
CutMix (Yun et al., 2019) and drop path (Tan & Le, 2019) in the image classiﬁcation.
For the input, we uniformly sample 8 frames from the whole video and the 224 × 224 region is cropped after shorter side of images is resized to the range of 256 to 320, resulting the size of the input video as 8 × 224 × 224.
During evaluation, we use the same way to sample frames but take 3 224 × 224 spatial crops (top-left, center and bottom-right) after resizing shorter side of image to 256, and then ensemble the predictions as ﬁnal prediction for the input video.
We trained the models with 16 GPUs.
[Table] [Table]
B M ORE D ETAILED R ESULTS
We provided more details of some Tables shown in the main paper here.
Table A2 and Table A include more metrics as compared to Table 5.
Table A4 shows the results on keypoint detection.
RegionViT outperforms ResNet-50 with moderate increases in FLOPs and parameters.
This suggests that RegionViT can model the global context in keypoint detection as well.
Table A5 includes complexity comparison for the ablation study on regional tokens as shown in Table 8. [Table] [Table]