C ROSS F ORMER : A V ERSATILE V ISION T RANSFORMER H INGING ON C ROSS - SCALE A TTENTION
A BSTRACT
Transformers have made great progress in dealing with computer vision tasks.
However, existing vision transformers have not yet possessed the ability of building the interactions among features of different scales, which is perceptually important to visual inputs.
The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacriﬁcing small-scale (ﬁne-grained) features of the embeddings and also disabling the cross-scale interactions.
To this end, we propose C ross-scale E mbedding L ayer (CEL) and L ong S hort D istance
A ttention (LSDA).
On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with crossscale features.
On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings.
Through the above two designs, we achieve cross-scale attention.
Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images.
Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs.
Extensive experiments show that CrossFormer outperforms the other vision transformers on image classiﬁcation, object detection, instance segmentation, and semantic segmentation tasks.
I NTRODUCTION
It turns out that transformer (Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020) has achieved great success in the ﬁeld of natural language processing (NLP).
Beneﬁtting from its selfattention module, transformer is born with the key ability to build long-distance dependencies.
Since long-distance dependencies are also needed by a number of vision tasks (Zhang & Yang, 2021; Chu et al., 2021), a surge of research work (Dosovitskiy et al., 2021; Touvron et al., 2021; Wang et al., 2021) has been conducted to explore various transformer-based vision architectures.
A transformer requires a sequence of embeddings 2 ( e.g. , word embeddings) as input.
To adapt this requirement to typical vision tasks, most existing vision transformers (Dosovitskiy et al., 2021;
Touvron et al., 2021; Wang et al., 2021; Liu et al., 2021b) produce embeddings by splitting an input image into equal-sized patches.
For example, a 224 × 224 image can be split into 56 ×patches of size 4 × 4 , and these patches are projected through a linear layer to yield an embedding sequence.
Inside a certain transformer, self-attention is engaged to build the interactions between any two embeddings.
Thus, the computational or memory cost of the self-attention module is O ( N 2 ) , where N is the length of an embedding sequence.
Such a cost is too big for a visual input because its embedding sequence is much longer than that of NLP.
Therefore, the recently proposed vision transformers (Wang et al., 2021; Liu et al., 2021b; Lin et al., 2021) develop multiple substitutes to approximate the vanilla self-attention module with a lower cost.
Though the aforementioned vision transformers have made some progress, they do not explicitly utilize features of different scales, whereas multi-scale features are very vital for a lot of vision tasks.
For example, an image often contains many objects of different scales, and to fully understand the image, building the interactions among those objects is helpful.
Besides, some particular tasks such as instance segmentation need the interactions between large-scale (coarse-grained) features and small-scale (ﬁne-grained) ones.
Existing vision transformers fail to deal with the above cases due to two reasons: (1) The embeddings are generated from equal-sized patches.
Though these patches theoretically have a chance to extract any scale features if only the receptive ﬁeld is large enough, it is difﬁcult to promise that they can learn appropriate multi-scale features automatically in practice.
(2) Inside the self-attention module, adjacent embeddings are often grouped together and merged (Wang et al., 2021).
Since the number of groups is smaller than that of embeddings, such a behavior can reduce the computational budget of the self-attention.
In this case, however, even if embeddings have both small-scale and large-scale features, merging operations will lose the small-scale features of each individual embedding, thereby disabling the cross-scale attention.
To enable the building of cross-scale interactions, we co-design a novel embedding layer and a selfattention module as follows.
1) Cross-scale Embedding Layer (CEL) – Following Wang et al.
(2021), we also employ a pyramid structure for our transformer, which naturally splits the vision transformer model into multiple stages.
CEL appears at the start of each stage, which receives last stage’s output (or an input image) as input and samples patches with multiple kernels of different scales ( e.g. , 4 ×or 8 × 8 ).
Then, each embedding is constructed by projecting and concatenating these patches.
Through this way, we enforce some dimensions ( e.g. , dimensions from 4 × 4 patches) to focus on small-scale features only, while others ( e.g. , those from 8 × 8 patches) have a chance to learn largescale features, leading to an embedding with explicitly cross-scale features.
Long Short Distance
Attention (LSDA) –
We propose a substitute of the vanilla self-attention, but to preserve small-scale features, the embeddings will not be merged.
In contrast, we split the self-attention module into
Short Distance Attention (SDA) and Long Distance Attention (LDA).
SDA builds the dependencies among neighboring embeddings, while LDA takes charge of the dependencies among embeddings far away from each other.
The proposed LSDA can also reduce the cost of the self-attention module like previous studies (Wang et al., 2021), but different from them, LSDA does not undermine either small-scale or large-scale features.
As a result, attention with cross-scale interactions is enabled.
Besides, following prior work (Shaw et al., 2018; Liu et al., 2021b), we employ a relative position bias for embeddings’ position representations.
The Relative Position Bias (RPB) only supports ﬁxed image/group size 3 .
However, image size for many vision tasks such as object detection is variable, so does group size for many architectures, including ours.
To make the RPB more ﬂexible, we further introduce a trainable module called Dynamic Position Bias (DPB), which receives two embeddings’ relative distance as input and outputs their position bias.
The DPB module is optimized end-to-end in the training phase, inducing an ignorable cost but making RPB apply to variable image/group size.
All our proposed modules can be implemented with about ten lines of code.
Based on them, we construct four versatile vision transformers of different sizes, dubbed CrossFormers .
Other than image classiﬁcation, the proposed CrossFormer can handle a variety of tasks with variable-sized inputs such as object detection.
Experiments on four representative vision tasks ( i.e. , image classiﬁcation, object detection, instance segmentation, and semantic segmentation) demonstrate that CrossFormer outperforms the other state-of-the-art vision transformers on all the tasks.
Remarkably, the performance gains brought by CrossFormer are substantially signiﬁcant on dense prediction tasks, e.g. , object detection and instance/semantic segmentation.
It is worth highlighting our contributions as follows:
• We propose cross-scale embedding layer (CEL) and long short distance attention (LSDA), which together compensate for existing transformers’ incapability of building cross-scale attention.
• The dynamic position bias module (DPB) is further proposed to make the relative position bias more ﬂexible, i
e. , accommodating variable image size or group size.
C EL ( 4 × 4 ,× 8 ,× 16 ,× 32 )
C r o ss
F o r m e r B l o c k × " !
C EL ( 2 × 2 ,× 4 )
C r o ss
F o r m e r B l o c k × " "
C EL ( 2 × 2 ,× 4 )
C r o ss
F o r m e r B l o c k × " #
C EL ( 2 × 2 ,× 4 )
C r o ss
F o r m e r B l o c k × "
$
C l a ss i f
i ca ti on H ea d
StageStage- StageStage- ! !
4 ×$ ! 4 ×% ! ! 8 ×$ ! 8 ×2% ! !
16×$ ! 16 ×4% ! !
32×$ ! 32 ×8%
LN
SDA
LN
MLP
DPB
LN
LDA
LN
MLP
DPB (b) Two consecutive CrossFormer blocks. $ ! (a) The architecture of CrossFormer for image classification.
• Multiple CrossFormers with different sizes are constructed, and we corroborate their effectiveness through sufﬁcient experiments on four representative vision tasks  B ACKGROUND Vision Transformers. Motivated by the transformers developed for NLP, researchers design speciﬁc visual transformers for vision tasks to take full advantage of their powerful attention mechanism.
In particular, ViT and DeiT transfer the original transformer Vaswani et al.
(2017) to vision tasks (Touvron et al., 2021;
Dosovitskiy et al., 2021), achieving impressive performance.
Later, PVT (Wang et al., 2021), HVT (Pan et al., 2021), Swin (Liu et al., 2021b), and ViTAE (Xu et al., 2021b) introduce a pyramid structure into the visual transformers, greatly decreasing the number of patches in the later layers of a respective model.
They also extend the visual transformers to other vision tasks like object detection and segmentation (Wang et al., 2021; Liu et al., 2021b).
Cross-scale Feature Extraction.
Szegedy et al.
; Tan & Le (2019) use multi-scale convolutional kernels at every layer of models for cross-scale features, while the largest kernel size can only be 7 × 7 , and larger kernel size will induce unaccpetable computational budget.
For vision transformers, CoaT (Xu et al., 2021a) uses multi-scale features at later layers of models by mixing features from different layers; CViT (Chen et al., 2021a) keeps embeddings of different scales in different branches, and introduces cross-scale interaction through self-attention between branches.
More discussions can be seen in the appendix C.1.
Sparse Self-attention.
Many substitutes have been proposed (Liu et al., 2021b;
Wang et al., 2021;
Chu et al., 2021; Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020) to alleviate the cost of the vanilla self-attention module.
Instead of going self-attention among all embeddings, each embedding only interacts with part of other embeddings, thus changing dense self-attention to sparse attention.
More discussions can be seen in the appendix C.2.
Position Representations.
To make the respective model aware of position information of embeddings, many different position representations (Vaswani et al., 2017) are proposed.
For example,
Dosovitskiy et al.
(2021) directly added the embeddings with the vectors that contain absolute position information.
In contrast, Relative Position Bias (RPB) (Shaw et al., 2018) resorts to position information indicating the relative distance of two embeddings.
C ROSS F ORMER
The overall architecture of CrossFormer is plotted in Figure 1.
Following (Wang et al., 2021; Liu et al., 2021b; Lin et al., 2021), CrossFormer also employs a pyramid structure, which naturally splits the transformer model into four stages.
Each stage consists of a cross-scale embedding layer (CEL,
Section 3.1) and several CrossFormer blocks (Section 3.2).
A CEL receives last stage’s output (or an input image) as input and generates cross-scale embeddings.
In this process, CEL (except that in Stage-1 ) reduces the number of embeddings to a quarter while doubles their dimensions for a pyramid structure.
Then, several CrossFormer blocks, each of which involves long short distance attention (LSDA) and dynamic position bias (DPB), are set up after CEL.
A specialized head ( e.g. , the classiﬁcation head in Figure 1) follows after the ﬁnal stage accounting for a speciﬁc task.
C ROSS - SCALE E MBEDDING L AYER (CEL)
Cross-scale embedding layer (CEL) is leveraged to generate input embeddings for each stage.
Figure 2 takes the ﬁrst CEL, which is ahead of Stage-1 , as an example.
It receives an image as input, then sampling patches using four kernels of different sizes.
The stride of four kernels is kept the same
[Figure]
means the total dimension of the embedding.
so that they generate the same number of embeddings 4 .
As we can observe in Figure 2, every four corresponding patches have the same center but different scales, and all these four patches will be projected and concatenated as one embedding.
In practice, the process of sampling and projecting can be fulﬁlled through four convolutional layers.
For a cross-scale embedding, one problem is how to set the projected dimension of each scale.
The computational budget of a convolutional layer is proportional to K 2 D 2 , where K and D represent kernel size and input/output dimension, respectively (assuming the input dimension equals to the output dimension).
Therefore, given the same dimension, a large kernel consumes more budget than a smaller one.
To control the total budget of the CEL, we use a lower dimension for large kernels while a higher dimension for small kernels.
Figure 2 provides the speciﬁc allocation rule in its subtable, and a 128 -dimensional example is given.
Compared with allocating the dimension equally, our scheme saves much computational cost but does not explicitly affect the model’s performance.
The cross-scale embedding layers in other stages work in a similar way.
As shown in Figure 1, CELs for Stage-2/3/4 use two different kernels ( 2 × 2 and 4 × 4 ).
Further, to form a pyramid structure, the strides of CELs for Stage-2/3/4 are set as 2 × 2 to reduce the number of embeddings to a quarter.
3. C ROSS F ORMER B LOCK
Each CrossFormer block consists of a long short distance attention module ( i.e. , LSDA, which involves a short distance attention (SDA) module or a long distance attention (LDA) module) and a multilayer perceptron (MLP).
As shown in Figure 1b, SDA and LDA appear alternately in different blocks, and the dynamic position bias (DPB) module works in both SDA and LDA for obtaining embeddings’ position representations.
Following the prior vision transformers, residual connections are used in each block.
3.2. L ONG S HORT D ISTANCE A TTENTION (LSDA)
We split the self-attention module into two parts: short distance attention (SDA) and long distance attention (LDA).
For SDA, every G × G adjacent embeddings are grouped together.
Figure 3a gives an example where G = 3 .
For LDA with input of size S × S , the embeddings are sampled with a ﬁxed interval I .
For example in Figure 3b ( I = 3 ), all embeddings with a red border belong to a group, and those with a yellow border comprise another group.
The group’s height or width for LDA is computed as G = S/I ( i.e. , G = 3 in this example).
After grouping embeddings, both SDA and
LDA employ the vanilla self-attention within each group.
As a result, the memory/computational cost of the self-attention module is reduced from O ( S 4 ) to O ( S 2 G 2 ) , and G (cid:28) S in most cases.
It is worth noting that the effectiveness of LDA also beneﬁts from cross-scale embeddings.
Specifically, we draw all the patches comprising two embeddings in Figure 3b.
As we can see, the smallscale patches of two embeddings are non-adjacent, so it is difﬁcult to judge their relationship without the help of the context.
In other words, it will be hard to build the dependency between these two embeddings if they are only constructed by small-scale patches ( i.e. , single-scale feature).
On the contrary, adjacent large-scale patches provide sufﬁcient context to link these two embeddings, which makes long-distance cross-scale attention easier and more meaningful. [Figure]
We provide the pseudo-code of LSDA in the appendix (A.1).
Based on the vanilla multi-head selfattention, LSDA can be implemented with only ten lines of code.
Further, only reshape and permute operations are used, so no extra computational cost is introduced.
3.2. D YNAMIC P OSITION B IAS (DPB)
Relative position bias (RPB) indicates embeddings’ relative position by adding a bias to their attention.
Formally, the LSDA’s attention map with RPB becomes:
[Equation]
where Q , K , V ∈ R G 2 × D represent query, key, value in the self-attention module, respectively.
√ d is a constant normalizer, and B ∈ R G 2 × G [Equation] is the RPB matrix.
In previous work (Liu et al., 2021b),
B i,j = ˆ B ∆ x ij , ∆ y ij , where ˆ B is a ﬁxed-sized matrix, and (∆ x ij , ∆ y ij ) is the coordinate distance between the i th and j th embeddings.
It is obvious that the image/group size is restricted in case that (∆ x ij , ∆ y ij ) exceeds the size of ˆ B .
In contrast, we propose an MLP-based module called DPB to generate the relative position bias dynamically, i.e. ,
[Equation]
The structure of DPB is displayed in Figure 3c.
Its non-linear transformation consists of three fullyconnected layers with layer normalization (Ba et al., 2016) and ReLU (Nair & Hinton, 2010).
The input dimension of DPB is 2 , i.e. , (∆ x ij , ∆ y ij ) , and intermediate layers’ dimension is set as D/ 4 , where D is the dimension of embeddings.
The output B ij is a scalar, encoding the relative position feature between the i th and j th embeddings.
DPB is a trainable module optimized along with the whole transformer model.
It can deal with any image/group size without worrying about the bound of (∆ x ij , ∆ y ij ) .
In the appendix (A.2), we prove that DPB is equivalent to RPB if the image/group size is ﬁxed.
In this case, we can transform a trained DPB to RPB in the test phase.
We also provide an efﬁcient O ( G 2 ) implementation of DPB when image/group size is variable (the complexity is
O ( G 4 ) in a normal case because B ∈ R G 2 × G 2 ).
3. V ARIANTS OF C ROSS F ORMER
Table 1 lists the detailed conﬁgurations of CrossFormer’s four variants (-T, -S, -B, and -L for tiny, small, base, and large, respectively) for image classiﬁcation.
To re-use the pre-trained weights, the models for other tasks ( e.g. , object detection) employ the same backbones as classiﬁcation except that they may use different G and I .
Speciﬁcally, besides the conﬁgurations same to classiﬁcation, we also test with G 1 = G 2 = 14 , I 1 = 16 , and I 2 = 8 for the detection (and segmentation) models’ ﬁrst two stages to adapt to larger images.
The speciﬁc conﬁgurations are described in the appendix (A.3).
Notably, the group size or the interval ( i.e. , G or I ) does not affect the shape of weight tensors, so the backbones pre-trained on ImageNet can be readily ﬁne-tuned on other tasks even though they use different G or I . [Table]
E XPERIMENTS
The experiments are carried out on four challenging tasks: image classiﬁcation, object detection, instance segmentation, and semantic segmentation.
To entail a fair comparison, we keep the same data augmentation and training settings as the other vision transformers as far as possible.
The competitors are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021),
T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a),
Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 2021), ViL (Zhang et al., 2021a), CAT (Lin et al., 2021), ResT (Zhang & Yang, 2021), TransCNN (Liu et al., 2021a),
Shufﬂe (Huang et al., 2021), BoTNet (Srinivas et al., 2021), and RegionViT (Chen et al., 2021b).
4. I MAGE C LASSIFICATION
Experimental Settings.
The experiments on image classiﬁcation are done with the ImageNet (Russakovsky et al., 2015) dataset.
The models are trained on 1 .
M training images and tested on 50 K validation images.
The same training settings as the other vision transformers are adopted.
In particular, we use an AdamW (Kingma & Ba, 2015) optimizer training for 300 epochs with a cosine decay learning rate scheduler, and 20 epochs of linear warm-up are used.
The batch size is 1, split on 8 V100 GPUs.
An initial learning rate of 0.001 and a weight decay of 0.05 are used.
Besides, we use drop path rate of 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 for CrossFormer-T, CrossFormer-S, CrossFormer-B,
CrossFormer-L, respectively.
Further, Similar to Swin (Liu et al., 2021b), RandAugment (Cubuk et al., 2020), Mixup (Zhang et al., 2018a), Cutmix (Yun et al., 2019), random erasing (Zhong et al., 2020), and stochastic depth (Huang et al., 2016) are used for data augmentation.
Results.
The results are shown in Table 2.
As we can see, CrossFormer achieves the highest accuracy with parameters and FLOPs comparable to other state-of-the-art vision transformer structures.
In speciﬁc, compared against strong baselines DeiT, PVT, and Swin, our CrossFormer outperforms them at least absolute 1 . 2% in accuracy on small models.
Further, though RegionViT achieves the same accuracy ( 82 . 5% ) as ours on a small model, it is 0 .
7% ( 84 . 0% vs. 83 .
3% ) absolutely lower than ours on the large model.
4. O BJECT D ETECTION AND I NSTANCE S EGMENTATION
Experimental Settings.
The experiments on object detection and instance segmentation are both done on the COCO 2017 dataset (Lin et al., 2014), which contains 118 K training and 5 K val images.
[Table]
[Table]
− −
[Table]
[Table]
We use MMDetection-based (Chen et al., 2019)
RetinaNet (Lin et al., 2020) and Mask R-CNN (He et al., 2017) as the object detection and instance segmentation head, respectively.
For both tasks, the backbones are initialized with the weights pre-trained on ImageNet.
Then the whole models are trained with batch size 16 on 8 V100 GPUs, and an AdamW optimizer with an initial learning rate of 1 × 10 − 4 is used.
Following previous works, we adopt 1 × training schedule ( i.e. , the models are trained for 12 epochs) when taking RetinaNets as detectors, and images are resized to 800 pixels for the short side.
While for Mask R-CNN, both 1 × and 3 × training schedules are used.
It is noted that multi-scale training (Carion et al., 2020) is also employed when taking 3 × training schedules.
Results.
The results on RetinaNet and Mask R-CNN are shown in Table 3 and Table 4, respectively.
As we can see, the second-placed architecture changes along with the experiment, that is, these architectures may perform well on one task but poorly on another task.
In contrast, our CrossFormer outperforms all the others on both tasks (detection and segmentation) with both model sizes (small and base).
Further, CrossFormer’s performance gain over the other architectures gets sharper when enlarging the model, indicating that CrossFormer enjoys greater potentials.
4. S EMANTIC S EGMENTATION
Experimental Settings.
ADE20K
(Zhou et al., 2017) is used as the benchmark for semantic segmentation.
It covers a broad range of 150 semantic categories, including 20 K images for training
[Table] [Table]
and 2 K for validation. Similar to models for detection, we initialize the backbones with weights pretrained on ImageNet, and MMSegmentation-based (Contributors, 2020) semantic FPN and UPerNet (Xiao et al., 2018) are taken as the segmentation head.
For FPN (Kirillov et al., 2019), we use an AdamW optimizer with learning rate and weight deacy of 1 × 10 − 4 .
Models are trained for 80 K iterations with batch size 16 .
For UPerNet, an AdamW optimizer with an initial learning rate of 6 × 10 − 5 and a weight decay of 0 .
01 is used, and models are trained for 160 K iterations.
Results.
All results are shown in Table 5.
Similar to object detection, CrossFormer exhibits a greater performance gain over the others when enlarging the model.
For example, CrossFormer-T achieves 1 .
4% absolutely higher on IOU than Twins-SVT-B, but CrossFormer-B achieves 3 .
1% absolutely higher on IOU than Twins-SVT-L.
Totally, CrossFormer shows a more signiﬁcant advantage over the others on dense prediction tasks ( e.g. , detection and segmentation) than on classiﬁcation, implying that cross-scale interactions in the attention module are more important for dense prediction tasks than for classiﬁcation.
[Table] [Table]
The base model is CrossFormer-S (
[Table]
[Table]
A BLATION S TUDIES
Cross-scale Embeddings vs. Single-scale Embeddings.
We conduct the experiments by replacing cross-scale embedding layers with single-scale ones.
As we can see in Table 6, when using single-scale embeddings, the 8 × 8 kernel in Stage-1 brings 0 .
4% ( 81 . 9% vs. 81 .
5% ) absolute improvement compared with the 4 × 4 kernel.
It tells us that overlapping receptive ﬁelds help improve the model’s performance.
Besides, all models with cross-scale embeddings perform better than those with single-scale embeddings.
In particular, our CrossFormer achieves 1% ( 82 . 5% vs. 81 .
5% ) absolute performance gain compared with using single-scale embeddings for all stages.
For cross-scale embeddings, we also try several different combinations of kernel sizes, and they all show similar performance ( 82 .
3% ∼ 82 .
In summary, cross-scale embeddings can bring a large performance gain, yet the model is relatively robust to different choices of kernel size.
LSDA vs. Other Self-attentions.
Two self-attention modules used in PVT and Swin are compared.
Speciﬁcally, PVT sacriﬁces the small-scale features when computing the self-attention, while Swin restricts the self-attention in a local region, giving up the long-distance attention.
As we can observe in Table 7a, compared against the PVT-like and Swin-like self-attention mechanisms, our CrossFormer outperforms them at least absolute 0 .
6% accuracy ( 82 . 5% vs. 81 .
The results show that performing the self-attention in a long-short distance manner is most conducive to improving the model’s performance.
DPB vs. Other Position Representations.
We compare the parameters, FLOPs, throughputs, and accuracies of the models among absolute position embedding (APE), relative position bias (RPB), and DPB.
The results are shown in Table 7b.
DPB-residual means DPB with residual connections.
Both DPB and RPB outperform APE for absolute 0 .
4% accuracy, which indicates that relative position representations are more beneﬁcial than absolute ones.
Further, DPB achieves the same accuracy ( 82 . 5% ) as RPB with an ignorable extra cost; however, as we described in Section 3.2.2, it is more ﬂexible than RPB and applies to variable image size or group size.
The results also show that residual connection in DPB does not help improve or even degrades the model’s performance.
C ONCLUSIONS
We proposed a novel transformer-based vision architecture, namely CrossFormer.
Its core ingredients are Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA), thereby yielding the cross-attention module.
We further proposed a dynamic position bias, making the relative position bias apply to any input size.
Extensive experiments show that CrossFormer achieves superior performance over other state-of-the-art vision transformers on several representative vision tasks.
Particularly, CrossFormer is demonstrated to gain great improvements on object detection and segmentation, which indicates that CEL and LSDA are together essential for dense prediction tasks.
P SEUDO CODE OF LSDA
The pseudo code for LSDA is shown in Algorithm 1.
As we can see, based on the vanilla selfattention module, both SDA and LDA are implemented with only ten lines of code, and only reshape and permute operations are used.
# H: height, W: width, G: group size of SDA/LDA
# x: input tensor (H, W, D) class LSDA(): def forward(x, type): ## group the embeddings if type == "SDA": x = x.reshaspe(H // G, G, W // G, G, D).permute(0, 2, 1, 3, 4) elif type ==
"LDA": x = x.reshaspe(G, H // G, G, W // G, D).permute(1, 3, 0, 2, 4) x = x.reshape(H * W // (G ** 2), G ** 2, D) ## the vanilla self-attention module x = Attention(x) ## un-group the embeddings x = x.reshaspe(H // G, W // G, G, G, D) if type == "SDA": x = x.permute(0, 2, 1, 3, 4).reshaspe(H, W, D) elif type == "LDA": x = x.permute(2, 0, 3, 1, 4).reshaspe(H, W, D) return x
A. D YNAMIC P OSITION B IAS (DPB)
[Equation]
𝑦
Figure 4 gives an example of computing (∆ x ij , ∆ y ij ) with G =in the DPB module.
For a group of size G × G , it is easy to deduce that:
[Equation]
Thus, motivated by the relative position bias, we construct a matrix ˆ
B ∈ R (2 G − 1) ×
(2 G − 1) , where
[Equation]
The complexity of computing ˆ B is O
( G 2 ) .
Then, the bias matrix
B in DPB can be drawn from ˆ B , i.e. ,
[Equation]
When the image/group size ( i.e. , G ) is ﬁxed, both ˆ B and B will be also unchanged in the test phase.
Therefore, we only need to compute ˆ B and B once, and DPB is equivalent to relative position bias in this case.
A. V ARIANTS OF C ROSS F ORMER FOR D ETECTION AND S EGMENTATION
We test two different backbones for dense prediction tasks.
The variants of CrossFormer for dense prediction (object detection, instance segmentation, and semantic segmentation) are in Table 8.
The architectures are the same as those for image classiﬁcation except that different G and I in the ﬁrst two stages are used.
Notably, group size ( i.e. , G and I ) does not affect the shape of weight tensors, so backbones pre-trained on ImageNet can be ﬁne-tuned directly on other tasks even if they use different G and I .
[Table] [Table]
[Table]
B E XPERIMENTS B. O BJECT D ETECTION
Table 9 provides more results on object detection with RetinaNet and Mask-RCNN as detection heads.
As we can see, a smaller ( G, I ) achieves a higher AP than a larger one, but the performance gain is marginal.
Considering that a larger ( G, I ) can save more memory cost, we think ( G 1 = 14 , I 1 = 16 , G 2 = 14 , I 2 = 8) , which accords with conﬁgurations in Table 8, achieves a better trade-off between the performance and cost.
[Table]
Similar to object detection, we test two different conﬁgurations of ( G, I ) for semantic segmentation’s backbones.
The results are shown in Table 10.
As we can see, the memory costs of the two conﬁgurations are almost the same, which is different from experiments on object detection.
Further, when taking semantic FPN as the detection head, CrossFormers ‡ show advantages over CrossFormers on both IOU ( e.g. , 46 . 4 vs. 46 . 0 ) and FLOPs ( e.g. , 209 .
8 G vs. 220 .
7 G).
When taking UPerNet as the segmentation head, a smaller ( G, I ) achieves higher performance like object detection.
B. C LASSIFICATION WITH CEL
Table 11 shows results on ImageNet dataset after plugging CEL into other vision transformers.
As we can see, CEL brings about 0 .
5% performance gain for both Swin Transformer and NesT, which further shows the effectiveness of CEL.
C D ISCUSSION C. C ROSS - SCALE F EATURE E XTRACTION
CEL vs. GoogLeNet/MixConv.
Since GoogLeNet (Szegedy et al., 2015) and MixConv (Tan &
Le, 2019) use multi-scale convolutional kernels at every layer of models, the largest kernel size can only be 7 × 7 , and larger kernel size will induce unaccpetable computational budget.
In contrast,
CEL only introduces cross-scale features at the start of each stage (at most 4 layers), so kernels with very large size (e.g., 32 × 32 ) can also be used for diversiﬁed-size objects.
CEL vs. CoaT/
CViT.
Some vision transformers also take advantage of cross-scale features.
For example, CoaT (Xu et al., 2021a) only uses multi-scale features at later layers by mixing features from front layers, while CEL introduces multi-scale features at the start of each stage, thus tokens embeddings at all layers (not only those at later layers) could be seen as cross-scale.
Besides,
CViT (Chen et al., 2021a) keeps embeddings of different scales in different branches and introduces cross-scale interaction through self-attention between branches.
This method can well preserve the information of each scale, but multiple branches take more memory/computational/parameters cost than a single-branch model.
As a result, for a controllable cost, CViT can only introduce two different scales ( 4 × 4 and 8 × 8 ).
In contrast, CEL only appears at the start of each stage (at mostlayers).
So, CEL can introduce more different scales (e.g., 4 × 4 , 8 × 8 , 16 × 16 , 32 × 32 ) than CViT with ignorable extra cost ( < 0 .
1 M parameters and < 0 .
3 G FLOPs in most cases).
C. S PARSE S ELF - ATTENTION A ND G ROUP C ONVOLUTION
LSDA vs. ShufﬂeNet/IGCNet.
LSDA, ShufﬂeNet (Zhang et al., 2018b) and IGCNets (Zhang et al., 2017) devote to reducing the computational cost by approximating the original operations (self-attention and convolution) in a lower cost.
Nevertheless, they are essentially different: (1)
Different target and dimension : ShufﬂeNet/IGCNet devotes to reducing the budget caused by the large channels of convolution.
So, both ShufﬂeNet and IGCNet do shufﬂe-and-permutation along channel dimension.
While for the self-attention module, the budget mainly attributes to the large spatial size of feature maps (i.e., the large number of embeddings).
Thus, LSDA is mainly operating along the spatial dimension.
(2) Different motivation : During the process of designing LSDA, we ﬁrst consider the locality of images and propose to extract short-distance attention through SDA.
Meanwhile, to preserve the ability of modeling long-distance dependency, LDA is further proposed to ﬁll this gap.
In contrast, ShufﬂeNet/ICGNets do not need to consider these aspects because all channels are equivalent, and locality does not exist along channel dimmension.
Though they also perform convolution among adjacent channels ﬁrst, it is for the sake of fast memory access.
LSDA vs. Other Sparse Self-attention.
Previous works proposed some sparse self-attention, such as Big Bird (Zaheer et al., 2020), Longformer (Beltagy et al., 2020), Sparse Transformer (Child et al., 2019), etc.
Wherein, as a generative model, Sparse Transformer proposed a strided attention in the one-dimensional case, and every pixel can see the pixel generated ahead of itself.
Instead,
LSDA works in two-dimensional case.
Further, LSDA has a co-design with CEL.
As we described in Section 3.2.1 and Figure 3, in long-distance attention (LDA), the small-scale patches of two embeddings are non-adjacent, so it is difﬁcult to judge their relationship without the help of the context.
In other words, it will be hard to build the dependency between these two embeddings if they are only constructed by small-scale patches (i.e., single-scale feature).
On the contrary, adjacent large-scale patches provide sufﬁcient context to link these two embeddings, which makes long-distance cross-scale attention easier and more meaningful.