C ONDITIONAL P OSITIONAL E NCODINGS FOR V ISION T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a conditional positional encoding (CPE) scheme for vision
Transformers (Dosovitskiy et al., 2021; Touvron et al., 2020).
Unlike previous ﬁxed or learnable positional encodings that are predeﬁned and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens.
As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during the training.
Besides, CPE can keep the desired translation equivalence in vision tasks, resulting in improved performance.
We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework.
Built on
PEG, we present Conditional Position encoding Vision Transformer (CPVT).
We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results.
I NTRODUCTION
Recently, Transformers (Vaswani et al., 2017) have been viewed as a strong alternative to Convolutional Neural Networks (CNNs) in visual recognition tasks such as classiﬁcation (Dosovitskiy et al., 2021) and detection (Carion et al., 2020; Zhu et al., 2021).
Unlike the convolution operation in
CNNs, which has a limited receptive ﬁeld, the self-attention mechanism in the Transformers can capture the long-distance information and dynamically adapt the receptive ﬁeld according to the image content.
Consequently, Transformers are considered more ﬂexible and powerful than CNNs, being promising to achieve more progress in visual recognition.
[Figure]
cls
PE
L × Encoders
MLP
Encoder (L-1) ×
Encoders
Embedding cls
Encoder
PEG (L-1) × Encoders
Embedding
GAP
PEG
MLP
MLP (
[Figure]
ViT
[Figure]
L × Encoders
MLP
Encoder (L-1) ×
Encoders
Embedding cls
Encoder
PEG (L-1) × Encoders
Embedding
GAP
PEG
MLP MLP
[Figure]
Embedding cls
PE
L × Encoders
MLP
Encoder (
[Figure]
× Encoders
Embedding cls
Encoder
PEG (
[Figure] [Figure]
Encoders
Embedding
[Figure]
However, the self-attention operation in Transformers is permutation-invariant, which discards the order of the tokens in an input sequence.
To mitigate this issue, previous works (Vaswani et al., 2017; Dosovitskiy et al., 2021) add the absolute positional encodings to each input token (see Figure 1a), which enables order-awareness.
The positional encoding can either be learnable or ﬁxed with sinusoidal functions of different frequencies.
Despite being effective, these positional encodings seriously harm the ﬂexibility of the Transformers, hampering their broader applications.
Taking the learnable version as an example, the encodings are often a vector of equal length to the input sequence, which are jointly updated with the network weights during training.
As a result, the length and the value of the positional encodings are ﬁxed once trained.
During testing, it causes difﬁculties of handling the sequences longer than the ones in the training data.
The inability to adapt to longer input sequences during testing greatly limits the range of generalization.
For instance, in vision tasks like object detection, we expect the model can be applied to the images of any size during inference, which might be much larger than the training images.
A possible remedy is to use bicubic interpolation to upsample the positional encodings to the target length, but it degrades the performance without ﬁne-tuning as later shown in our experiments.
For vision in general, we expect that the models be translation-equivariant.
For example, the output feature maps of CNNs shift accordingly as the target objects are moved in the input images.
However, the absolute positional encoding scheme might break the translation equivalence because it adds unique positional encodings to each token (or each image patch).
One may overcome the issue with relative positional encodings as in (Shaw et al., 2018).
However, relative positional encodings not only come with extra computational costs, but also require modifying the implementation of the standard
Transformers.
Last but not least, the relative positional encodings cannot work equally well as the absolute ones, because the image recognition task still requires absolute position information (Islam et al., 2020), which the relative positional encodings fail to provide.
In this work, we advocate a novel positional encoding (PE) scheme to incorporate the position information into Transformers.
Unlike the predeﬁned and input-agnostic positional encodings used in previous works (Dosovitskiy et al., 2021; Vaswani et al., 2017; Shaw et al., 2018),
the proposed
PE is dynamically generated and conditioned on the local neighborhood of input tokens.
Thus, our positional encodings can change along with the input size and try to keep translation equivalence.
We demonstrate that the vision transformers (Dosovitskiy et al., 2021; Touvron et al., 2020) with our new
PE ( i.e . CPVT, see Figure 1c) achieve even better performance.
We summarize our contributions as,
• We propose a novel positional encoding (PE) scheme, termed conditional position encodings (CPE).
CPE is dynamically generated with Positional Encoding Generators (PEG) and can be effortlessly implemented by the modern deep learning frameworks (Paszke et al., 2019;
Abadi et al., 2016; Chen et al., 2015), requiring no changes to the current Transformer APIs.
• As opposed to widely-used absolute positional encodings, CPE can provide a kind of stronger explicit bias towards the translation equivalence which is important to improve the performance of Transformers.
Moreover, for image classiﬁcation tasks we employ the global average pooling (GAP, also translation-invariant) to replace the default class token (Dosovitskiy et al., 2021).
Our use of GAP in ViT is prior to Swin (Liu et al., 2021) and it further improves the performance by 1% .
• Built on CPE, we propose Conditional Position encoding Vision Transformer (CPVT).
It achieves better performance than previous vison transformers (Dosovitskiy et al., 2021; Touvron et al., 2020).
• CPE is adaptable to arbitrary input sizes, which are required in many important downstream tasks such as segmentation and detection.
Through experiments we show that CPE can boost the segmentation and detection performance for a recent pyramid transformer (Wang et al., 2021) by a clear margin.
R ELATED W ORK 2. V ISION T RANSFORMERS
Transformer (Vaswani et al., 2017) has achieved great success in natural language processing (NLP) and it is used in many state-of-the-art models such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018; 2019), and GPT-3 (Brown et al., 2020).
Due to the ﬂexibility of the transformer, recent works attempt to apply it to vision tasks such as classiﬁcation (Dosovitskiy et al., 2021; Touvron et al., 2020).
It has been shown that the self-attention module is able to replace the convolution operation (Ramachandran et al., 2019), since self-attention layers can attend pixel-grid patterns similarly to the CNN layers (Cordonnier et al., 2020).
One of the issues is that applying self-attention to vision tasks may lead to huge computational costs because an image may have tens of thousands of pixels.
ViT (Dosovitskiy et al., 2021) bypasses the issue by splitting an image into patches.
Specifically, it decomposes a 224 × 224 image into a series of 196 ﬂattened patches (16 × 16 pixels each), which are analogous to a sequence of words in language processing.
Based on ViT, DeiT (Touvron et al., 2020) further makes the transformers more data-efﬁcient, and it can be directly trained on
ImageNet.
Besides, Transformer is used to solve other vision tasks such as detection (Carion et al., 2020; Zhu et al., 2021), segmentation (Wang et al., 2020; Zheng et al., 2020), image generation (Parmar et al., 2018), and low-level tasks (Chen et al., 2020).
We give preliminary knowledge in A.
2. P OSITIONAL E NCODINGS
Since self-attention itself is permutation-equivariant (see A.3), positional encodings are commonly employed to incorporate the order of sequences (Vaswani et al., 2017).
The positional encodings can either be ﬁxed or learnable, while either being absolute or relative.
Vision transformers follow the same fashion to imbue the network with positional information.
Absolute Positional Encoding.
The absolute positional encoding is the most widely used.
In the original transformer (Vaswani et al., 2017), the encodings are generated with the sinusoidal functions of different frequencies and then they are added to the inputs.
Alternatively, the positional encodings can be learnable, where they are implemented with a ﬁxed-dimension matrix/tensor and jointly updated with the model’s parameters with SGD.
Relative Positional Encoding.
The relative position encoding (Shaw et al., 2018) considers distances between the tokens in the input sequence.
Compared to the absolute ones, the relative positional encodings can be translation-equivariant and can naturally handle the sequences longer than the longest sequences during training ( i.e ., being inductive).
A 2-D relative position encoding is proposed for image classiﬁcation in (Bello et al., 2019), showing superiority to 2D sinusoidal embeddings.
The relative positional encoding is further improved in XLNet (Yang et al., 2019b) and
DeBERTa (He et al., 2020), showing better performance.
Other forms.
Complex-value embeddings (Wang et al., 2019) are an extension to model global absolute encodings and show improvement.
RoFormer (Su et al., 2021) utilizes a rotary position embedding to encode both absolute and relative position information for text classiﬁcation.
FLOATER (Liu et al., 2020) proposes a novel continuous dynamical model to capture position encodings.
It is not limited by the maximum sequence length during training, meanwhile being parameter-efﬁcient.
Similar designs to CPE.
Convolutions are used to model local relations in ASR and machine translation (Gulati et al., 2020; Mohamed et al., 2019;
Yang et al., 2019a; Yu et al., 2018).
However, they are mainly limited to 1D signals.
We instead process 2D vision images.
V ISION T RANSFORMER WITH C ONDITIONAL P OSITION E NCODINGS 3. M OTIVATION
In vision transformers, an input image of size H × W is split into patches with size S × S , the number of patches is N = HWS 2 1 .
The patches are added with the same number of learnable absolute positional encoding vectors.
In this work, we argue that the positional encodings used here have two issues.
First, it prevents the model from handling the sequences longer than the learnable PE.
Second, it makes the model not translation-equivariant because a unique positional encoding vector is added to every one patch.
The translation equivalence plays an important role in classiﬁcation because we hope the networks’ responses changes accordingly as the object moves in the image.
can directly be applied to longer sequences.
However, this solution severely deteriorates the performance. This is understandable because the order of the input sequence is an important clue and the model has no way to extract the order without the positional encodings.
The experiment results on
ImageNet are shown in Table 1.
By removing the positional encodings, DeiT-tiny’s performance on
ImageNet dramatically degrades from 72.2% to 68.2%.
[Table]
Second, in DeiT (Touvron et al., 2020), they show that we can interpolate the position encodings to make them have the same length of the longer sequences.
However, this method requires ﬁnetuning the model a few more epochs, otherwise the performance will remarkably drop, as shown in Table 1.
This goes contrary to what we would expect.
With the higher-resolution inputs, we often expect a remarkable performance improvement without any ﬁne-tuning.
Finally, the relative position encodings (Shaw et al., 2018; Bello et al., 2019) can cope with both the aforementioned issues.
However, the relative positional encoding cannot provide absolute position information , which is also important to the classiﬁcation performance (Islam et al., 2020).
As shown in Table 1, the model with relative position encodings has inferior performance (70.5% vs. 72.2%).
3. C ONDITIONAL P OSITIONAL E NCODINGS
We argue that a successful positional encoding for vision tasks should meet these requirements, (1) Making the input sequence permutation-variant and providing stronger explicit bias towards translation-equivariance .
(2) Being inductive and able to handle the sequences longer than the ones during training.
(3) Having the ability to provide the absolute position to a certain degree.
This is important to the performance as shown in (Islam et al., 2020).
In this work, we ﬁnd that characterizing the local relationship by positional encodings is sufﬁcient to meet all of the above.
First, it is permutation-variant because the permutation of input sequences also affects the order in some local neighborhoods.
However, translation of an object in an input image does not change the order in its local neighborhood, i.e ., translation-equivariant (see Section A.3).
Second, the model can easily generalize to longer sequences since only the local neighborhoods of a token are involved.
Besides, if the absolute position of any input token is known, the absolute position of all the other tokens can be inferred by the mutual relation between input tokens.
We will show that the tokens on the borders can be aware of their absolute positions due to the commonly-used zero paddings.
[Figure]
Therefore, we propose positional encoding generators (PEG) to dynamically produce the positional encodings conditioned on the local neighborhood of an input token.
Positional Encoding Generator.
PEG is illustrated in Figure 2.
To condition on the local neighbors, we ﬁrst reshape the ﬂattened input sequence
X ∈ R B × N × C of DeiT back to X (cid:48) ∈
R B × H × W × C in the 2-D image space.
Then, a function (denoted by F in Figure 2) is repeatedly applied to the local patch in X (cid:48) to produce the conditional positional encodings E B × H × W × C .
PEG can be efﬁciently implemented with a 2-D convolution with kernel k ( k ≥ 3 ) and k − 12 zero paddings.
Note that the zero paddings here are important to make the model be aware of the absolute positions, and F can be of various forms such as a various types of convolutions and many others.
3. C ONDITIONAL P OSITIONAL E NCODING V ISION T RANSFORMERS
Built on the conditional positional encodings, we propose our Conditional Positional Encoding Vision Transformers (CPVT).
Except that our positional encodings are conditional, we exactly follow
ViT and DeiT to design our vision transformers and we also have three sizes CPVT-Ti, CPVT-S and
CPVT-B. Similar to the original positional encodings in DeiT, the conditional positional encodings are also added to the input sequence, as shown in Figure 1 (b).
In CPVT, the position where PEG is applied is also important to the performance, which will be studied in the experiments.
In addition, both DeiT and ViT utilize an extra learnable class token to perform classiﬁcation ( i.e ., cls token shown in Figure 1 (a) and (b)).
By design, the class token is not translation-invariant although it can learn to be so.
A simple alternative is to directly replace it with a global average pooling (GAP), which is inherently translation-invariant, resulting in our CVPT-GAP.
Together with the conditional positional encodings, CVPT-GAP can achieve much better image classiﬁcation performance.
E XPERIMENTS 4. S ETUP
Datasets.
Following DeiT (Touvron et al., 2020), we use ILSVRC-2012 ImageNet dataset (Deng et al., 2009) with 1K classes and 1.3M images to train all our models.
We report the results on the validation set with 50K images.
Unlike ViT (Dosovitskiy et al., 2021), we do not use the much larger undisclosed
JFT-300M dataset (Sun et al., 2017).
Model variants.
We have three models with various sizes to adapt to various computing scenarios.
The detailed settings are shown in Table 8 (see B.1).
All experiments in this paper are performed on
Tesla V100 machines.
Training the tiny model for 300 epochs takes about 1.3 days on a single node with 8 V100 GPU cards.
CPVT-S and CPVT-B take about 1.6 and 2.5 days, respectively.
Training details All the models (except for CPVT-B) are trained for 300 epochs with a global batch size of 2048 on Tesla V100 machines using AdamW optimizer (Loshchilov & Hutter, 2019).
We don’t tune the hyper-parameters and strictly comply with the settings in DeiT (Touvron et al., 2020).
The learning rate is scaled with this formula lr scale = 0 .
0005 ∗ BatchSize global.
The detailed hyperparameters are in the B.2.
4. G ENERALIZATION TO H IGHER R ESOLUTIONS
As mentioned before, our proposed PEG can directly generalize to larger image sizes without any ﬁne-tuning.
We conﬁrm this here by evaluating the models trained with 224 × 224 images on the 384 × 384 , 448 × 448 ,
× 512 images.
The results are shown in Table 2.
With the 384 ×input images, The DeiT-tiny with learnable positional encodings degrades from 72.2% to 71.2%.
When equipped with sine encoding, the tiny model degrades from 72.2% to 70.8%.
In constrat,
CPVT model with the proposed PEG can directly process the larger input images, and CPVT-Ti’s performance is boosted from 73.4% to 74.2% when applied to 384 × 384 images.
Our CPVT-Ti outperforms DeiT-tiny by 3.0%.
This gap continues to increase as the input resolution enlarges.
4. CPVT WITH G LOBAL A VERAGE P OOLING
By design, the proposed PEG is translation-equivariant (ignore paddings).
Thus, if we further use the translation-invariant global average pooling (GAP) instead of the cls token before the ﬁnal classiﬁcation layer of CPVT.
CPVT can be totally translation-invariant, which should be beneﬁcial to the ImageNet classiﬁcation task.
Note the using GAP here results in even less computation complexity because we do not need to compute the attention interaction between the class token and the [Table] image patches.
As shown in Table 3, using GAP here can boost CPVT by more than 1% .
For example, equipping CPVT-Ti with GAP obtains 74.9% top-1 accuracy on ImageNet validation dataset, which outperforms DeiT-tiny by a large margin (+2.7%).
Moreover, it even exceeds DeiT-tiny model with distillation (74.5%).
In contrast, DeiT with GAP cannot gain so much improvement (only 0.4% as shown in Table 3) because the original learnable absolute PE is not translation-equivariant and
GAP with the PE is not translation-invariant.
Given the superior performance, we hope our model can be a strong alternative in vision transformers.
[Table]
4. C OMPLEXITY OF PEG
Few Parameters.
Given the model dimension d , the extra number of parameters introduced by
PEG is d × l × k 2 if we choose l depth-wise convolutions with kernel k .
Even if we use l separable convolutions, this value becomes l ( d 2 + k 2 d ) .
When k = 3 and l = 1 , CPVT-Ti ( d = 192 ) brings about 1, 728 parameters.
Note that DeiT-tiny utilizes learnable position encodings with 192 × 14 × 14 = 37632 parameters.
Therefore, CPVT-Ti has 35, 904 fewer number of parameters than DeiTtiny.
Even using 4 layers of separable convolutions, CPVT-Ti introduces only 38952 − 37632 =
more parameters, which is negelectable compared to the 5.7M model parameters of DeiT-tiny.
FLOPs.
As for FLOPs, l layers of k × k depth-wise convolutions possesses 14 × 14 ×
d × l × k
FLOPS.
Taking the tiny model for example, it involves 196 × 192 × 9 = 0 .
34 M FLOPS for the simple case k
= 3 , l = 1 , which is neglectable because the model has 2.1G FLOPs in total.
4. Q UALITATIVE A NALYSIS OF CPVT
Thus far, we have shown that PEG can have better performance than the original positional encodings.
However, because PEG provides the position in an implicit way, it is interesting to see if PEG can indeed provide the position information as the original positional encodings.
Here we investigate this by visualizing the attention weights of the transformers.
Speciﬁcally, given a 224 × image (i.e. 14 × 14 patches), the score matrix within a single head is 196 × 196.
We visualize the normalized self-attention score matrix of the second encoder block.
We ﬁrst visualize the attention weights of DeiT with the original positional encodings.
As shown in
Figure 3 (middle), the diagonal element interacts strongly with its local neighbors but weakly with those far-away elements, which suggests that DeiT with the original positional encodings learn to attend the local neighbors of each patch.
After the positional encodings are removed (denoted by
DeiT w/o PE), all the patches produce similar attention weights and fail to attend to the patches near themselves, see Figure 3 (left).
[Figure]
Position encodings are key to developing a schema of locality in lower layers of DeiT.
Meantime,
CPVT proﬁts from conditional encodings and follows a similar locality pattern.
Finally, we show the attention weights of our CPVT model with PEG.
As shown in Figure 3 (right), like the original positional encodings, the model with PEG can also learn a similar attention pattern, which indicates that the proposed PEG can provide the position information as well.
4. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS
[Figure]
We evaluate the performance of CPVT models on the ImageNet validation dataset and report the results in Table 4.
Compared with DeiT, CPVT models have much better top1 accuracy with similar throughputs .
Our models can enjoy performance improvement when inputs are upscaled without ﬁne-tuning, while DeiT degrades as discussed in
Table 2, see also Figure 4 for a clear comparison.
Noticeably, Our model with GAP marks a new state-of-the-art for vision Transformers.
We further train CPVT-Ti and DeiT-tiny using the aforementioned training settings plus the hard distillation proposed in (Touvron et al., 2020).
Speciﬁcally, we use
RegNetY-160 (Radosavovic et al., 2020) as the teacher.
CPVT obtains 75.9%, exceeding DeiT-tiny by 1.4%.
4. PEG ON P YRAMID T RANSFORMER A RCHITECTURES
PVT (Wang et al., 2021) is a recently-proposed vision transformer with the multi-stage design like
ResNet (He et al., 2016).
Swin (Liu et al., 2021) is a follow-up work and comes with higher performance.
We apply our method on both to demonstrate its generalization ability.
ImageNet classiﬁcation.
Speciﬁcally, we remove its learnable PE and apply our PEG in position 0 of each stage with a GAP head.
We use the same training settings to make a fair comparison and show the results in Table 12.
Our method can signiﬁcantly boost PVT-tiny by 3.1% and Swin-tiny by 1.15% on ImageNet (c.f. B.5).
We also evaluate the performance of PEG on some downstream semantic segmentation and object detection tasks (see B.6).
Note these tasks usually handle the various input resolutions as the training because multi-scale data augmentation is extensively used.
[Table]
A BLATION S TUDY 5. P OSITIONAL ENCODING OR MERELY A HYBRID ?
One might suspect that the PEG’s improvement comes from the extra learnable parameters introduced by the convolutional layers in PEG, instead of the local relationship retained by PEG.
One way to test the function of PEG is only adding it when calculating Q and K in the attention layer, so that only the positional information of PEG is passed through.
We can achieve 71.3% top-1 accuracy on ImageNet with DeiT-tiny.
This is signiﬁcantly better than DeiT-tiny w/o PE (68.2%) and is similar to the one with PEG on Q, K and V (72.4%), which suggests that PEG mainly serves as a positional encoding scheme.
[Table]
We also design another experiment to remove this concern.
By randomly-initializing a 3 × 3 PEG and ﬁxing its weights during the training, we can obtain 71.3% accuracy (Table 5), which is much higher (3.1% ↑ ) than DeiT without any PE (68.2%).
Since the weights of PEG are ﬁxed and the performance improvement can only be due to the introduced position information.
On the contrary, when we exhaustively use 12 convolutional layers (kernel size being 1, i.e ., not producing local relationship) to replace the PEG, these layers have much more learnable parameters than PEG.
However, it only boosts the performance by 0.4% to 68.6%.
Another interesting ﬁnding is that ﬁxing a learned PEG also helps training.
When we initialize with a learned PEG instead of the random values and train the tiny version of the model from scratch while keeping the PEG ﬁxed, the model can also achieve 72.3% top-1 accuracy on ImageNet.
This is very close to the learnable PEG (72.4%).
5. PEG P OSITION IN CPVT
We also experiment by varying the position of the PEG in the model.
Table 6 left presents the ablations for variable positions based on the tiny model.
We denote the input of the ﬁrst encoder by index -1.
Therefore, position 0 is the output of the ﬁrst encoder block.
PEG shows strong performance ( ∼ 72.4%) when it is placed at [0, 3].
Note that positioning the PEG at 0 can have much better performance than positioning it at -
( i
[Table]
e ., before the ﬁrst encoder), as shown in Table 6 left.
We observe that the difference between the two situations is they have different receptive ﬁelds.
Speciﬁcally, the former has a global ﬁeld while the latter can only see a local area.
Hence, they are supposed to work similarly well if we enlarge the convolution’s kernel size .
To verify our hypothesis, we use a quite large kernel size 27 with a padding size 13 at position -1, whose result is reported in Table 6 right.
It achieves similar performance to the one positioning the PEG at 0 (72.5%), which veriﬁes our assumption.
[Table]
5. C OMPARISONS WITH OTHER POSITIONAL ENCODINGS
We compare PEG with other commonly used encodings: absolute positional encoding (e.g. sinusoidal (Vaswani et al., 2017)), relative positional encoding (RPE) (Shaw et al., 2018) and learnable encoding (LE) (Devlin et al., 2019; Radford et al., 2018), as shown in Table 7.
[Table]
DeiT-tiny obtains 72.2% with learnable absolute PE.
We experiment with the 2-D sinusoidal encodings to achieve on-par performance.
For
RPE, we follow (Shaw et al., 2018) and set the local range hyper-parameter K as 8, with which we obtain 70.5%.
RPE here does not encode any absolute position information, see discussion in D.1 and B.3.
Moreover, we combine the learnable absolute
PE with a single-layer PEG.
This boosts the baseline CPVT-Ti (0-1) by 0.5.
If we use 4layer PEG, it can achieve 72.9%.
Moreover, if we add a single layer PEG to the ﬁrst ﬁve blocks, we can obtain 73.4%, which indicates that it is better to add the positional encodings to different blocks than stacking them at one.
C ONCLUSION
We introduced CPVT, a novel method to provide the position information in vision Transformers, which dynamically generates the position encodings based on the local neighbors of each input token.
Through extensive experimental studies, we demonstrate that our proposed positional encodings can achieve stronger performance than the previous positional encodings.
The transformer models with our positional encodings can naturally process longer input sequences and keep the desired translation equivalence in vision tasks.
Moreover, our positional encodings are easy to implement and come with negligible cost.
We look forward to a broader application of our method in transformer-driven vision tasks like segmentation and video processing.
A P RELIMINARY K NOWLEDGE
Transformer (Vaswani et al., 2017) features a series of encoders and decoders of an identical structure.
Every encoder and decoder has a multi-head self-attention layer (MHSA) and a feed-forward network layer (FFN), while each decoder has an extra attention layer to process the output of the encoder.
Positional encodings instills position information for a sequence of input tokens.
A. T RANSFORMER C OMPONENTS
Self-attention.
An attention function on input x = { x 1 , . . .
, x n } is computed simultaneously on a set of queries Q with keys K and values V by the following,
[Equation]
where W Q , W K , and W V are weight matrices to generate Q , K , and V via linear transformations on x .
And Q · K (cid:62) calculates attention score as the dot product of the query and all the keys, scaled by the dimension
d k of keys K .
Multi-head Self-Attention.
First, Q , K , and V are linearly projected for h times with different learned weights.
Then the self-attention function is applied in parallel to generate h outputs, socalled heads .
All heads are concatenated to give the ﬁnal output, i.e .,
MultiHead( Q, K,
[Equation]
) =
Concat
[Equation] [Equation]
where
[Equation]
Feed-forward network.
The attention output is typically processed by a two-layer linear transformation with an activation in between,
[Equation]
Layer Normalization.
A residual function and a layer normalization is added for each of the sub-layers ( e.g ., attention layer and feed-forward layer) in every encoder and decoder, i.e .,
LayerNorm( x + Sublayer( x )) .
A. P OSITIONAL E NCODINGS
Positional Encoding.
Transformer (Vaswani et al., 2017) adopts a sinusoidal function to encode positions, e.g.
[Equation]
where pos denotes the word position in the sequence, d model means the total encoding dimension, and i is the current dimension.
However, the position can also be encoded as relative positional encodings (RPE) (Shaw et al., 2018).
Formally,
[Equation]
where a ij ∈ R d k denotes edge distances between x i and x j when seeing input elements as a directed and fully-connected graph.
RPE creates practical space complexity for implementation, which require more efﬁcient version as in (Huang et al., 2018).
Moreover, it also requires substantial intrusion into standard Transformer APIs.
A. M ORE D ISCUSSIONS ABOUT T RANSLATION E QUIVARIANCE
The term translation-equivariance means the output feature maps can be equally translated with the input signal.
Imagine there is a person in the left-top of an image, if the person is moved to the right-bottom, the output feature maps will change accordingly.
This property is very important to the success of convolution network.
Convolution (ignoring paddings), RPE, and self-attention are all translation-equivariant operations (regardless of their receptive ﬁeld).
It’s nontrivial to make absolute positional encodings like DeiT (using learnable positional encoding) translation-equivariant since different absolute positions will be added if the input signal is translated.
Note that our method is not strictly translation-equivariant because of the zero padding.
Instead, it provides a kind of stronger explicit bias towards the translation-equivariant property.
B E XPERIMENT D ETAILS B. A RCHITECTURE VARIANTS OF CPVT
[Table]
T HE H YPERPARAMETERS OF CPVT
As for the ImageNet classiﬁcation task, we use exactly the same hyperparameters as DeiT, as detailed in Table 9.
[Table]
[Figure] [Table] [Figure]
[Table]
B. I MPORTANCE OF Z ERO P ADDINGS
We design an experiment to verify the importance of the zero paddings [Table] which can help the model infer the absolute positional information.
Speciﬁcally, we use CPVT-S and simply remove the zero paddings from CPVT while keeping all other settings unchanged.
Table 10 shows that this can only obtain 70.5%
[Table]
which indicates that the zero paddings and absolute positional information play important roles in classifying objects
[Table]
[Table]
B. S INGLE PEG VS . M ULTIPLE PEG S
We further evaluate whether or not using multi-position encodings can beneﬁt the performance in
Table 11.
Notice we denote by i - j the inserted positions of PEG which start from the i -th encoder and end at the j − 1 -th one (inclusion).
By inserting PEGs to ﬁve positions, the top-1 accuracy of the tiny model can achieve 73.4%, which surpasses DeiT-tiny by 1.2%.
Similarly, CPVT-S can achieve 80.5%.
It turns out more PEGs do help, but up to a level where more PEGs become incremental (0vs. 0-11).
[Table] [Table]
C LASSFICATION E VALUATION OF S WIN WITH PEG
We show the validation curves when training Swin (Liu et al., 2021) equipped with PEG in Figure
[Table]
It can boost Swin-tiny from 81.10% to 82.25% (+1.15% ↑ ) on ImageNet.
[Figure]
B. E VALUATION ON S EGMENTATION AND D ETECTION
Semantic segmentation on ADE20K.
We evaluate the performance of PEG on the ADE20K (Zhou et al., 2017) segmentation task.
Based on the Semantic FPN framework (Kirillov et al., 2019), PVT achieves much better results than ResNet (He et al., 2016) baselines.
Under carefully controlled settings, PEG further boosts PVT-tiny by 3.1% mIoU.
Object detection on COCO.
We also perform controlled experiments with the RetinaNet (Lin et al., 2017) framework on the COCO detection task.
The results are shown in Table 12.
In the standard 1 × schedule, PEG improves PVT-tiny by 2.0% mAP.
PEG brings 2.4% higher mAP under the 3 × schedule.
[Table]
B. A BLATION ON O THER F ORMS OF PEG
We explore several forms of PEG based on the tiny model, which change the type of convolution, kernel size and layers.
The inserted position is 0.
The result is shown in Table 13.
When we use large kernel of 7 × 7 or dense convolution, the performance improvement is limited.
Stacking more layers of depth-wise convolution doesn’t bring signiﬁcant improvement.
Therefore, we use the simplest form as our default implementation.
It indicates that this design is enough to provide good position information.
[Table]
C E XAMPLE C ODE C. PEG
In the simplest form, we use a single depth-wise convolution and show its usage in Transformer by the following PyTorch snippet.
Through experiments, we ﬁnd that such a simple design ( i.e ., depthwise 3 × 3) readily achieves on par or even better performance than the recent SOTAs.
We give the torch implementation example in Alg. 1.
D M ORE DISCUSSIONS D. W HY RPE WORKS LESS WELL THAN ABSOLUTE PE?
As mentioned in Section 5.3 (main text), RPE is inferior to the absolute positional encoding.
It is because RPE does not encode any absolute position information.
Also discussed in Section B.
(main text), absolute position information is also important even for ImageNet classiﬁcation as it is needed to determine which object is at the center of the image.
Note that there might be multiple objects in an image, and the label of an image is the category of the object at the center.
Additionally, although RPE becomes popular recently, it is often jointly used with absolute positional encodings (e.g., in ConViT (d’Ascoli et al., 2021)), or the absolute position information is leaked in other ways (e.g., convolution paddings in CoAtNet (Dai et al., 2021)).
This further suggests absolute position information is crucial.
[Table]
D. C OMPARISON TO L AMBDA N ETWORKS
Our work is also related to Lambda Networks (Bello, 2021) which uses 2D relative positional encodings.
We evaluate its lambda module with an embedding size of 128, where we denote its encoding scheme as RPE2D-d128.
Noticeably, this conﬁguration has about 5.9M parameters (comparable to
DeiT-tiny) but only obtains 68.7%.
We attribute its failure to the limited ability in capturing the correct positional information.
After all, lambda layers are designed with the help of many CNN backbones components such as down-sampling to form various stages, to replace ordinary convolutions in ResNet (He et al., 2016).
In contrast, CPVT is transformer-based.
E E XTRA F IGURES E. F IGURES OF A TTENTION S CORES
We illustrate the attention scores in several encoder blocks of DeiT (Touvron et al., 2020) and CPVT in the Fig.
It shows both methods learn similar locality patterns. As attention scores are computed over the tokens projected in different subspaces (Q and K), they do not necessarily show a strict diagonal pattern, where some may have slight shift, see DeiT in Fig. 6c and CPVT of Fig. 3 right.
E. C OMPARISON WITH O THER A PPROACHES
We further compare our method with other approaches such as CvT (Wu et al., 2021), ConViT (d’Ascoli et al., 2021) and CoAtNet (Dai et al., 2021) on ImageNet validation set in Table 14.
To make fair comparisons, we categorize these methods into two groups: plain and pyramid models.
Since our models are primarily for plain models, we adapt our methods on two popular pyramid frameworks PVT and Swin.
Our CPVT-S-GAP slightly outperforms ConViT-S by 0.2% with 4M fewer parameters and 0.8G fewer FLOPs.
When equipped with pyramid designs, our methods are still comparable to CvT and CoAtNet.
Comparison with DeiT w/ Convolutional Projection.
Note CvT uses a depth-wise convolution in q - k - v projection which they call it Convolutional Projection .
Instead of using it in all layers, we put only one of such design into DeiT-tiny and train such a model from scratch under strictly controlled settings.
We insert it in the position 0 as in our method.
The result is shown in Table 15.
This [Figure]
and CPVT on the same input sequence.
DeiT and CPVT share similar locality patterns that are aligned diagonally (some might shift).
[Table]
CvT-ﬂavored DeiT achieves 70.6% top-1 accuracy on ImageNet validation set, which is lower than ours (72.4%).
Note that q - k - v projections in CvT utilize three depthwise convolutions, therefore, this setting has more parameters than ours.
This attests the difference of CvT and CPVT, verifying our advantage by learning better position encodings other than inserting them in all layers to have the ability to capture local context and to remove ambiguity in attention.
[Table]