C ONDITIONAL P OSITIONAL E NCODINGS FOR V ISION T RANSFORMERS
A BSTRACT
We propose a conditional positional encoding
CPE) scheme for vision Trans formers (Dosovitskiy et al., Touvron et al., 2020).
Unlike previous ﬁxed or learnable positional encodings that are predeﬁned and independent of input to- kens , CPE is dynamically generated and conditioned on the local neighborhood of the input tokens.
As a result , CPE can easily generalize to the input sequences that are longer than what the model has ever seen during the training.
Besides, CPE can keep the desired translation equivalence in vision tasks, resulting in improved performance.
We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework.
Built on
PEG , we present Conditional Position encoding Vision Transformer (CPVT).
We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our Code is available at: https git. io/CPVT .
I NTRODUCTION
Recently, Transformers (Vaswani et al., 2017) have been viewed as a strong alternative to Convolutional Neural Networks (CNNs) in visual recognition tasks such as classiﬁcation (Dosovitskiy et al., 2021) and detection (Carion et al., 2020; Zhu et al., 2021).
Unlike the convolution operation in
CNNs, which has a limited receptive ﬁeld, the self-attention mechanism in the Transformers can capture the long-distance information and dynamically adapt the receptive ﬁeld according to the image content.
Consequently, Transformers are considered more ﬂexible and powerful than CNNs, being promising to achieve more progress in visual recognition.
However, the self-attention operation in Transformers is permutation-invariant, which discards the order of the tokens in an input sequence.
To mitigate this issue, previous works (Vaswani et al., 2017; Dosovitskiy et al., 2021) add the absolute positional encodings to each input token (see Figure 1a), which enables order-awareness.
The positional encoding can either be learnable or ﬁxed with sinusoidal functions of different frequencies.
Despite being effective, these positional encodings seriously harm the ﬂexibility of the Transformers, hampering their broader applications.
Taking the learnable version as an example, the encodings are often a vector of equal length to the input sequence, which are jointly updated with the network weights during training.
As a result, the length and the value of the positional encodings are ﬁxed once trained. During testing, it causes difﬁculties of handling the sequences longer than the ones in the training data. The inability to adapt to longer input sequences during testing greatly limits the range of generalization. For instance, in vision tasks like object detection, we expect the model can be applied to the images of any size during inference, which might be much larger than the training images.
A possible remedy is to use bicubic interpolation to upsample the positional encodings to the target length, but it degrades the performance without ﬁne-tuning as later shown in our experiments.
For vision in general, we expect that the models be translation-equivariant. For example, the output feature maps of CNNs shift accordingly as the target objects are moved in the input images. However, the absolute positional encoding scheme might break the translation equivalence because it adds unique Embedding cls
PE
L × Encoders
MLP
Encoder (L-1) ×
Encoders
Embedding cls
Encoder
PEG (L-1) × Encoders
Embedding
GAP
PEG
MLP MLP
[Figure]
Embedding cls
PE
L × Encoders
MLP
Encoder (L-1) ×
Encoders
Embedding cls
Encoder
PEG (L-1) × Encoders
Embedding
GAP
PEG
MLP MLP
[Figure]
Embedding cls
PE
L × Encoders
MLP
Encoder (L-1) × Encoders
Embedding cls
Encoder
PEG (L-1) × Encoders
Embedding
GAP
PEG
MLP
MLP
[Figure] positional encodings to each token (or each image patch).
One may overcome the issue with relative positional encodings as in (Shaw et al., 2018).
However, relative positional encodings not only come with extra computational costs, but also require modifying the implementation of the standard
Transformers.
Last but not least, the relative positional encodings cannot work equally well as the absolute ones, because the image recognition task still requires absolute position information (Islam et al., 2020), which the relative positional encodings fail to provide.
In this work, we advocate a novel positional encoding (PE) scheme to incorporate the position information into Transformers.
Unlike the predeﬁned and input-agnostic positional encodings used in previous works (Dosovitskiy et al., 2021; Vaswani et al., 2017; Shaw et al., 2018),
the proposed
PE is dynamically generated and conditioned on the local neighborhood of input tokens.
Thus, our positional encodings can change along with the input size and try to keep translation equivalence.
We demonstrate that the vision transformers (Dosovitskiy et al., 2021; Touvron et al., 2020) with our new
PE ( i.e . CPVT, see Figure 1c) achieve even better performance.
We summarize our contributions as,
• We propose a novel positional encoding (PE) scheme, termed conditional position encodings (CPE).
CPE is dynamically generated with Positional Encoding Generators (PEG) and can be effortlessly implemented by the modern deep learning frameworks (Paszke et al., 2019;
Abadi et al., 2016; Chen et al., 2015), requiring no changes to the current Transformer APIs.
Through an in-depth analysis and thorough experimentations, we unveil that this design affords both absolute and relative encoding yet it goes above and beyond.
• As opposed to widely-used absolute positional encodings, CPE can provide a kind of stronger explicit bias towards the translation equivalence which is important to improve the performance of Transformers.
• Built on CPE, we propose Conditional Position encoding Vision Transformer (CPVT).
It achieves better performance than previous vison transformers (Dosovitskiy et al., 2021; Touvron et al., 2020).
• CPE can well generalize to arbitrary input resolutions, which are required in many important downstream tasks such as segmentation and detection.
Through experiments we show that CPE can boost the segmentation and detection performance for pyramid transformers like (Wang et al., 2021) by a clear margin.
R ELATED W ORK
Since self-attention itself is permutation-equivariant (see A), positional encodings are commonly employed to incorporate the order of sequences (Vaswani et al., 2017).
The positional encodings can either be ﬁxed or learnable, while either being absolute or relative.
Vision transformers follow the same fashion to imbue the network with positional information.
Absolute Positional Encoding.
The absolute positional encoding is the most widely used.
In the original transformer (Vaswani et al., 2017), the encodings are generated with the sinusoidal functions of different frequencies and then they are added to the inputs.
Alternatively, the positional encodings can be learnable, where they are implemented with a ﬁxed-dimension matrix/tensor and jointly updated with the model’s parameters with SGD.
Relative Positional Encoding.
The relative position encoding (Shaw et al., 2018) considers distances between the tokens in the input sequence.
Compared to the absolute ones, the relative positional encodings can be translation-equivariant and can naturally handle the sequences longer than the longest sequences during training ( i.e ., being inductive).
A 2-D relative position encoding is proposed for image classiﬁcation in (Bello et al., 2019), showing superiority to 2D sinusoidal embeddings.
The relative positional encoding is further improved in XLNet (Yang et al., 2019b) and
DeBERTa (He et al., 2020), showing better performance.
Other forms.
Complex-value embeddings (Wang et al., 2019) are an extension to model global absolute encodings and show improvement.
RoFormer (Su et al., 2021) utilizes a rotary position embedding to encode both absolute and relative position information for text classiﬁcation.
FLOATER (Liu et al., 2020) proposes a novel continuous dynamical model to capture position encodings.
It is not limited by the maximum sequence length during training, meanwhile being parameter-efﬁcient.
Similar designs to CPE.
Convolutions are used to model local relations in ASR and machine translation (Gulati et al., 2020; Mohamed et al., 2019;
Yang et al., 2019a; Yu et al., 2018).
However, they are mainly limited to 1D signals.
We instead process 2D vision images.
V ISION T RANSFORMER WITH C ONDITIONAL P OSITION E NCODINGS 3. M OTIVATION
In vision transformers, an input image of size H × W is split into patches with size S × S , the number of patches is N = HWS 2 1 .
The patches are added with the same number of learnable absolute positional encoding vectors.
In this work, we argue that the positional encodings used here have two issues.
First, it prevents the model from handling the sequences longer than the learnable PE.
Second, it makes the model not translation-equivariant because a unique positional encoding vector is added to every one patch.
The translation equivalence plays an important role in classiﬁcation because we hope the networks’ responses changes accordingly as the object moves in the image.
One may note that the ﬁrst issue can be remedied by removing the positional encodings since except for the positional encodings, all other components ( e.g ., MHSA and FFN) of the vision transformer can directly be applied to longer sequences.
However, this solution severely deteriorates the performance.
This is understandable because the order of the input sequence is an important clue and the model has no way to extract the order without the positional encodings.
The experiment results on
ImageNet are shown in Table 1.
By removing the positional encodings, DeiT-tiny’s performance on
ImageNet dramatically degrades from 72.2% to 68.2%.
Second, in DeiT (Touvron et al., 2020), they show that we can interpolate the position encodings to make them have the same length of the longer sequences.
However, this method requires ﬁnetuning the model a few more epochs, otherwise the performance will remarkably drop, as shown in Table 1.
This goes contrary to what we would expect.
With the higher-resolution inputs, we often expect a remarkable performance improvement without any ﬁne-tuning.
Finally, the relative position encodings (Shaw et al., 2018; Bello et al [Table] issues.
However, the relative positional encoding cannot provide absolute position information , which is also important to the classiﬁcation performance (Islam et al., 2020).
As shown in Table 1, the model with relative position encodings has inferior performance (70.5% vs. 72.2%).
3. C ONDITIONAL P OSITIONAL E NCODINGS
We argue that a successful positional encoding for vision tasks should meet these requirements,
(1) Making the input sequence permutation-variant and providing stronger explicit bias towards translation-equivariance .
(2) Being inductive and able to handle the sequences longer than the ones during training.
(3) Having the ability to provide the absolute position to a certain degree.
This is important to the performance as shown in (Islam et al., 2020).
In this work [Table] we ﬁnd that characterizing the local relationship by positional encodings is sufﬁcient to meet all of the above.
First, it is permutation-variant because the permutation of input sequences also affects the order in some local neighborhoods.
However, translation of an object in an input image does not change the order in its local neighborhood, i.e ., translation-equivariant (see Section A).
Second, the model can easily generalize to longer sequences since only the local neighborhoods of a token are involved.
Besides, if the absolute position of any input token is known, the absolute position of all the other tokens can be inferred by the mutual relation between input tokens.
We will show that the tokens on the borders can be aware of their absolute positions due to the commonly-used zero paddings.
[Figure]
position encodings reshape
F reshape
PEG (PEG). Note d is the embedding size, N is the number of tokens. Therefore, we propose positional encoding generators (PEG) to dynamically produce the positional encodings conditioned on the local neighborhood of an input token.
Positional Encoding Generator.
PEG is illustrated in Figure 2.
To condition on the local neighbors, we ﬁrst reshape the ﬂattened input sequence X ∈ R B × N × C of DeiT back to X (cid:48) ∈ R B
× H × W × C in the 2-D image space.
Then, a function (denoted by F in Figure 2) is repeatedly applied to the local patch in X (cid:48) to produce the conditional positional encodings E B × H × W × C .
PEG can be efﬁciently implemented with a 2-D convolution with kernel k ( k ≥ 3 ) and k − 12 zero paddings.
Note that the zero paddings here are important to make the model be aware of the absolute positions, and F can be of various forms such as various types of convolutions and many others.
3. C ONDITIONAL P OSITIONAL E NCODING V ISION T RANSFORMERS
Built on the conditional positional encodings, we propose our Conditional Positional Encoding Vision Transformers (CPVT).
Except that our positional encodings are conditional, we exactly follow
ViT and DeiT to design our vision transformers and we also have three sizes CPVT-Ti, CPVT-S and
CPVT-B. Similar to the original positional encodings in DeiT, the conditional positional encodings are also added to the input sequence, as shown in Figure 1 (b).
In CPVT, the position where PEG is applied is also important to the performance, which will be studied in the experiments.
In addition, both DeiT and ViT utilize an extra learnable class token to perform classiﬁcation ( i.e ., cls token shown in Figure 1 (a) and (b)).
By design, the class token is not translation-invariant, although it can learn to be so.
A simple alternative is to directly replace it with a global average pooling (GAP), which is inherently translation-invariant, resulting in our CVPT-GAP.
Together with CPE, CVPT-GAP achieves much better image classiﬁcation performance.
E XPERIMENTS 4. S ETUP
Datasets.
Following DeiT (Touvron et al., 2020), we use ILSVRC-2012 ImageNet dataset (Deng et al., 2009) with 1K classes and 1.3M images to train all our models.
We report the results on the validation set with 50K images.
Unlike ViT (Dosovitskiy et al., 2021), we do not use the much larger undisclosed
JFT-300M dataset (Sun et al., 2017).
Model variants.
We have three models with various sizes to adapt to various computing scenarios.
The detailed settings are shown in Table 9 (see B.1).
All experiments in this paper are performed on
Tesla V100 machines.
Training the tiny model for 300 epochs takes about 1.3 days on a single node with 8 V100 GPU cards.
CPVT-S and CPVT-B take about 1.6 and 2.5 days, respectively.
Training details All the models (except for CPVT-B) are trained for 300 epochs with a global batch size of 2048 on Tesla V100 machines using AdamW optimizer (Loshchilov & Hutter, 2019).
We do not tune the hyper-parameters and strictly comply with the settings in DeiT (Touvron et al., 2020).
The learning rate is scaled with this formula lr scale = 0 . 0005 ·
BatchSize global / 512 .
The detailed hyperparameters are in the B.2.
4. G ENERALIZATION TO H IGHER R ESOLUTIONS
As mentioned before, our proposed PEG can directly generalize to larger image sizes without any ﬁne-tuning.
We conﬁrm this here by evaluating the models trained with 224 × 224 images on the 384 × 384 , 448 × 448 ,
× 512 images, respectively.
The results are shown in Table 2.
With the 384 × 384 input images, the DeiT-tiny with learnable positional encodings degrades from 72.2% to 71.2%.
When equipped with sine encoding, the tiny model degrades from 72.2% to 70.8%.
In constrat, our CPVT model with the proposed PEG can directly process the larger input images, and CPVT-Ti’s performance is boosted from 73.4% to 74.2% when applied to 384 × 384 images.
Our CPVT-Ti outperforms DeiT-tiny by 3.0%.
This gap continues to increase as the input resolution enlarges.
[Table]
4. CPVT WITH G LOBAL A VERAGE P OOLING
By design, the proposed PEG is translation-equivariant (ignore paddings).
Thus, if we further use the translation-invariant global average pooling (GAP) instead of the cls token before the ﬁnal classiﬁcation layer of CPVT.
CPVT can be translation-invariant, which should be beneﬁcial to the ImageNet classiﬁcation task.
Note the using GAP here results in even less computation complexity because we do not need to compute the attention interaction between the class token and the image patches.
As shown in Table 3, using GAP here can boost CPVT by more than 1% .
For example, equipping CPVT-Ti with GAP obtains 74.9% top-1 accuracy on the ImageNet validation dataset, which outperforms DeiT-tiny by a large margin (+2.7%).
Moreover, it even exceeds DeiT-tiny model with distillation (74.5%).
In contrast, DeiT with GAP cannot gain so much improvement (only 0.4% as shown in Table 3) because the original learnable absolute PE is not translation-equivariant and thus GAP with the PE is not translation-invariant.
Given the superior performance, we hope our model can be a strong PE alternative in vision transformers. [Table] CPVT-S ‡ [Table]
23M
[Table]
CPVT-S ‡
{+GAP 
+}
M
[Table]
4. C OMPLEXITY OF PEG
Few Parameters.
Given the model dimension d , the extra number of parameters introduced by PEG is d × l × k 2 if we choose l depth-wise convolutions with kernel k .
If we use l separable convolutions, this value becomes l ( d 2 + k 2 d ) .
When k = 3 and l = 1 , CPVT-Ti ( d = 192 ) brings about 1,parameters.
Note that DeiT-tiny utilizes learnable position encodings with 192 × 14 × 14 =parameters.
Therefore, CPVT-Ti has 35, 904 fewer number of parameters than DeiT-tiny.
Even using 4 layers of separable convolutions, CPVT-Ti introduces only 38952 − 37632 = 960 more parameters, which is negelectable compared to the 5.7M model parameters of DeiT-tiny.
FLOPs.
As for FLOPs, l layers of k × k depth-wise convolutions possesses 14 × 14 ×
d × l × k
FLOPS.
Taking the tiny model for example, it involves 196 × 192 × 9 = 0 .
34 M FLOPS for the simple case k
= 3 and l = 1 , which is neglectable because the model has 2.1G FLOPs in total.
4. P ERFORMANCE
[Figure]
under various conﬁgurations.
Note CPVT@ has improved performance.
More PEGs can result in better performance. CPVT-GAP is the best.
We evaluate the performance of CPVT models on the ImageNet validation dataset and report the results in Table 4.
Compared with DeiT, CPVT models have much better top-1 accuracy with similar throughputs .
Our models can enjoy performance improvement when inputs are upscaled without ﬁne-tuning, while DeiT degrades as discussed in Table 2, see also Figure 3 for a clear comparison.
Noticeably, Our model with GAP marked a new state-of-the-art for vision Transformers. [Table] [Table] [Table]
DeiT-small (Touvron [Table] [Table] DeiT-
base
[Table]
Touvron et [Table] [Table] We further train CPVT-Ti and DeiT-tiny using the aforementioned training settings plus the hard distillation proposed in (Touvron et al., 2020).
Speciﬁcally, we use RegNetY-160 (Radosavovic et al., 2020) as the teacher.
CPVT obtains 75.9%, exceeding DeiT-tiny by 1.4%.
4. PEG ON P YRAMID T RANSFORMER A RCHITECTURES
PVT (Wang et al., 2021) is a vision transformer with the multi-stage design like ResNet (He et al., 2016).
Swin (Liu et al., 2021) is a follow-up work and comes with higher performance.
We apply our method on both to demonstrate its generalization ability.
ImageNet classiﬁcation.
Speciﬁcally, we remove its learnable PE and apply our PEG in position 0 of each stage with a GAP head.
We use the same training settings to make a fair comparison and show the results in Table 13.
Our method can signiﬁcantly boost PVT-tiny by 3.1% and Swin-tiny by 1.15% on ImageNet (c.f. B.5).
We also evaluate the performance of PEG on some downstream semantic segmentation and object detection tasks (see B.6).
Note these tasks usually handle the various input resolutions as the training because multi-scale data augmentation is extensively used.
A BLATION S TUDY 5. P OSITIONAL ENCODING OR MERELY A HYBRID ?
One might suspect that the PEG’s improvement comes from the extra learnable parameters introduced by the convolutional layers in PEG, instead of the local relationship retained by PEG.
One way to test the function of PEG is only adding it when calculating Q and K in the attention layer, so that only the positional information of PEG is passed through.
We can achieve 71.3% top-1 accuracy on ImageNet with DeiT-tiny.
This is signiﬁcantly better than DeiT-tiny w/o PE (68.2%) and is similar to the one with PEG on Q, K and V (72.4%), which suggests that PEG mainly serves as a positional encoding scheme.
[Table]
We also design another experiment to remove this concern.
By randomly-initializing a 3 × PEG and ﬁxing its weights during the training, we can obtain 71.3% accuracy (Table 5), which is much higher (3.1% ↑ ) than DeiT without any PE (68.2%).
Since the weights of PEG are ﬁxed and the performance improvement can only be due to the introduced position information.
On the contrary, when we exhaustively use 12 convolutional layers (kernel size being 1, i.e ., not producing local relationship) to replace the PEG, these layers have much more learnable parameters than PEG.
However, it only boosts the performance by 0.4% to 68.6%.
Another interesting ﬁnding is that ﬁxing a learned PEG also helps training.
When we initialize with a learned PEG instead of the random values and train the tiny version of the model from scratch while keeping the PEG ﬁxed, the model can also achieve 72.3% top-1 accuracy on ImageNet.
This is very close to the learnable PEG (72.4%).
5. PEG P OSITION IN CPVT
We also experiment by varying the position of the PEG in the model.
Table 6 (left) presents the ablations for variable positions (denoted as PosIdx) based on the tiny model.
We consider the input of the ﬁrst encoder by index -1.
Therefore, position 0 is the output of the ﬁrst encoder block.
PEG shows strong performance ( ∼ 72.4%) when it is placed at [0, 3].
Note that positioning the PEG at 0 can have much better performance than positioning it at -
( i.e ., before the ﬁrst encoder), as shown in Table 6 (left).
We observe that the difference between the two situations is they have different receptive ﬁelds.
Speciﬁcally, the former has a global ﬁeld while the latter can only see a local area.
Hence, they are supposed to work similarly well if we enlarge the convolution’s kernel size .
To verify our hypothesis, we use a quite large kernel size 27 with a padding size 13 at position -1, whose result is reported in Table 6 (right).
It achieves similar performance to the one positioning the PEG at 0 (72.5%), which veriﬁes our assumption.
[Table]
C OMPARISONS WITH OTHER POSITIONAL ENCODINGS
We compare PEG with other commonly used encodings: absolute positional encoding (e.g. sinusoidal (Vaswani et al., 2017)), relative positional encoding (RPE) (Shaw et al., 2018) and learnable encoding (LE) (Devlin et al., 2019; Radford et al., 2018), as shown in Table 7.
DeiT-tiny obtains 72.2% with the learnable absolute PE.
We experiment with the 2-D sinusoidal encodings and it achieves on-par performance.
For RPE, we follow (Shaw et al., 2018) and set the [Table] local range hyper-parameter K as 8, with which we obtain 70.5%.
RPE here does not encode any absolute position information, see discussion in D.1 and B.3.
Moreover, we combine the learnable absolute PE with a single-layer PEG.
This boosts the baseline CPVT-Ti (0-1) by 0.5%.
If we use 4-layer PEG, it can achieve 72.9%.
If we add a PEG to each of the ﬁrst ﬁve blocks, we can obtain 73.4%, which is better than stacking them within one block. CPE is not a simple combination of APE and RPE.
We further compare our method with a baseline with combination of APE and RPE.
Speciﬁcally, we use learnable positional encoding (LE) as DeiT at the beginning of the model and supply 2D RPE for every transformer block. This setting achieves 72.4% top-1 accuracy on ImageNet, which is comparable to a single PEG (72.4%). Nevertheless, this experiment does not necessarily indicate that our CPE is a simple combination of APE and RPE. When tested on different resolutions, this baseline cannot scale well compared to ours (Table 8).
RPE is not able to adequately mitigate the performance degradation on top of LE. This shall be seen as a major difference. [Table] PEG can continuously improve the performance if stacked more.
We use LE not only at the beginning but also in the next 5 layers to have a similar thing as 0-5 PEG conﬁguration.
This setting achieves 72.7% top-1 accuracy on ImageNet, which is 0.7% lower than PEG (0-5).
This setting suggests that it is also beneﬁcial to have more of LEs, but not as good as ours.
It is expected since we exploit relative information via PEGs at the same time.
C ONCLUSION
We introduced CPVT, a novel method to provide the position information in vision transformers, which dynamically generates the position encodings based on the local neighbors of each input token.
Through extensive experimental studies, we demonstrate that our proposed positional encodings can achieve stronger performance than the previous positional encodings.
The transformer models with our positional encodings can naturally process longer input sequences and keep the desired translation equivalence in vision tasks.
Moreover, our positional encodings are easy to implement and come with negligible cost.
We look forward to a broader application of our method in transformer-driven vision tasks like segmentation and video processing.
A T RANSLATION E QUIVARIANCE
The term translation-equivariance means the output feature maps can be equally translated with the input signal.
Imagine there is a person in the left-top of an image, if the person is moved to the right-bottom, the output feature maps will change accordingly.
This property is very important to the success of convolution network.
Convolution (ignoring paddings), RPE, and self-attention are all translation-equivariant operations (regardless of their receptive ﬁeld).
It’s nontrivial to make absolute positional encodings like DeiT (using learnable positional encoding) translation-equivariant since different absolute positions will be added if the input signal is translated.
Note that our method is not strictly translation-equivariant because of the zero padding.
Instead, it provides a kind of stronger explicit bias towards the translation-equivariant property.
B E XPERIMENT D ETAILS B. A RCHITECTURE VARIANTS OF CPVT
[Table]
T HE H YPERPARAMETERS OF CPVT
As for the ImageNet classiﬁcation task, we use exactly the same hyperparameters as DeiT except for the base model because it is not always stably trained using AdamW. The detailed setting is shown in Table 10.
[Table]
[Figure]
[Table]
[Figure]
[Table]
B. I MPORTANCE OF Z ERO P ADDINGS
We design an experiment to verify the importance of the zero paddings , which can help the model infer the absolute positional information.
Speciﬁcally, we use CPVT-S and simply remove the zero paddings from CPVT while keeping all other settings unchanged.
Table 11 shows that this can only obtain 70.5%
[Table]
which indicates that the zero paddings and absolute positional information play important roles in classifying objects
[Table]
[Table]
S INGLE PEG VS . M ULTIPLE PEG S
We further evaluate whether or not using multi-position encodings can beneﬁt the performance in
Table 12.
Notice we denote by i - j the inserted positions of PEG which start from the i -th encoder and end at the j − 1 -th one (inclusion).
By inserting PEGs to ﬁve positions, the top-1 accuracy of the tiny model can achieve 73.4%, which surpasses DeiT-tiny by 1.2%.
Similarly, CPVT-S can achieve 80.5%.
It turns out more PEGs do help, but up to a level where more PEGs become incremental (0vs. 0-11).
[Table]
C LASSFICATION E VALUATION OF S WIN WITH PEG
We show the validation curves when training Swin (Liu et al., 2021) equipped with PEG in Figure 4.
It can boost Swin-tiny from 81.10% to 82.25% (+1.15% ↑ ) on ImageNet.
[Figure]
B. E VALUATION ON S EGMENTATION AND D ETECTION
Semantic segmentation on ADE20K.
We evaluate the performance of PEG on the ADE20K (Zhou et al., 2017) segmentation task.
Based on the Semantic FPN framework (Kirillov et al., 2019), PVT achieves much better results than ResNet (He et al., 2016) baselines.
Under carefully controlled settings, PEG further boosts PVT-tiny by 3.1% mIoU.
Object detection on COCO.
We also perform controlled experiments with the RetinaNet (Lin et al., 2017) framework on the COCO detection task.
The results are shown in Table 13.
In the standard 1 × schedule, PEG improves PVT-tiny by 2.0% mAP.
PEG brings 2.4% higher mAP under the 3 × schedule.
[Table]
B. A BLATION ON O THER F ORMS OF PEG
We explore several forms of PEG based on the tiny model, which change the type of convolution, kernel size and layers.
The inserted position is 0.
The result is shown in Table 14.
When we use large kernel of 7 × 7 or dense convolution, the performance improvement is limited.
Stacking more layers of depth-wise convolution doesn’t bring signiﬁcant improvement.
Therefore, we use the simplest form as our default implementation.
It indicates that this design is enough to provide good position information.
[Table]
C E XAMPLE C ODE C. PEG
In the simplest form, we use a single depth-wise convolution and show its usage in Transformer by the following PyTorch snippet.
Through experiments, we ﬁnd that such a simple design ( i.e ., depthwise 3 × 3) readily achieves on par or even better performance than the recent SOTAs.
We give the torch implementation example in Alg. 1.
D M ORE DISCUSSIONS D. W HY RPE WORKS LESS WELL THAN ABSOLUTE PE?
As mentioned in Section 5.3 (main text), RPE is inferior to the absolute positional encoding.
It is because RPE does not encode any absolute position information.
Also discussed in Section B.
(main text), absolute position information is also important even for ImageNet classiﬁcation as it is needed to determine which object is at the center of the image.
Note that there might be multiple objects in an image, and the label of an image is the category of the object at the center.
Additionally, although RPE becomes popular recently, it is often jointly used with absolute positional encodings (e.g., in ConViT (d’Ascoli et al., 2021)), or the absolute position information is leaked in other ways (e.g., convolution paddings in CoAtNet (Dai et al., 2021)).
This further suggests absolute position information is crucial.
[Table]
D. C OMPARISON TO L AMBDA N ETWORKS
Our work is also related to Lambda Networks (Bello, 2021) which uses 2D relative positional encodings.
We evaluate its lambda module with an embedding size of 128, where we denote its encoding scheme as RPE2D-d128.
Noticeably, this conﬁguration has about 5.9M parameters (comparable to
DeiT-tiny) but only obtains 68.7%.
We attribute its failure to the limited ability in capturing the correct positional information.
After all, lambda layers are designed with the help of many CNN backbones components such as down-sampling to form various stages, to replace ordinary convolutions in ResNet (He et al., 2016).
In contrast, CPVT is transformer-based.
D. Q UALITATIVE A NALYSIS OF CPVT
Thus far, we have shown that PEG can have better performance than the original positional encodings.
However, because PEG provides the position in an implicit way, it is interesting to see if PEG can indeed provide the position information as the original positional encodings.
Here we investigate this by visualizing the attention weights of the transformers. Speciﬁcally, given a 224 ×image (i.e. 14 × 14 patches), the score matrix within a single head is 196 × 196. We visualize the normalized self-attention score matrix of the second encoder block. We ﬁrst visualize the attention weights of DeiT with the original positional encodings.
As shown in
Figure 5 (middle), the diagonal element interacts strongly with its local neighbors but weakly with those far-away elements, which suggests that DeiT with the original positional encodings learn to attend the local neighbors of each patch.
After the positional encodings are removed (denoted by
DeiT w/o PE), all the patches produce similar attention weights and fail to attend to the patches near themselves, see Figure 5 (left).
Finally, we show the attention weights of our CPVT model with PEG.
As shown in Figure 5 (right), like the original positional encodings, the model with PEG can also learn a similar attention pattern, which indicates that the proposed PEG can provide the position information as well.
We illustrate the attention scores in several encoder blocks of DeiT (Touvron et al., 2020) and CPVT in the Fig.
It shows both methods learn similar locality patterns.
As attention scores are computed over the tokens projected in different subspaces (Q and K), they do not necessarily show a strict diagonal pattern, where some may have slight shift, see DeiT in Fig.
6c and CPVT of Fig.
5 right.
[Figure]
[Figure]
D. C OMPARISON WITH O THER A PPROACHES
We further compare our method with other approaches such as CvT (Wu et al., 2021), ConViT (d’Ascoli et al., 2021) and CoAtNet (Dai et al., 2021) on ImageNet validation set in Table 15.
To make fair comparisons, we categorize these methods into two groups: plain and pyramid models.
Since our models are primarily for plain models, we adapt our methods on two popular pyramid frameworks PVT and Swin.
Our CPVT-S-GAP slightly outperforms ConViT-S by 0.2% with 4M fewer parameters and 0.8G fewer FLOPs.
When equipped with pyramid designs, our methods are still comparable to CvT and CoAtNet.
Comparison with DeiT w/ Convolutional Projection.
Note CvT uses a depth-wise convolution in q - k - v projection which they call it Convolutional Projection .
Instead of using it in all layers, we put only one of such design into DeiT-tiny and train such a model from scratch under strictly controlled settings.
We insert it in the position 0 as in our method.
The result is shown in Table 16.
This
CvT-ﬂavored DeiT achieves 70.6% top-1 accuracy on ImageNet validation set, which is lower than ours (72.4%).
Note that q - k - v projections in CvT utilize three depthwise convolutions, therefore, this setting has more parameters than ours.
This attests the difference of CvT and CPVT, verifying our advantage by learning better position encodings other than inserting them in all layers to have the ability to capture local context and to remove ambiguity in attention.
[Table]
[Table]