ResT V2: Simpler, Faster and Stronger
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper proposes ResTv2, a simpler, faster, and stronger multi-scale vision
Transformer for visual recognition.
ResTv2 simplifies the EMSA structure in
ResTv1 (i.e., eliminating the multi-head interaction part) and employs an upsampleoperation to reconstruct the lost medium- and high-frequency information caused bythe downsampling operation.
In addition, we explore different techniques for betterapply ResTv2 backbones to downstream tasks.
We found that although combining
EMSAv2 and window attention can greatly reduce the theoretical matrix multiply
FLOPs, it may significantly decrease the computation density, thus causing loweractual speed.
We comprehensively validate ResTv2 on ImageNet classification,
COCO detection, and ADE20K semantic segmentation.
Experimental results showthat the proposed ResTv2 can outperform the recently state-of-the-art backbones bya large margin, demonstrating the potential of ResTv2 as solid backbones.
Codesare available in the Supplementary Material .
Introduction
Recent advances in Vision Transformers (ViTs) have created new state-of-the-art results on manycomputer vision tasks.
While scaling up ViTs with billions of parameters [22, 9, 45, 40, 13] is a wellproven way to improve the capacity of the ViTs, it is more important to explore more energy-efficientapproaches to build simpler ViTs with fewer parameters and less computation cost while retaininghigh model capacity.
Toward this direction, there are a few works that significantly improve the efficiency of ViTs [35,10, 12, 23, 5].
The first kind is reintroducing the “sliding window” strategy to ViTs.
Among them,
Swin Transformer [23] is a milestone work that partitions the patched inputs into non-overlappingwindows and computes multi-head self-attention (MSA) independently within each window.
Basedon Swin, Focal Transformer [41] further splits the feature map into multiple windows in whichtokens share the same surroundings to effectively capture short- and long-range dependencies.
Thesecond type to improve efficiency is downsampling one or several dimension of MSA.
PVT [35]is a pioneer work in this area, which adopts another non-overlapping patch embedding moduleto reduce the spatial dimension of keys and values in MSA.
ResTv1 [47] further explores threetypes of overlapping spatial reduction methods (i.e., max pooling, average pooling, and depth-wiseconvolution) in MSA to balance the computation and effectiveness in different scenarios.
However,the downsampling operation in MSA will inevitably impair the model’s performance since it destroysthe global dependency modeling ability of MSA to a certain extent (shown in Figure 1).
In this paper, we propose EMSAv2, which explores different upsample strategies adding to EMSA tocompensate for the performance degradation caused by the downsampling operation.
Surprisingly,the “downsample-upsample” combination happens to build an independent convolution hourglass
Results show that downsampling operation will impair the performance while adding an upsampling operation can address this issue.
Detailed comparisons are shown in Appendix A. architecture, which can efficiently capture the local information that is complementary to long-distancedependency with fewer extra parameters and computation costs.
Besides, EMSAv2 eliminates themulti-head interaction module in EMSA to simply the self-attention structure.
Based on EMSAv2, webuild simpler, faster, and stronger general-purpose backbones, ResTv2.
In addition, we explore fourmethods of applying ResTv2 backbones to downstream tasks.
We found that combining EMSAv2and window attention is not that good when the inputs’ resolution is high (e.g., 800 × 1333 ), althoughit can significantly reduce the theoretical matrix multiply FLOPs.
Due to the padding operation inwindow partition and grouping operation of window attention, the computation density of EMSAv2will be significantly decreased, causing lower actual inference speed.
We hope the observationsand discussions can challenge some common beliefs and encourage people to rethink the relationsbetween theoretical FLOPs and actual speeds, particularly running on GPUs.
We evaluate ResTv2 on various vision tasks such as ImageNet classification, object detection/segmentation on COCO, and semantic segmentation on ADE20K.
Experimental results revealthe potential of ResTv2 as strong backbones.
For example, our ResTv2-L yields 84.2% Top-1accuracy (with size 224 2 ) on ImageNet-1k, which is significantly better than Swin-B [23] (83.5%)and ConvNeXt-B
[24] (83.8%), while ResTv2-L has fewer parameters (87M vs. 88M vs. 89M) andmuch higher throughput (415 vs. 278 vs. 292 images/s).
Related Work
Efficient self-attention structures.
MSA has shown great power to capture global dependencyin computer vision tasks [11, 2, 3, 43, 43, 50].
However, the computation complexity of MSAis quadratic to the input size, which might be acceptable for ImageNet classification, but quicklybecomes intractable with higher-resolution inputs.
One typical way to improve efficiency is partitioning the patched inputs into non-overlapping windows and computing self-attention independentlywithin each of these windows (i.e., windowed self-attention).
To enable information communicateacross windows, researchers have developed several integrate techniques, such as shift window[23], spatial shuffle [17], or alternately running global attention and local attention [5, 42] betweensuccessive blocks.
Other ways are trying to reduce spatial dimension of the MSA.
For example, PVT[35] and ResTv1 [47] designed different downsample strategies to reduce the spatial dimension ofkeys and values in MSA.
MViT [12] proposed pooling attention to downsample queries, keys, andvalues spatial resolution.
However, either the windowed self-attention or downsampled self-attentionwill impair the long-distance modeling ability to some content, i.e., surrendering some importantinformation for efficiency.
Our target in this paper is to reconstruct the lost information in a light way.
Convolution enhanced MSA.
Recently, designing transformer models with convolution operationshas become popular since convolutions can introduce inductive biases, which is complementary to
MSA.
ResTv1 [47] and [38] reintroduce convolutions at the early stage to achieve stabler training.
CoAtNet [9] and UniFormer [19] replace MSA blocks with convolution blocks in the former twostages.
CvT [36] adopts convolution in the tokenization process and utilizes stride convolution toreduce the computation complexity of self-attention.
CSwin Transformer [10] and CPVT [6] adopta convolution-based positional encoding technique and show improvements on downstream tasks.
Conformer [28] and Mobile-Former[4] combine Transformer with an independent ConvNet modelto fuse convolutional features and MSA representations under different resolutions.
ACmix [26]explores a closer relationship between convolution and self-attention by sharing the 1 × 1 convolutionsand combining them with the remaining lightweight aggregation operations.
The “downsampleupsample” branch in ResTv2 happens to build an independent convolutional module, which caneffectively reconstruct information lost by the MSA module.
Proposed Method3.1 A brief review of ResTv1
ResTv1[47] is an efficient multi-scale vision Transformer, which can capably serve as a generalpurpose backbone for image recognition.
ResTv1 effectively reduces the memory of standard MSA[34, 11] and models the interaction between multi-heads while keeping the diversity ability.
To tackleinput images with an arbitrary size, ResTv1 constructs the positional embedding as spatial attention,which models absolute positions between pixels with the help of zero paddings in the transformationfunction.
EMSA is the critical component in ResTv1 [47] (shown in Figure 2(a) ).
Given a 1D input tokenx ∈ R n × d m , where n is the token length, d m is the channel dimension.
EMSA first projects x usinga linear operation to get the query: Q = xW
q + b q , where W q and b q are the weights and bias oflinear projection.
After that, Q is split into k groups (i.e., k heads) to prepare for the next step, i.e.,
Q ∈ R k × n × d k , where d k = d m /k is the head dimension.
To compress memory, x is reshaped to its2D size and then are downsampled by a depth-wise convolution to reduce the height and width.
Afterthat, the output x ′ is reshaped to the 1D size, and then a Layer Norm [1] is added.
Then the authoremploys the same way as to obtain Q to get key K and value V on x ′ .
The output of EMSA can becalculated by
[Equation]
where “Conv” is applied to model the interactions among different heads.
“Norm” can be Instance
Norm [33] or Layer Norm [1], which is applied to re-weight the attention matrix captured by differentheads.
3.2 ResTv2
As shown in Figure 1, although downsample operation in EMSA can significantly reduce thecomputation cost, it will inevitably lose some vital information, particularly in the earlier stages,where the downsampling ratio is relatively higher, e.g., 8 in the first stage.
To address this issue,one feasible solution is to introduce spatial pyramid structural information.
That is, setting differentdownsampling rates for the input, calculating the corresponding keys and values respectively, andthen combining these multi-scale keys and values along the channel dimension.
The obtained newkeys and values are then sent to the EMSA module to model global dependencies or directly calculatemulti-scale self-attention with the original multi-scale keys and values.
However, the multi-path calculation of keys and values will greatly reduce the computational densityof self-attention, although the theoretical FLOPs do not seem to change much.
For example, themulti-path Focal-T [41] and the single-path Swin-T [23] have comparable theoretical FLOPs (4.9G
[Figure]
vs. 4.5G), but the actual inference throughput of Focal-T is only 0.42 times of Swin-T (319 vs. 755images/s).
In order to effectively reconstruct the lost information without having a large impact on the actualrunning speed, in this paper, we propose to execute an upsampling operation on the values directly.
There are many upsampling strategies, such as “nearest”, “bilinear”, “pixel-shuffle”, etc.
We find thatall of them can improve the model’s performance, but “pixel-shuffle” works better.
We call this newself-attention structure EMSAv2.
The detailed structure is shown in Figure 2(b).
Surprisingly, the “downsample-upsample” combination in EMSAv2 happens to build an independentconvolution branch, which can efficiently reconstruct the lost information with fewer extra parametersand computation costs.
Besides, we find that the multi-head interaction module of the self-attentionbranch in EMSAv2 will decrease the actual inference speed of EMSAv2, although it can increase thefinal performance.
And the performance improvements will be decreased as the channel dimensionfor each head increases.
Therefore, we remove it for faster speed under default settings.
However, ifthe head dimension is small (e.g., d k = 64 or smaller), the multi-head interaction module will makea difference (Detailed Results can be found in Appendix B).
By doing so, we can also increase thetraining speed since the computation gap between the self-attention branch and the upsample branchis reduced.
The mathematical definition of the EMSAv2 module can be represented as
[Equation]
3.3 Model configurations
We construct different ResTv2 variants based on EMSAv2.
ResTv2-T/B/L, to be of similar complexities to Swin-T/S/
B. We also build ResTv2-S to make a better speed-accuracy trade-off.
The fourvariants only differ in the number of channels, heads’ number of EMSAv2, and blocks in each stage.
Other hyper-parameters are the same as ResTv1[47].
Note that the upsampling module in ResTv2introduces extra parameters and FLOPs.
To make a fair comparison, the block number in the firststage of ResTv2-T/S/B is set to 1, half of the one in ResTv1.
Assume C is the channel number ofhidden layers in the first stage.
We summarize the configurations below:
• ResTv2-T: C = 96 , heads = {1, 2, 4, 8}, blocks number = {1, 2, 6, 2} • ResTv2-S: C = 96 , heads = {1, 2, 4, 8}, blocks number = {1, 2, 12, 2}
• ResTv2-B: C = 96 , heads = {1, 2, 4, 8}, blocks number = {1, 3, 16, 3}
• ResTv2-L: C = 128 , heads = {2, 4, 8, 16}, blocks number = {2, 3, 16, 2}
Detailed model size, theoretical computational complexity (FLOPs), and hyper-parameters of themodel variants for ImageNet image classification are listed in Appendix E.
3.4 Explanation of upsample branch
To better explain the role of the upsample branch in EMSAv2, we plot the Fourier transformed featuremaps of EMSAv2, the separate self-attention branch, and upsample branch of ResTv2-T following[27].
As shown in Figure 3, the upsample module can indeed reconstruct the lost information, whichmainly focuses on the medium- and high-frequency part, particularly in the earlier blocks.
Thisaligns with our expectations since EMSAv2 downsamples the input with larger stride depth-wiseconvolutions in earlier blocks. [Figure]
Empirical Evaluations on ImageNet
[Figure]
4.1 Settings
[Figure]
The ImageNet-1k dataset consists of 1.28M training images and 50k validation images from 1,000classes.
We report the Top-1 and Top-5 accuracy on the validation set.
We summarize our trainingand fine-tuning setups below.
More details can be found in Appendix D.1.
We train ResTv2 for 300 epochs using AdamW [25], with a cosine decay learning rate schedulerand 50 epochs of linear warm-up.
An initial learning rate of 1.5e-4 × batch_size / 256, a weightdecay of 0.05, and gradient clipping with a max norm of 1.0 are used.
For data augmentations, weadopt common schemes including Mixup [46], Cutmix [44], RandAugment [8], and Random Erasing[48].
We regularize the networks with Stochastic Depth [16] and Label Smoothing [32].
We use
Exponential Moving Average (EMA) [29] as we find it alleviates larger models’ over-fitting.
Thedefault training and testing resolution is 224 2 .
Additionally, we fine-tune at a large resolution of 384 2 ,adopting AdamW for 30 epochs, with a learning rate 1.5e-5 × batch_size / 256, a cosine decayingschedule afterward, no warm up, and weight decay of 1e-8.
4.2 Main Results
Table 1 shows the result comparison of the proposed ResTv2 with three recent Transformer variants,
ResTv1 [47], Swin Transformer [23], and Focal Transformer [41], as well as two strong ConvNets:
RegNet [30] and ConvNeXt [24].
ResTv2 competes favorably with them in terms of a speedaccuracy trade-off.
Specifically, ResTv2 outperforms ResTv1 of similar complexities across theboard, sometimes with a substantial margin, e.g., +0.7% (82.3% vs. 81.6%) in terms of Top-1accuracy for ResTv2-T.
Besides, ResTv2 outperforms the Focal counterparts with an average × 1 .
[Table]
[Table] [Table] [Table]
[Table]
inference throughput acceleration, although both of them share similar FLOPs.
A highlight from theresults is ResTv2-B: it outperforms Focal-S by +0.1% (83.7% vs. 83.6%), but with +203% higherinference throughput (582 vs. 192 images/s).
ResTv2 also enjoys improved accuracy and throughputcompared with similar-sized Swin Transformers, particularly for tiny models, the Top-1 accuracyimprovement is +1.0% and (82.3% vs. 81.3%).
Additionally, we observe a highlight accuracyimprovement when the resolution increases from 224 2 to 384 2 .
An average +1.4% Top-1 accuracy isachieved.
We can conclude that the proposed ResTv2 also possesses the ability to scale up capacityand resolution.
4.3 Ablation Study
Here, we ablate essential design elements in ResTv2-T using ImageNet-1k image classification.
Tosave computation energy, all experiments in this part are trained for 100 epochs, and 10 of them areapplied for linear warm-up, with other settings unchanged.
Upsampling Targets.
There are three options for upsampling, the output of down-sample operationx ′ , K, and V. Table 2(a) shows the results of upsampling these targets.
Undoubtedly, upsampling K or
V achieves better results than x ′ since K and V are obtained from x ′ via linear projection, enablingthe communication of information between different features.
Upsampling V works best.
This canbe attributed to the fact that unified modeling of the same variable (i.e., V) can better enhance thefeature representation.
Upsampling Strategies.
Table 2(b) varies the upsampling strategies.
We can see that all of thethree upsample strategies can increase the Top-1 accuracy, which means the upsample operation canprovide information not captured by self-attention.
In addition, pixel-shuffle operation obtains muchstronger feature extraction capabilities with a few parameters and FLOPs increase.
[Table]
ConvNet or EMSA?
As mentioned in Section 3.2, we point out that the “downsampling-upsampling”pipeline in EMSAv2 can constitute a complete ConvNet block for extracting features.
Here, weseparate it (i
[Table]
e., a ResTv2-T variant without self-attention) to see whether it can replace the MSAmodule in ViTs.
Table 2(c) shows that with the same number of blocks, the performance of the
ConvNet version is quite poor.
In order to show that insufficient parameters and computation do notmainly cause this issue, we constructed ConvNetv2, i.e., adding a block in both the first and secondstages, so that the model complexity of the ConvNetv2 and EMSA versions (without upsample) isequivalent.
Experimental results show that ConvNetv2 still performs inferior to the EMSA version(77.91 vs. 79.04 in terms of Top-1 accuracy).
This observation indicates that ConvNet does not actlike EMSA.
Thus, it is not reasonable to replace MSA with ConvNet in ViTs.
However, combining the upsample module and EMSA(i.e., EMSAv2) indeed improves the overall performance.
We can conclude that the downsampling operation of
EMSAv2 will lead to the loss of input information, resulting in insufficient information extracted by the EMSAmodule constructed on this basis, and the upsamplingoperation can reconstruct the lost information.
We further plot the linear CKA [18] curves to measurewhich is more critical for EMSAv2 (i.e., the combinationvariant), the self-attention branch (i.e., EMSA) or the upsample module?
Figure 4 illustrates that in earlier blocks,feature representations extracted by self-attention andupsample module show a relatively low similarity, whilein deeper blocks, they exhibit a surprisingly high similarity (shown in the red polyline).
We can conclude thatfeatures in earlier blocks extracted by self-attention andupsample module are complementary.
Combining them can boost the final performance.
In deeperblocks, particularly the last block, self-attention behaves like the upsample module (linear CKA> 0.8), although it shows a higher similarity with EMSAv2 (shown in the purple polyline).
Theseobservations could provide a guide for designing hybrid models, i.e., integrating ConvNets and MSAsin the early stages can significantly improve the performance of ViTs.
Positional Embedding.
We also validate whether Positional Embedding (short for PE) still works in
ResTv2.
Table 2(d) shows PE can still improve the performance, but not that obvious as ResTv1 [47].
Specifically, both RPE and PA work well, but PA is more flexible, which can process input images ofarbitrary size without interpolation or fine-tuning.
Therefore, we apply PA as the default PE strategy.
Detailed settings about these positional embedding can be found in Appendix F.
Empirical Evaluation on Downstream Tasks
5.1 Object Detection and Segmentation on COCO
Settings.
Object detection and instance segmentation experiments are conducted on COCO 2017,which contains 118K training, 5K validation, and 20K test-dev images.
We report results using thevalidation set.
We fine-tune Mask R-CNN [14] with ResTv2 backbones.
Following [24], we adoptmulti-scale training, AdamW optimizer, “ × 1 schedule” for ablation study, and “ × 3 schedule” formain results.
Further details and hyper-parameter settings can be found in Appendix D.2.
Ablation Study.
There are several ways to fine-tune ImageNet pre-trained ViT backbones.
Theconventional one is the global style, which directly adopts ViTs into downstream tasks.
The recentpopular one is window-style (short for Win), which constrained part or all MSA modules of ViTs intoa fixed window to save computation overhead.
However, performing all MSA into a limited-sizedwindow will lose the MSA’s long-range dependency ability.
To alleviate this issue, we add a 7 × 7depth-wise convolution layer after the last block in each stage to enable information to communicateacross windows.
We call this style CWin.
In addition, [20] provides a hybrid approach (HWin) tointegrate window information, i.e., computes MSA within a window in all but the last blocks in eachstage that feed into FPN [21].
Window sizes in Win, CWin, and HWin are set as [64, 32, 16, 8] forthe four stages.
tuning styles on
[Table]
Table 3(a) shows that although restricted EMSAv2 into fixed windows can effectively reduce theoretical FLOPs, the actual inference speed is almost double the global style, and the box/mask AP is lowerthan the global one.
Therefore, we adopt the Global fine-tuning strategy as default in downstreamtasks to get better accuracy and inference speed.
There are mainly two reasons for the decrease in inference speed: (1) padding to inputs is required tosatisfy the divisible non-overlapped window partition.
In our settings, the theoretical upper limit ofpadding in the first stage is 63 × 63 , close to the lower bound of the input features’ size (i
[Table]
e., 64 × 64 ).(2)
the process of window partition is similar to feature grouping, which reduces the computationaldensity of GPUs.
Table 3(b) shows the detailed FLOPs of different modules.
We can see that window-based fine-tunemethods can effectively reduce the “Matmul” (short of matrix multiply) FLOPs with the cost ofintroducing extra “Linear” FLOPs, demonstrating that window partition padding is common indetection tasks.
In addition, the “Matmul” operation is not the most time-consuming part of thefour settings ( ≤ 16% ).
Therefore, it is reasonable to speculate that window attention will reducecomputational density.
We hope the observations and discussions can challenge some common beliefs and encourage peopleto rethink the relations between theoretical FLOPs and actual speeds, particularly running on GPUs.
[Table]
Main Results.
Table 4 shows main results of ResTv2 comparing with ConvNeXt [24], Swin
Transformer [23], and traditional ConvNet such as ResNet [15].
Across different model complexities,
ResTv2 outperforms Swin Transformer and ConvNeXt with higher mAP and inference FPS (framesper second), particularly for tiny models.
The mAP improvements over Swin Transformer are +1.6box AP (47.6 vs. 46.0), and +1.6 mask AP (43.2 vs. 41.6).
When comparing with ConvNeXt, theimprovements are +1.4 box AP (47.6 vs. 46.2), and +1.5 mask AP (43.2 vs. 41.7).
5.2 Semantic Segmentation
on ADE20K
Settings.
We also evaluate ResTv2 backbones on the ADE20K [49] semantic segmentation taskwith UperNet [39].
ADE20K contains a broad range of 150 semantic categories.
It has 25K imagesin total, with 20K for training, 2K for validation, and another 3K for testing.
All model variantsare trained for 160k iterations with a batch size of 16.
Other experimental settings follow [23] (see
Appendix D.2 for more details).
[Table]
Results.
In Table 5, we report validation mIoU with multi-scale testing.
ResTv2 models can achievecompetitive performance across different model capacities, further validating the effectiveness of ourarchitecture design.
Conclusion
In this paper, we proposed ResTv2, a simpler, faster, and stronger multi-scale vision
Transformer for image recognition.
ResTv2 adopts pixel-shuffle in EMSAv2 to reconstruct the lost information due to the downsampling operation.
In addition, we explore different techniques for better apply ResTv2 to downstream tasks.
Results show that the theoretical FLOPs is not a good reflection of actual speed, particularly running on GPUs.
We hope that these observations could encourage people to rethink architecture design techniques that can actually prompt the network’s efficiency
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] In the supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] In the Appendix.(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] In the Appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Comparison with different MSA variants.
This section provides more results about the effectiveness of the upsampling operation in different
MSA variants under 100 training epochs settings following Section 4.3.
As shown in Table 6, MSAvariants with the “downsample-upsample” branch can compete favorably with the standard MSA interms of the Top-1 accuracy but with much higher inference throughput.
[Table]
B Results of Multi-head Interaction Module
In this section
[Table]
we estimate the role of the Multi-head Interaction Module (short for MHIM).
Trainingsettings are the same as Section 4.3, i.e., under the 100 training epochs settings.
As shown in Table 7,
MHIM can significantly improve the Top-1 accuracy with the cost of decreased inference throughputwhen the channel dimension of each head
d k is smaller.
When d k is larger (e.g., 96), the accuracyimprovement is limited.
Therefore, we may choose whether to apply MHIM in EMSAv2 accordingto different scenarios.
[Table]
C Experiments for ViTs without downsample strategies.
We also provide the results of Swin Transformer [23] to estimate the effectiveness of the proposedupsample design, results are shown in Table 8.
[Table]
We can see that the designed upsample operation is a general strategy that can greatly improve theperformance of ViTs even without downsample operations.
D Experimental Settings
D.1 ImageNet pre-training and finetuning settings
We list the settings for training and fine-tuning on ImageNet-1k in Table 9.
The settings are used forour main results in Table 1 (Section 4.2).
The fine-tuning starts from the best checkpoint weightsobtained in pre-training.
[Table] [Table]
learning rate schedule cosine decay cosine decay warmup epochs N/A warmup schedule linear N/A RandAugment [8] (9, 0.5) (9, 0.5) label smoothing [32] 0.
Mixup [46] 0. N/A Cutmix [44] 1.
N/A stochastic depth [16] 0.1/0.2/0.3/0. gradient clip 1. EMA [29] 0. N/A
[Table]
For COCO and ADE20K experiments, we follow the training settings used in ResTv1 [47] and
ConvNeXt [24].
We adopt Detectron2 [37] toolboxes for object detection, and MMSegmentation [7] toolboxes for semantic segmentation.
We use the best model weights (instead of EMA weights) from
ImageNet pre-training as network initialization.
Object Detection on COCO.
We utilize multi-scale training, i.e., resizing the input such that theshort side is between 480 and 800 while the longer side is at most 1333, AdamW optimizer withinitial learning rate of 0.0001, weight decay of 0.05, and batch size of 16 (8 V100 GPUs with twoimages per GPU) , “ × 1 ” schedule (90K iterations with learning rate decayed by × 10 at 60K and80K steps) for ablation study, and “ × 3 ” schedule (270K iterations with learning rate decayed by× 10 at 210K and 250K steps) for main results.
We apply standard data augmentation, that is resize,random flip and normalize.
Additionally, we adopt stochastic depth with ratio of 0.1/0.2/0.3 for
ResTv2-T/S/B. All results are reported on the validation set.
Semantic segmentation on ADE20K.
We employ the AdamW optimizer with an initial learningrate of 1.5e-4, a weight decay of 0.05, stage-wise learning rate decay with a ratio of 0.9, and a linearwarmup of 1,500 iterations.
Models are trained on 8 GPUs with two images per GPU for 160Kiterations.
Stochastic depth with ratio of 0.1/0.2/0.3 for ResTv2-T/S/B. All models are trained onthe standard setting with an input of 512 × 512 .
We report validation mIoU results using multi-scaletesting.
E Detailed Architectures
The detailed architecture specifications are shown in Table 10, where an input image size of 224 × 224is assumed for all architectures.
“Conv-K_C_S” means convolution layers with kernel size K , outputchannel C and stride S .
“MLP_C” is the FFN layer with hidden channel 4 C and output channel C . [Table] [Table]
[Table]
[Table] [Table]
F Positional Embedding
[Table]
In Section 4.
[Table]
we explore three types of positional embedding in ResTv2.
Here
[Table]
we give themathematical definition of them.
Assume x ∈ R hw × d m be the input token sequence
[Table]
where h , w andd m are the input’s height, width and channel dimensions [Table] respectively.
APE.
Let θ ∈ R n × c be position parameters, then APE[11] can be represented as
[Equation]
In APE, θ should be the same size as the input x , i.e., n = h · w , whatever θ is sinusoidal or learnable.
RPE.
RPE applied in this paper is the relative position variant introduced in [31], which is added tothe attention map to represent the structure of inputs (i.e., 2D images).
Let Q be the queries, K bethe keys, k be the number of heads, and d k be the head dimension.
Then position P can be obtainedby P
= P h + P w , where P h ∈ R k ×
h
×
1 × d k and P w ∈ R k × 1 × w ′ ×
d k represent the positions alongthe height dimension and the width dimension, respectively.
The RPE can then be defined as:
[Equation]
Therefore, RPE can capture both the “content-content” and “content-position” information.
However,
P is required to share the same resolution as K , i.e., h ′ =
h, w ′ =
w in the standard MSA, andh ′ =
h/r, w ′ =
w/r in EMSAv2, where r is the reduction factor.
PA. PA is one kind of spatial attention that utilizes a simple 2D depth-wise convolution as thetransform function [47].
Thanks to the shared parameters and locality of convolution, PA can tacklearbitrary lengths of inputs and capture their absolute positions with the help of zero paddings.
Themathematical formulation of PA can be defined as
[Equation]
[Equation]
where σ ( · ) is the sigmoid function adopted to scale the spatial attention weights.