Original: G LOBAL C ONTEXT V ISION T RANSFORMERS
Revised: G LOBAL C ONTEXT V ISION T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision.
Revised: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision tasks.

Original: Our method leverages global context self-attention modules, joint with standard local selfattention, to effectively yet efficiently model both long and short-range spatial interactions, without the need for expensive operations such as computing attention masks orshifting local windows.
Revised: The core of the novel model are global context self-attention modules, joint with standard local self-attention, to effectively yet efficiently model both long and short-range spatial interactions, as an alternative to complex operations such as an attention masks or local windows shifting. While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values.

Original: In addition, we address the lack ofthe inductive bias in ViTs, and propose to leverage a modified fused inverted residual blocks in our architecture.
Revised: In addition, we address the lack of inductive bias in ViTs and improve the modeling of inter-channel dependencies by proposing a novel downsampler which leverages a parameter-efficient fused inverted residual block.

Original: Our proposed GC ViT achieves  state-of-the-art results across image classification, object detection and semantic segmentation tasks.
Revised: The proposed GC ViT achieves new state-of-the-art performance across image classification, object detection and semantic segmentation tasks.

Original: On ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28 M, 51 M and 90 M parameters achieve 83.4% , 83.9% and 84.5% Topaccuracy, respectively, surpassing comparably-sized prior art such as CNN-based ConvNeXt and ViT-based Swin Transformer by a large margin.
Revised: On ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28 M, 51 M and 90 M parameters achieve 83.4% , 83.9% and 84.4% Top-1 accuracy, respectively, surpassing comparablysized prior art such as CNN-based ConvNeXt and ViT-based Swin Transformer.

Original: Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins.
Revised: Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation on MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins.

Original: Code and pre-trained models available at https://anonymous.4open.science/r/GCViTs-8852/.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: During the recent years, Transformers (Vaswani et al., 2017) have achieved State-Of-The-Art (SOTA) performance in Natural Language Processing (NLP) benchmarks and became the de facto model for various tasks.
Revised: During the recent years, Transformers (Vaswani et al., 2017) have achieved State-Of-The-Art (SOTA) performance in Natural Language Processing (NLP) benchmarks and became the de facto model for various tasks.

Original: A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021).
Revised: A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021).

Original: Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in a monolithic architecture with minor differences comparing to encoder of the original Transformer.
Revised: Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in a monolithic architecture with minor differences comparing to encoder of the original Transformer.

Original: Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA or competitive performance in various computer vision tasks.
Revised: Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA or competitive performance in various computer vision tasks.

Original: In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison to CNN.
Revised: In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison to CNN.

Original: However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) in which capturing multi-scale long-range information is crucial for accurate representation modeling.
Revised: However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) in which capturing multi-scale long-range information is crucial for accurate representation modeling.

Original: Several efforts (Liu et al., 2021; Dong et al., 2021; Chu et al., 2021a;
Revised: Several efforts (Liu et al., 2021; Dong et al., 2022; Chu et al., 2021a;

Original: Tu et al., 2022), most notably
Revised: Tu et al., 2022), most notably

Original: Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and longrange spatial dependencies by proposing multi-resolution architectures in which the self-attention is computed in local windows and cross-window connections such as window shifting are used for modeling the interactions across different regions.
Revised: Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and longrange spatial dependencies by proposing multi-resolution architectures in which the self-attention is computed in local windows. In this paradigm, cross-window connections such as window shifting are used for modeling the interactions across different regions.

Original: Despite this progress, the limited receptive field of local windows baffles the capability of self-attention to capture long-range information, and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window.
Revised: Despite the progress, the limited receptive field of local windows challenges the capability of self-attention to capture long-range information, and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window.

Original: Subsequent efforts such as Focal Transformer (Yang et al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.
Revised: Subsequent efforts such as Focal Transformer (Yang et al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.

Original: The burden of computationally expensive operations such as token unfolding and rolling and additional computations of key and values can significantly affect the run-time and throughput.

Original: In this work, we introduce the Global Context (GC) ViT network to address these limitations.
Revised: In this work, we introduce the Global Context (GC) ViT network to address these limitations.

Original: Specifically, we propose a hierarchical ViT architecture consisting of local and global self-attention modules.
Revised: Specifically, we propose a hierarchical ViT architecture consisting of local and global self-attention modules.

Original: At each stage, we compute global query tokens, using modified fused inverted residual 
Revised: At each stage, we compute global query tokens, using a novel fused inverted residual blocks,

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: Swin
Revised: Swin

Original: [Figure]
Revised: [Figure]

Original: T ConvNext
Revised: ConvNext-T

Original: [Figure]

Original: S
Revised: ConvNext-S ConvNext

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: DeiT-S
Revised: DeiT-S

Original: DeiT
Revised: DeiT

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: PVT-S
Revised: PVT-S

Original: PVT-M
Revised: PVT-M

Original: PVT-L
Revised: PVT-L

Original: Focal-T
Revised: Focal-T

Original: Focal-S
Revised: Focal-S

Original: Focal [Figure] B
Revised: Focal-B

Original: GC ViT-XT
Revised: GC ViT-XT

Original: GC ViT-T
Revised: GC ViT-T

Original: GC ViT-S GC ViT
Revised: GC ViT-S GC ViT-B

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ConvNext
Revised: Swin

Original: [Figure]
Revised: [Figure]

Original: T ConvNext
Revised: ConvNext-T

Original: [Figure]

Original: S ConvNext
Revised: ConvNext-S ConvNext-B

Original: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: DeiT
Revised: DeiT-S

Original: [Figure]

Original: S

Original: DeiT
Revised: DeiT-

Original: [Figure]
Revised: [Figure]

Original: PVT-S
Revised: PVT-S

Original: PVT-M
Revised: PVT-M

Original: PVT-L
Revised: PVT-L

Original: Focal-T
Revised: Focal-T

Original: Focal-S Focal
Revised: Focal-S Focal-B

Original: [Figure]

Original: GC ViT-XT
Revised: GC ViT-XT

Original: GC ViT-T GC ViT-
Revised: GC ViT-T GC ViT-S

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  P
Revised: L P

Original:  D o w n s a m p l e
Revised: M L P D o w n s a m p l e

Original: Lo c a l M SA
Revised: Lo c a l M SA

Original: Global Token Gen.
Revised: Global Token Gen.

Original: G l oba l M
Revised: G l oba l M

Original: SA M L P
Revised: SA M L P

Original: M L P
Revised: M L P

Original: Lo c a l M SA
Revised: Lo c a l M SA

Original: Global Token Gen.
Revised: Global Token Gen.

Original: G l oba l M
Revised: G l oba l M

Original: SA M L P 2 D
Revised: SA M L P 2 D

Original: A v g P oo l
Revised: A v g P oo l

Original: H ead D
Revised: H ead D

Original: o w n s a m p l e
Revised: o w n s a m p l e

Original: M L Pblocks, which we refer to as  Fused-MBConv blocks, that encompass global contextual information from different image regions.
Revised: M L P which we refer to as modified Fused-MBConv blocks, that encompass global contextual information from different image regions.

Original: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values.
Revised: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and value representations.

Original: As we demonstrate in Sec. 5, sharing global queries allow for reducing number of parameters and FLOPs and increasing generalizability.

Original: The design of our proposed framework for global query generation and self-attention is intuitive and simple and can be efficiently implemented using major deep learning framework.
Revised: The design of our proposed framework for global query generator and self-attention is intuitive and simple and can be efficiently implemented using major deep learning framework.

Original: Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied to high-resolution images.
Revised: Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied to high-resolution images.

Revised: In addition, we propose a novel downsampling block with a parameter-efficient fused-MBConv layer to address the lack of inductive bias in ViTs and enhancing the modeling of inter-channel dependencies.

Original: We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks.
Revised: We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks.

Original: For image classification using ImageNet-1K dataset, 
Revised: For image classification using ImageNet-1K dataset, GC

Original: GC ViT with 28 M, 51  M and 90 M parameters, referred to as tiny, small and base  variants, achieve new SOTA benchmarks of 83.4% , 83.9%  and 84.5% Top-1 accuracy.
Revised: ViT with 28 M, 51 M, 90 M and 201 M parameters, referred to as tiny, small, base and large variants, achieve new SOTA benchmarks of 83.4% , 83.9% , 84.4% and 84.6% Top-1 accuracy.

Original: Hence, GC ViT consistently outperforms both ConvNeXt (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1).
Revised: Hence, GC ViT consistently outperforms both ConvNeXt (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1).

Original: Using a pre-trained GC ViT base backbone with  Cascade Mask RCNN (He et al., 2017) head, our model achieves a box mAP of 52.2 for object detection and a mask mAP of 45.2 for instance segmentation on the MS COCO test-dev.
Revised: Using a pre-trained GC ViT base backbone with a Cascade Mask RCNN (He et al., 2017) head, our model achieves a box mAP of 52.9 for object detection and a mask mAP of 45.8 for instance segmentation on the MS COCO dataset.

Original: In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves a mIoU of 49.0 on ADE20K for semantic segmentation.
Revised: In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves a mIoU of 49.0 on ADE20K for semantic segmentation.

Original: Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sized models on both MS COCO and ADE20K datasets.
Revised: Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sized models on both MS COCO and ADE20K datasets.

Original: Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.
Revised: Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.

Original: The main contributions of our work are summarized as follows:
Revised: The main contributions of our work are summarized as follows:

Original: • A novel hierarchical Transformer model, called GC ViT, that can be employed as a general backbone in various computer vision tasks such as classification, detection, instance segmentation down-stream tasks such as detection, instance and semantic segmentation.
Revised: • We introduce a compute and parameter-optimized hierarchical ViT with reparametrization of the design space ( e.g. , embedding dimension, number of heads, MLP ratio). • We design an efficient CNN-like token generator that encodes spatial features at different resolutions for global query representations.

Original: • A novel and yet simple design comprising of global self-attention and token generation modules that allow for modeling long-range dependencies by capturing global contextual information  and hence eliminating the need for highly sophisticated or complex operations.
Revised: • We propose global query tokens that can effectively capture contextual information in an efficient manner and model both local and global interactions. • We introduce a parameter-efficient downsampling module with modified Fused MB-Conv blocks that not only integrates inductive bias but also enables the modeling of inter-channel dependencies.

Original: • As demonstrated in Fig. 1, the proposed GC ViT achieves new SOTA benchmarks  on ImageNet-1Kdataset fora variety of model sizes and FLOPs, outperforming both CNN and ViT-based models by a significant margin. Using GC ViT as the backbone yields SOTA or competitive performance for object detection and semantic segmentation on MS COCO and ADE20K datasets, respectively.
Revised: • We demonstrate new SOTA benchmarks for : (1) ImageNet classification with Pareto fronts on ImageNet-1K for model size and FLOPs (see Fig. 1), and (2) downstream tasks such as detection, instance segmentation and semantic segmentation on MS COCO and ADE20K, respectively.

Original:  GC V I T ARCHITECTURE
Revised:  GC V I T ARCHITECTURE

Original: Architecture.
Revised: Architecture.

Original: Fig.
Revised: Fig.

Original: 2 depicts the architecture of GC ViT.Similar to previous efforts (Liu et al., 2021;
Revised: 2 depicts the architecture of GC ViT.

Original: Yang et al., 2021b), we use a hierarchical framework to obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension by factors of 2 and 2 , respectively.
Revised: We propose a hierarchical framework to obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension, both by factors of 2 .

Original: At first, given an input image with resolution of x ∈ R H × W × 3 ,following Wang et al. (2022), we obtain overlapping patches by applying a 3 ×convolutional layer with a stride of 2 and appropriate padding.
Revised: At first, given an input image with resolution of x ∈ R H × W × 3 , we obtain overlapping patches by applying a 3 ×convolutional layer with a stride of 2 and appropriate padding.

Original: Then patches are projectedit into a C -dimensional embedding space.
Revised: Then patches are projected into a C -dimensional embedding space with another 3 × 3 convolutional layer with stride 2 .

Original: We show the benefit of using overlapping as opposed to non-overlapping patches in Sec. 5.
Revised: Every

Original: Every GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features.
Revised: GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features.

Original: Both operate in local windows like Swin Transformer (Liu et al., 2021), however, the global self-attention accesses global features extracted by the Global Token Generator (GTG).
Revised: Both operate in local windows like Swin Transformer (Liu et al., 2021), however, the global self-attention has access to global features extracted by the global token generator.

Original: The GTG is a CNN-like module that extracts features from the entire image only once at every stage.
Revised: The token generator is a CNN-like module that extracts features from the entire image only once at every stage.

Original: The spatial resolution is decreased by 2 whileincreasing the number of channels  by  a downsampling block after every stage.
Revised: After each stage, the spatial resolution is decreased by 2 while the number of channels is increased by 2 via a downsampling block.

Original: Resulting features are passed through average pooling and linear layers to create an embedding for a downstream task.
Revised: Resulting features are passed through average pooling and linear layers to create an embedding for a downstream task.

Original: The GC ViT architecture relies on novel blocks such as a downsampling operator, a global query generator and a global self-attention module described in the next sections.Downsamplingblock
Revised: The GC ViT architecture benefits from novel blocks such as a downsampling operator , a global query generator and a global self-attention module described in the next sections.

Original: F u s ed
Revised: [Figure]

Original: M B C on v C on v 3 x 3 , s t r i de La y e r N o r m
Revised: [Figure]

Original: Downsampling.
Revised: Downsampler.

Original: We borrow an idea of spatial feature contraction from
Revised: We leverage an idea of spatial feature contraction from

Original: CNN models that imposes locality bias and cross channel communication while reducing dimensions.
Revised: CNN models that imposes locality bias and cross channel interaction while reducing dimensions.

Original: We use a modified Fused-MBConv block, followed by a max pooling layer with a kernel size of 3 and stride ofas a downsampling operator, see Fig 3.
Revised: We utilize a modified Fused-MBConv block, followed by a max pooling layer with a kernel size of 3 and stride ofas a downsampling operator, see Fig 3.

Original: The Fused-MBConv block in our work is similar to the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in
Revised: The Fused-MBConv block in our work is similar to the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in

Original: [Equation]
Revised: [Equation]

Original: where SE, GELU and DW-Conv 3 × 3 denote Squeeze and Excitation block (Hu et al., 2018b), Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016) and 3 ×depth-wise convolution, respectively.
Revised: where SE, GELU and DW-Conv 3 × 3 denote Squeeze and Excitation block (Hu et al., 2018), Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016) and 3 × 3 depth-wise convolution, respectively.

Original: In our proposed architecture, the Fused-MBConv blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies. 
Revised: In our proposed architecture, the Fused-MBConv blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies. It is ablated in Table 5.

Original: Attention.
Revised: Attention.

Original: Multi-head self-attention is the the core computational operator in the GC ViT architecture to extract semantic information from the image.
Revised: Multi-head self-attention is the the core computational operator in the GC ViT architecture, it extracts semantic information from the image.

Original: GC ViT is composed of local and global selfattention modules illustrated in Fig 4.
Revised: GC ViT is composed of local and global selfattention modules illustrated in Fig 4.

Original: Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size.
Revised: Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size.

Original: The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross-patch communication with those far beyond the local window.
Revised: The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross-patch communication with those far beyond the local window.

Original: Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module. 
Revised: Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module. [Figure] [Figure]

Original: 2. G LOBAL Q UERY G ENERATOR
Revised: 2. G LOBAL Q UERY G ENERATOR

Original: We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value features.
Revised: We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value feature pairs.

Original: Specifically, as shown in Fig.
Revised: Specifically, as shown in Fig.

Original: 5, a layer f in [Figure]
Revised: 5, a layer f in the the generator consists of a Fused-MBConv block followed by a max pooling layer, similar to the one described in Sec.

Original: [Figure]
Revised: [Figure]

Original: the the generator consists of a Fused-MBConv block followed by a max pooling layer, similar to the one described in Sec. 2, and the final global query q g , i at stage i ( i ∈ { 1 , 2 , 3 , 4 } ) of GC ViT is computed according to
Revised: and the final global query q g , i at stage i ( i ∈ { 1 , 2 , 3 , 4 } ) of GC ViT is computed according to

Original: [Equation]
Revised: [Equation]

Original: These query tokens are computed once at every stage of GC ViT model and shared across all global attention blocks, hence decreasing number of parameters and FLOPs and improving the generalizability.
Revised: These query tokens are computed once at every stage of the model and shared across all global attention blocks, hence decreasing number of parameters and FLOPs and improving the generalizability.

Original: In addition, the Global attention layers only learn local key and value features which will be used for interaction with global query tokens
Revised: In addition, the global attention layers only learn local key and value features which will be used for interaction with global query tokens.

Original: . 2. G LOBAL S ELF -A TTENTION
Revised: 2. G LOBAL S ELF -A TTENTION

Original: Fig.
Revised: Fig.

Original: 4 demonstrates the main idea of our contribution.
Revised: 4 demonstrates the main idea behind our contribution.

Original: Local self-attention can only query patches within a local window, whereas with global attentionattention can query  image globally while still operating in the window.
Revised: Local self-attention can only query patches within a local window, whereas the global attention can query different image regions while still operating within the window.

Original: The only difference in implementation is that query component is pre-computedin case of global attention and the rest is the same as  in local attention. In each stage, GC ViT employs alternating local and global self-attention blocks to effectively capture both local and global spatial information.
Revised: At each stage, the global query component is pre-computed as described in Sec.2.1.

Original: The global self-attention utilizes  global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with extracted local key and value features. Fig.
Revised: The global self-attention utilizes the extracted global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with the local key and value representations.

Original: 6 illustrates the difference between local and global self-attention. The global attention query q g has a size of B × C × h × w , wherein B , C , h and w denote batch size, embedding dimension, local window height and width, respectively. Moreover, q g [Figure]
Revised: In addition, GC ViT employs alternating local and global self-attention blocks to effectively capture

Original:  MSA
Revised: Local MSA

Original: La y e r N o r m
Revised: La y e r N o r m

Original: MLP MLP
Revised: MLP MLP

Original: MLP
Revised: MLP

Original: C on c a t
Revised: C on c a t

Original: P r o j e c t i on multi head
Revised: P r o j e c t i on multi head

Original: Global MSA
Revised: Global MSA

Original: La y e r N o r m
Revised: La y e r N o r m

Original: MLP
Revised: MLP

Original: MLP
Revised: MLP

Original: C on c a t
Revised: C on c a t

Original: P r o j e c t i on 
Revised: P r o j e c t i on global token both local and global spatial information.

Original: [Figure]
Revised: Fig.

Revised: 6 illustrates the difference between local and global self-attention.

Revised: The global attention query q g has a size of B × C × h × w , wherein B , C , h and w denote batch size, embedding dimension, local window height and width, respectively.

Original:  is repeated along the batch dimension to compensate for the overall number of windows and batch size B ∗ =
Revised: Moreover, q g is repeated along the batch dimension to compensate for the overall number of windows and batch size B ∗ =

Original: B × N where N is the number of local windows.
Revised: B × N where N is the number of local windows.

Original: q g is further reshaped into multiple head.
Revised: q g is further reshaped into multiple heads.

Original: The value and key are computed within each local window using a linear layer. 
Revised: The value and key are computed within each local window using a linear layer. The global self-attention query, key and value features are computed as follows [Equation] [Equation]

Original: # Input/output shape: (B*, N, C)
Revised: [Equation] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: # B*: Batchsize*Num Windows; H: Height; # W: Width; C: dim; q_g: Global Token; # F: Num Attention Head; N: Num Windows; def init():f=nn.
Revised: ; H: Height; # [Table] : Width; C

Revised: [Table]

Revised: [Table]

Revised: Token ; #

Revised: [Table]

Revised: [Table]

Revised: Windows; def

Revised: [Table]

Revised: [Table]

Revised: =

Revised: [Table]

Revised: .Linear

Revised: [Table]

Revised: [Table]

Original: Linear(C, 2*C) softmax = nn.Softmax(dim=-1) def forward(x,q_g): B*, N, C = x.shape
Revised: *C) softmax = nn. Softmax(dim= [Table] 1) def forward(x,

Revised: [Table]

Revised: g):

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  B, C, h, w = q_global.shape kv =
Revised: = [Table] .shape B, C, h, w = q_g.shape kv =

Original: f(x).reshape(B*, N, 2, F, C // 
Revised: f(x).reshape(B*, N, 2, F, C // F) kv =

Original: F) kv = kv.permute(2, 0, 3, 1, 4) k, v = split(kv, (1, 1), 0) q_g
Revised: kv.permute(2, 0, 3, 1, 4) k, v = split(kv, (1, 1), 0) q_g

Original: = q_g.repeat(1, B* // B, 1, 1) q_g
Revised: = q_g.repeat(1, B* // B, 1, 1) q_g

Original: = q_g.reshape(B*, F, N, C // F) qk =
Revised: = q_g.reshape(B*, F, N, C // F ) qk =

Original: matmul(q_g,k.transpose(-2, -1)) attn =
Revised: matmul(q_g,k.transpose(-2,-1)) attn =

Original: softmax(qk) return matmul(attn, v).reshape(B*, N, C)
Revised: softmax(qk) return matmul(attn, v).reshape(B*, N, C)

Original: The global self-attention query, key and value features are computed as follows

Original: Since the partitioned windows only contain local information, interaction with rich contextual information embedded in the global query tokens provides an effective way of enlarging the receptive field and attending to various regions in the input feature maps.
Revised: Since the partitioned windows only contain local information, interaction with rich contextual information embedded in the global query tokens provides an effective way of enlarging the receptive field and attending to various regions in the input feature maps.

Original: The self-attention module is computed as in 
Revised: The self-attention module is computed as in Attention

Original: [Equation]
Revised: [Equation]

Original: where d is scaling factor and b is a learnable relative position bias term.
Revised: where d is scaling factor and b is a learnable relative position bias term.

Original: Following previous efforts (Hu et al., 2018a; Liu et al., 2021), assuming position change between [ − p + 1 , p − 1] along horizontal and vertical axes, b is sampled from the grid ˆ b ∈ R (2 p − 1) × (2 p − 1) .
Revised: Assuming position change between [ − p + 1 , p − 1] along horizontal and vertical axes, b is sampled from the grid ˆ b ∈ R (2 p − 1) × (2 p − 1) .

Original: As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks.
Revised: As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks.

Original: In Algorithm 1, we present a PyTorch-like pseudocode for computing global self-attention in GC ViT. A complexity analysis of the global selfattention is presented in the supplementary materials.
Revised: In Algorithm 1, we present a PyTorch-like pseudocode for computing global selfattention in GC ViT. A complexity analysis of the global self-attention is presented in the supplementary materials.

Original: 2. GC V I T MODEL CONFIGURATIONS

Original: GC ViT model configurations are presented in Table 1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number.

Original: [Equation]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: ViT .
Revised: ViT .

Original: The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advantage of enlarged receptive field, due to its self-attention layers.
Revised: The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advantage of enlarged receptive field, due to its self-attention layers.

Original: However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance.
Revised: However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance.

Original: Data-efficient Image Transformers (DeiT) Touvron et al.
Revised: Data-efficient Image Transformers (DeiT) Touvron et al.

Original: (2021b)[Table] introduced a distillation-based training strategy which significantly improved the classification accuracy.
Revised: (2021b) introduced a distillation-based training strategy which significantly improved the classification accuracy.

Original: LeViT (Graham et al., 2021) proposed a hybrid model with re-designed multi-layer perceptron (MLP) and self-attention modules that are highly-optimized for fast inference.
Revised: LeViT (Graham et al., 2021) proposed a hybrid model with re-designed multi-layer perceptron (MLP) and self-attention modules that are highly-optimized for fast inference.

Original: Cross-covariance
Revised: Cross-covariance

Original: Image Transformer (XCiT) (Ali et al., 2021) introduced a transposed self-attention module for modeling the interactions of feature channels.
Revised: Image Transformer (XCiT) (Ali et al., 2021) introduced a transposed self-attention module for modeling the interactions of feature channels.

Original: Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs.
Revised: Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs.

Original: Conditional Position encoding Vision
Revised: Conditional Position encoding Vision

Original: Transformer (CPVT)
Revised: Transformer (CPVT)

Original: (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token.
Revised: (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token.

Original: Tokens-To-Token Vision Transformer (T2T-ViT)
Revised: Tokens-To-Token Vision Transformer (T2T-ViT)

Original: (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations.
Revised: (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations.

Original: Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the computational efficiency.
Revised: Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the computational efficiency.

Original: Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction.
Revised: Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction.

Original: Twins Transformer
Revised: Twins Transformer

Original: (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency.
Revised: (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency.

Original:  EdgeViT (Pan et al., 2022) introduced a lightweight ViT model, with global sparse attention and local aggregations and propagation modules for capturing short and long-range information. Please see the supplementary materials for further discussion.
Revised: Focal Transformer (Yang et al., 2021b) introduced the Focal self-attention to capture long-range spatial interactions. PVTv2 (Wang et al., 2022) improved performance and efficiency comparing to PVT (Wang et al., 2021) by introducing overlapping patch embedding, convolutional feed-forward network and linear attention. EdgeViT (Pan et al., 2022) introduced a lightweight ViT model, with global sparse attention and local aggregations and propagation modules for capturing short and long-range information

Original: ConvNet .
Revised: ConvNet .

Original: Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017;
Revised: Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017;

Original: He et al., 2016; Szegedy et al., 2016; Huang et al., 2017; Hu et al., 2018b) have dominated computer vision benchmarks with SOTA performance.
Revised: He et al., 2016; Szegedy et al., 2016; Huang et al., 2017; Hu et al., 2018) have dominated computer vision benchmarks with SOTA performance.

Original: Recently,  ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.
Revised: Recently, inspired by ViTs, ConvMixer (Trockman & Kolter, 2022) introduced a simple architecture with large-kernel depth-wise and point-wise convolutional layers and global pooling with competitive performance for classification. Furthermore, ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Image classification.
Revised: Image classification.

Original: For image classification, we trained and tested our model on ImageNet-1K dataset (Deng et al., 2009).
Revised: For image classification, we trained and tested our model on ImageNet-1K dataset (Deng et al., 2009).

Original: To allow for a fair comparison, all GC ViT variants are trained by following training configurations of previous efforts (Liu et al., 2021;
Revised: To allow for a fair comparison, all GC ViT variants are trained by following training configurations of previous efforts (Liu et al., 2021;

Original: Yang et al., 2021b;
Revised: Yang et al., 2021b;

Original: Chu et al., 2021a).
Revised: Chu et al., 2021a).

Original: Specifically, all models are trained  with the AdamW (Kingma & Ba, 2014) optimizer forepochs with an initial learning rate of 0 .
Revised: Specifically, all models are trained on 4 nodes ( 32 A100 GPUs) with the AdamW (Loshchilov & Hutter, 2017) optimizer for 300 epochs with an initial learning rate of 0 .

Original: 001 , weight decay of 0 . 05 , cosine decay scheduler andwarm-up and cool-down epochs, respectively.[Table] PVT [Table]
Revised: 001 , weight decay of 0 . 05 , cosine decay scheduler and 20 warm-up and cool-down epochs, respectively.

Original: Large

Original: [Table]

Original: [Table] [Table] DeiT [Table]
Revised: We use total batch sizes of 4096 for GC ViT-XXT, GC ViT-XT and GC ViT-T models and 1028 for all other variants.

Original: Base

Original: [Table] al [Table] 2021b
Revised: See supplementary materials for more training details.

Original: [Table]

Original: [Table]

Original: [Table]

Original: T2T

Original: [Table]

Original: ViT [Table] [Table] Object detection and semantic segmentation For object detection and instance segmentation, we trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using × 3 LR schedule with an initial learning rate of 0 . 0001 , a batch size of 16 and weight decay of 0 .
Revised: Object detection and semantic segmentation For object detection and instance segmentation, we trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using × 3 LR schedule with an initial learning rate of 0 . 0001 , a batch size of 16 and weight decay of 0 .

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2022), we compared against Tiny, Small and Base model variants using Cascade
Revised: (2022), we compared against Tiny, Small and Base model variants using Cascade

Original: Mask-RCNN but only compared against Tiny variant using Mask-RCNN.
Revised: Mask-RCNN but only compared against Tiny variant using Mask-RCNN.

Original: For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head.
Revised: For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head.

Original: Following previous efforts, we used a random crop size of 512 × 512 for the input images. [Table]
Revised: Following previous efforts, we used a random crop size of 512 × 512 for the input images. For fair assessment, we only compare against models with a pre-trained ImageNet-1K backbone.

Original: 4. C LASSIFICATION
Revised: 4. C LASSIFICATION

Original: [Table] We present the ImageNet-1K classification benchmarks in Table 2 and compare against CNN, ViT and MLP-based models across different model sizes.
Revised: We present the ImageNet-1K classification benchmarks in Table 1 and compare against CNN, ViTbased models across different model sizes.

Original: Our model achieves new SOTA benchmarks in all categories by a large margin.
Revised: Our model achieves new SOTA benchmarks in all categories.

Original: Specifically, the proposed GC ViT surpasses similar-sized counterpart models by +0.5% for GC ViT-XT (82.0%) compared to CPVT-SmallGAP{+T2T-ViT-
+}
Revised: Specifically, the proposed GC ViT surpasses similar-sized counterpart models by +0.5% for GC ViT-XT (82.0%) compared to {+T2T-ViT-
+}

Original: (Chu et al., 2021b) (81.5%), +1.2% for 
Revised: (Yuan et al., 2021) (81.5%), +0.7% for GC

Original: GC ViT-T (83.4%) over Focal-T (Yang et al., 2021b) (82.2%), +0.4% for GC ViT-S (83.9%) over Focal-S (Yang et al., 2021b) (83.5%) and +0.7% for GC ViT-B (84.5%) compared to FocalB (Yang et al., 2021b) (83.8%), respectively.
Revised: ViT-T (83.4%) over CSwin-T (Dong et al., 2022) (82.7%), +0.3% for GC ViT-S (83.9%) over CSwinS (Dong et al., 2022) (83.6%), +0.2% for GC ViT-B (84.4%) compared to CSwin-B (Dong et al., 2022) (84.2%) and +0.1% for GC ViT-L (84.6%) over CoAtNet-3 (Dai et al., 2021) (84.5%), respectively.

Original: In addition, our models demonstrate similar or better performance gap compared to other established benchmarks such as ConvNeXt-T (Liu et al., 2022).

Original: Furthermore, as shown in Fig. 1, GC ViT models have better or comparable computational efficiency in terms of number FLOPs over the competing counterpart models. 
Revised: Furthermore, as shown in Fig. 1, GC ViT models have better or comparable computational efficiency in terms of number FLOPs over the competing counterpart models. [Table] [Table]

Original: 4. D ETECTION
Revised: 4. D ETECTION

Original:  In Table 3, we present object detection and instance segmentation benchmarksusing on MS COCO dataset.
Revised: [Table] In Table 2, we present object detection and instance segmentation benchmarks on MS COCO dataset.

Original: Using a Mask-RCNN head,  model with pre-trained GC ViT-T (46.5/41.8) backbone outperforms counterparts with pre-trained ConvNeXt-T (Liu et al., 2022) (46.2/41.7) by +0.3 and +0.1 and Swin-T (Liu et al., 2021) (46.0/41.6) by +0.5 and +0.2 in terms of box AP and mask AP, respectively.
Revised: Using a Mask-RCNN head, the model with pre-trained GC ViTT (47.9/43.2) backbone outperforms counterparts with pre-trained ConvNeXt-T (Liu et al., 2022) (46.2/41.7) by +1.7 and +1.5 and SwinT (Liu et al., 2021) (46.0/41.6) by +1.9 and +1.6 in terms of box AP and mask AP, respectively.

Original: Using a Cascade Mask-RCNN head,  models with pre-trained GC ViT-T (51.1/44.4) and GC ViT-S (52.4/45.4) backbones surpass all competing counterparts.
Revised: Using a Cascade Mask-RCNN head, the models with pre-trained GC ViT-T (51.6/44.6) and GC ViT-S (52.4/45.4) backbones outperform ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +1.2 and +0.9 and

Original: Specifically, it outperforms ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +0.7 and +0.7 and ConvNeXt-S (Liu et al., 2022) (51.9/45.0) by +0.
Revised: ConvNeXt-S (Liu et al., 2022) (51.9/45.0) by +0.5 and +0.4 in terms of box AP and mask

Original: and +0.4 in terms of box AP and mask AP, respectively.
Revised: AP, respectively.

Original: However, ConvNeXt-B (Liu et al., 2022) (52.7/45.6)demonstrates better performance compared to GC ViT-B (52.2/45.2) by +0.5 and +0.4 in terms of box AP and mask AP, respectively.
Revised: Furthermore, the model with GC ViT-B (52.9/45.8) backbone outperforms the counterpart with ConvNeXt-B (Liu et al., 2022) (52.7/45.6) by +0.2 and +0.2 in terms of box AP and mask AP, respectively.

Original: 4. ADE20K S EMANTIC S EGMENTATION R ESULTS
Revised: 4. ADE20K S EMANTIC S EGMENTATION R ESULTS

Original: We presentthe semantic segmentation benchmarks on ADE20K  in Table 4.
Revised: We present semantic segmentation benchmarks on ADE20K dataset in Table 3.

Original: Models using pre-trained GC ViT-T (46.5), GC ViT-T (48.3)  backbones surpass competing models with pre-trained Twins-
Revised: The models using pretrained GC ViT-T (47.0), GC ViT-S (48.3) and GC ViT-B (49.0) backbones outperform counterpart models with pre-trained Twins-SVT-S

Original: [Figure]

Original: SVT-S (Chu et al., 2021a) (46.2) and Focal-S (Yang et al., 2021b) (48.0) by +0.3 and +0.3 in terms of mIoU, respectively.
Revised: (Chu et al., 2021a) (46.2), Twins-SVT-B (Chu et al., 2021a) (47.7) and Twins-SVT-L (Chu et al., 2021a) (48.8) by +0.8, +0.6 and +0.2 in terms of mIoU, respectively.

Original: Models pre-trained with GCViT-B (49.0) and Focal-B (Yang et al., 2021b) (49.0) both achieve the same segmentation performance. GC ViT  significantly outperforms Swin
Revised: In addition, models with GC ViT backbones significantly outperform counterparts with

Original:  Transformer counterparts, hence demonstrating the effectiveness of the global self-attention.
Revised: Swin Transformer backbones, hence demonstrating the effectiveness of the global self-attention.

Original:  A BLATION 
Revised:  A BLATION 5. C OMPONENT - WISE A NALYSIS

Original: [Table]
Revised: [Table]

Original: Component-wise Analysis. As shown in Table 5, replacing global self-attention blocks with  their local counterparts significantly impacts the performance.
Revised: As shown in Table 4, we study the role of each component in GC ViT model for classification, detection, instance and semantic segmentation. For simplicity, we start with Swin Transformer as the base model and progressively re-design the components to demonstrate their importance in improving the performance.

Original: Specifically, segmentation shows the most significant degradation.
Revised: Firstly, we remove the window shifting and predictably observe significant performance degradation across all tasks. Changing distribution of parameters to our design improves the performance by +1.7, +2.8, +2.2 and +1.7 in terms of accuracy, box AP, mask AP and mIoU. Such reparametrization includes changing the window size, MLP ratio, number of layers to name but a few. Adding the CNN-based stem of GC ViT to the model provides additional improvements of +0.3, +0.2, +0.2 and +0.2 in terms of accuracy, box AP, mask AP and mIoU. In addition, using our proposed downsampler further improves the accuracy, box AP, mask AP and mIoU by +0.4, +0.1, +0.1 and +0.3, respectively.

Revised: The last two changes demonstrate the importance of convolutional inductive bias and capturing the inter-channel dependencies in our model.

Original: Sharing global context query features consistently improves the performance  for all tasks, according to
Revised: Finally, leveraging the proposed global self-attention improves the performance by by +0.8, +0.8, +0.6 and +1.2 in terms of accuracy, box AP, mask AP and mIoU. Hence, this validates the effectiveness of the [Figure] proposed global self-attention, in particular for downstream tasks with high resolution images such as semantic segmentation in which modeling long-range spatial dependencies is critical.

Original: Table 5, and decreases the number of parameters and operations.
Revised: We studied the effectiveness of various downsampler blocks in Table 5.

Revised: The simplest alternative to our design is a pair of convolutional and maxpooling layers.

Original: Such improvements may imply that model generalizability improves by re-using the same tokens throughout the local and global attention blocks.
Revised: However, it results in a reduction of ImageNet Top-1 accuracy by -0.7. Patch merging is another variant which was introduced in Swin Transformers (Liu et al., 2021).

Revised: [Table]

Original: Our experiments also show the benefits of employing relative position bias in both local and global self-attention blocks.
Revised: However, it reduces the accuracy by -0.5. Finally, our down-sampler which consists of a modified Fused-MBConv block and strided convolution and shows the best result. Importance of the former component is explained by the SE operation which boosts cross channel interaction while keeping number of parameters and FLOPs low. We conclude that our proposed down-sampler is essential to achieve high accuracy as it introduces convolutional inductive bias.

Original:  I NTERPRETABILITY
Revised:  I NTERPRETABILITY

Original: To provide further insights on interpretability of the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017)maps in
Revised: To provide further insights on interpretability of the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017) maps in

Original: Fig.
Revised: Fig.

Original: The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act as an informative source for local attention modules. 
Revised: The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act as an informative source for local attention modules. In addition, corresponding Grad-CAM maps demonstrate accurate object localization with most intricate details.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced a novel hierarchical ViT, referred to as GC ViT, which can efficiently capture global context by utilizing global query tokens and interact with local regions. 
Revised: We introduced a new vision transformer architecture named GC ViT that can efficiently capture global context by utilizing global query tokens and interact with local regions. Through extensive experiments we show SOTA benchmarks for image classification on ImageNet-1K dataset, surpassing CNN/ViT-based counterparts by large margin.

Original: We have extensively validated the effectiveness of our model on various tasks.
Revised: We also consistently achieved SOTA for downstream tasks of detection, instance and semantic segmentation on MS COCO and ADE20K datasets.

Original: The proposed GC ViT model achieves new SOTA benchmarks for image classification across various model sizes on ImageNet-1K dataset, and surpasses both CNN and ViT-based counterparts by a significant margin.
Revised: GC ViT model configurations are presented in Table S.1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number [Table]

Original: We have also achieved SOTA or competitive performance for downstream tasks of detection and semantic segmentation on high-resolution images using MS COCO and ADE20K datasets.
Revised: We performed ablation studies to validate the effectiveness of the proposed global query. Using the same architecture, instead of global query, we compute: (1) global key and value features and interact them with local query (2) global value features and interact it with local query and key. As shown in

Revised: Table S.2, replacing global query may significantly impact the performance for image segmentation and downstream tasks such as object detection, instance segmentation and semantic segmentation

Revised: [Table]

Revised: C. EMA AND B ATCH S IZE

Revised: We also used used Exponential Moving Averages (EMA) and observed slight improvement in terms of ImageNet TOp-1 accuracy.

Revised: Furthermore, the performance of the model across different batch sizes were stable as we did not observe significant changes.

Revised: Table S.3 demonstrates the effect of EMA and batch size on the accuracy of a GCViT-T model

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: D I NTERPRETABILITY

Revised: In Fig.

Revised: S.1, we illustrate the learned global query token maps and demonstrate their effectiveness in capturing long-range contextual representations from different image regions.

Revised: E T RAINING D ETAILS

Revised: For image classification, GC ViT models were trained using four computational nodes with

Revised: NVIDIA A100 GPUs.

Revised: The total training batch size is 1024 ( 32 per GPU) for GC ViT-S, GC ViT-B,

Revised: GC ViT-L and 4096 ( 128 per GPU) for GC ViT-XXT, GC ViT-XT and GC ViT-T.

Revised: On average, each model required 32 hours of training with the specified hyper-parameters as indicated in the paper.

Revised: All classification models were trained using the timm package (Wightman, 2019).

Revised: Object detection and instance segmentation models as well as semantic segmentation models were trained using one computational node with 8 NVIDIA A40 GPUs using a total batch size of 16 , hence a batch size ofper GPU.

Revised: Detection and instance segmentation models were trained using mmdetection (Chen et al., 2019) package and on average required 56 hours of training.

Revised: Semantic segmentation models were trained using mmsegmentation (Contributors, 2020) package, and on average requiredhours of training.

Revised: F C OMPLEXITY A NALYSIS

Revised: Given an input feature map of x ∈ R H × W × C at each stage with a window size of h × w , the computational complexity of GC ViT is as follows

Revised: [Equation]

Revised: The efficient design of global query token generator and other components allows to maintain a similar computational complexity in comparison to Swin Transformer Liu et al.

Revised: (2021) while being able to capture long-range information and achieve better higher accuracy for classification and downstream tasks such as detection and segmentation.

Revised: G C OMPARISON TO O THER G LOBAL S ELF - ATTENTION M ODULES

Revised: Other efforts such as EdgeViT (Pan et al., 2022) in computer vision and BigBird (Zaheer et al., 2020) in NLP have proposed global self-attention in their respective applications.

Revised: In this section, we discuss the differences between the proposed global self-attention in GC ViT and these efforts.

Revised: [Table]

Revised: [Table]

Revised: EdgeViT : EdgeViT and GC ViT use completely different self-attention blocks.

Revised: The EdgeViT uses a series of local aggregation (convolution), sparse attention and local propagation(depthwise convolution), whereas GC ViT only uses an interleaved pattern of local and global self-attention layers without convolution in order to compute self-attention.

Revised: The proposed global sparse attention in

Revised: EdgeViT and GCViT are competently different.

Revised: EdgeViT samples representative tokens and only computes sparse self-attention between these representative tokens with reduced feature size.

Revised: On the contrary, GC ViT computes self-attention between the global queries (not just the token) and local keys and values without any subsampling in their respective local regions.

Revised: Furthermore, in EdgeViT, only subsampled representative tokens per region interact

Revised: In the self-attention module; however, in

Revised: GC ViT

Revised: [Table]

Revised: the global queries interact with the entire local regions, instead of interacting with each other

Revised: [Table]

Revised: and hence provide an effective mechanism for capturing both short and long-range spatial dependencies

Revised: [Table]

Revised: In addition, GC ViT generates global query tokens by using a series of modified Fused MB-Conv from the entire image and without subsampling.

Revised: Note that the resolution of global query tokens are the same as local query and values.

Revised: However, in EdgeViT: (A) the representative tokens are obtained per local window, not the entire image, and by subsampling and reducing the feature resolution.

Revised: Hence, since generated tokens have a lower resolution compared to their respective local windows, this could result in loss of spatial information and impact the effectiveness of self-attention.

Revised: Unlike

Revised: EdgeViT, the downsampler in GCViT also benefits from modified Fused-MBConv blocks which allows for modeling cross channel interactions and impose more locality and convolutional inductive bias.

Revised: BigBird : Bigbird, which is primarily introduced for NLP applications with 1D inputs, has significant differences compared to GC ViT, which is proposed for computer vision with mainly 2D inputs.

Revised: Firstly, BigBird uses a combination of random, window and global attention mechanisms, which is different from the proposed local and global self-attention scheme in GC ViT.

Revised: In addition,

Revised: BigBird does not have any specific mechanisms for extracting global tokens as the existing tokens or additional special tokens can be specified as global tokens.

Revised: On the contrary, the global tokens in

Revised: GC ViT are extracted by the proposed global query generator module which consists of a series of modified Fused MB-Conv blocks to extract contextual information from the entire input features.

Revised: Lastly, BigBird employs a set of global tokens which attend to the entire input sequence; in this case, select global query, key and values attend to local query, key and value tensors.

Revised: However, as opposed to this formulation, in GC ViT, the global query tokens attend to local key and value tokens in partitioned windows.

Revised: This is due to the fact that attending to the entire input sequence, as done in

Revised: BigBird, is not feasible considering the larger size of input features in computer vision.

Revised: H I MAGE N ET V2 B ENCHMARKS

Revised: In Table S.4, we have evaluated the performance of GC ViT on ImageNetV2 dataset ( ? ) to further measure its robustness.

Revised: Specifically, we have used different sampling strategies of Matched Frequency and Threshold-0.7.

Revised: These benchmarks demonstrate the competetive performance of GC ViT on

Revised: ImageNetV2 dataset and validates its effectiveness in robustness and generalizability.

Revised: I E FFECT OF G LOBAL C ONTEXT M ODULE

Revised: In order to demonstrate the effectiveness of Global Context (GC) module, we use Swin Transformers as the base model and add our propoped GC module.

Revised: In this analysis, we remove the window shifting operation from Swin Transformers, since GC module is capable of modeling cross-region interactions.

Revised: As shown in Table S.5, addition of GC module improves the ImageNet Top-1 accuracy by +0 . 9% and +0 .

Revised: 7% for Swin Transformers Tiny and Small variants respectively.

Revised: J I MAGE N ET CLASSIFICATION B ENCHMARKS

Revised: In Table S.6, we provide a comprehensive benchmark in terms of Top-1 accuracy for the models that are only trained on ImageNet-1K (Deng et al., 2009) dataset, and without additional data.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Medium

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: S

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: B

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Focal

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: GC

Revised: [Table]

Revised: S

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Dosovitskiy

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: B

Revised: [Table]

Revised: Touvron

Revised: [Table]

Revised: [Table]

Revised: 2021a

Revised: [Table]

Revised: [Table]

Revised: Large (Wang et al., 2021)

Revised: [Table]

Revised: Base/16 (Touvron et al., 2021b)

Revised: CrossViT-B (Chen et al., 2021) 21.

Revised: T2T-ViT-24 (Yuan et al., 2021) 14.

Revised: CPVT-B (Chu et al., 2021b) 17.

Revised: Twins-PCPVT-L (Chu et al., 2021a) 9.

Revised: Swin-B (Liu et al., 2021) 15.

Revised: CoAtNet-2 (Dai et al., 2021) 8.

Revised: PVT-v2-B4 (Wang et al., 2022) 10.

Revised: Twins-SVT-L (Chu et al., 2021a) 15.

Revised: ConvNeXt-B (Liu et al., 2022) 15.

Revised: Focal-B (Yang et al., 2021b) 16.

Revised: PVT-v2-B5 (Wang et al., 2022) 11.

Revised: CSwin-B (Dong et al., 2022) 15.

Revised: BoTNet (Dong et al., 2022) 19.

Revised: GC ViT-B 14.

Revised: ConvNeXt-L (Liu et al., 2022) 34.

Revised: CoAtNet-3 (Dai et al., 2021) 34.

Revised: GC ViT-L 32.