Original: G LOBAL C ONTEXT V ISION T RANSFORMERS
Revised: G LOBAL C ONTEXT V ISION T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision tasks.
Revised: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision.

Original: The core of the novel model are global context self-attention modules, joint with standard local self-attention, to effectively yet efficiently model both long and short-range spatial interactions, as an alternative to complex operations such as an attention masks or  local windows shifting.
Revised: Our method leverages global context self-attention modules, joint with standard local selfattention, to effectively yet efficiently model both long and short-range spatial interactions, without the need for expensive operations such as computing attention masks or shifting local windows.

Original: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values.

Original: In addition, we address the lack of  inductive bias in ViTs and improve the modeling of inter-channel dependencies by proposing a novel downsampler which leverages a parameter-efficient fused inverted residual block.
Revised: In addition, we address the lack of the inductive bias in ViTs, and propose to leverage a modified fused inverted residual blocks in our architecture.

Original: The proposed GC ViT achievesnew state-of-the-art performance across image classification, object detection and semantic segmentation tasks.
Revised: Our proposed GC ViT achieves state-of-the-art results across image classification, object detection and semantic segmentation tasks.

Original: On ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28 M, 51 M and 90 M parameters achieve 83.4% , 83.9% and 84.4% Top-1 accuracy, respectively, surpassing comparablysized prior art such as CNN-based ConvNeXt and ViT-based Swin Transformer.
Revised: On ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28 M, 51 M and 90 M parameters achieve 83.4% , 83.9% and 84.5% Topaccuracy, respectively, surpassing comparably-sized prior art such as CNN-based ConvNeXt and ViT-based Swin Transformer by a large margin.

Original: Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation on MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins. 
Revised: Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins. Code and pre-trained models available at https://anonymous.4open.science/r/GCViTs-8852/.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: During the recent years, Transformers (Vaswani et al., 2017) have achieved State-Of-The-Art (SOTA) performance in Natural Language Processing (NLP) benchmarks and became the de facto model for various tasks.
Revised: During the recent years, Transformers (Vaswani et al., 2017) have achieved State-Of-The-Art (SOTA) performance in Natural Language Processing (NLP) benchmarks and became the de facto model for various tasks.

Original: A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021).
Revised: A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021).

Original: Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in a monolithic architecture with minor differences comparing to encoder of the original Transformer.
Revised: Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in a monolithic architecture with minor differences comparing to encoder of the original Transformer.

Original: Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA or competitive performance in various computer vision tasks.
Revised: Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA or competitive performance in various computer vision tasks.

Original: In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison to CNN.
Revised: In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison to CNN.

Original: However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) in which capturing multi-scale long-range information is crucial for accurate representation modeling.
Revised: However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) in which capturing multi-scale long-range information is crucial for accurate representation modeling.

Original: Several efforts (Liu et al., 2021; Dong et al., 2022; Chu et al., 2021a;
Revised: Several efforts (Liu et al., 2021; Dong et al., 2021; Chu et al., 2021a;

Original: Tu et al., 2022), most notably
Revised: Tu et al., 2022), most notably

Original: Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and longrange spatial dependencies by proposing multi-resolution architectures in which the self-attention is computed in local windows. In this paradigm, cross-window connections such as window shifting are used for modeling the interactions across different regions.
Revised: Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and longrange spatial dependencies by proposing multi-resolution architectures in which the self-attention is computed in local windows and cross-window connections such as window shifting are used for modeling the interactions across different regions.

Original: Despite the progress, the limited receptive field of local windows challenges the capability of self-attention to capture long-range information, and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window.
Revised: Despite this progress, the limited receptive field of local windows baffles the capability of self-attention to capture long-range information, and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window.

Original: Subsequent efforts such as Focal Transformer (Yang et al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.
Revised: Subsequent efforts such as Focal Transformer (Yang et al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.

Revised: The burden of computationally expensive operations such as token unfolding and rolling and additional computations of key and values can significantly affect the run-time and throughput.

Original: In this work, we introduce the Global Context (GC) ViT network to address these limitations.
Revised: In this work, we introduce the Global Context (GC) ViT network to address these limitations.

Original: Specifically, we propose a hierarchical ViT architecture consisting of local and global self-attention modules.
Revised: Specifically, we propose a hierarchical ViT architecture consisting of local and global self-attention modules.

Original: At each stage, we compute global query tokens, using a novel fused inverted residualblocks,
Revised: At each stage, we compute global query tokens, using modified fused inverted residual

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: Swin
Revised: Swin

Original: [Figure]
Revised: [Figure]

Original: ConvNext-T
Revised: T ConvNext

Revised: [Figure]

Original: ConvNext-S ConvNext
Revised: S

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: DeiT-S
Revised: DeiT-S

Original: DeiT
Revised: DeiT

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: PVT-S
Revised: PVT-S

Original: PVT-M
Revised: PVT-M

Original: PVT-L
Revised: PVT-L

Original: Focal-T
Revised: Focal-T

Original: Focal-S
Revised: Focal-S

Original: Focal-B
Revised: Focal [Figure] B

Original: GC ViT-XT
Revised: GC ViT-XT

Original: GC ViT-T
Revised: GC ViT-T

Original: GC ViT-S GC ViT-B
Revised: GC ViT-S GC ViT

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Swin
Revised: ConvNext

Original: [Figure]
Revised: [Figure]

Original: ConvNext-T
Revised: T ConvNext

Revised: [Figure]

Original: ConvNext-S ConvNext-B
Revised: S ConvNext [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: DeiT-S
Revised: DeiT [Figure] S

Original: DeiT-
Revised: DeiT

Original: [Figure]
Revised: [Figure]

Original: PVT-S
Revised: PVT-S

Original: PVT-M
Revised: PVT-M

Original: PVT-L
Revised: PVT-L

Original: Focal-T
Revised: Focal-T

Original: Focal-S Focal-B
Revised: Focal-S Focal

Revised: [Figure]

Original: GC ViT-XT
Revised: GC ViT-XT

Original: GC ViT-T GC ViT-S
Revised: GC ViT-T GC ViT-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L P
Revised: P

Original: M L P

Original: D o w n s a m p l e
Revised: D o w n s a m p l e

Original: Lo c a l M SA
Revised: Lo c a l M SA

Original: Global Token Gen.
Revised: Global Token Gen.

Original: G l oba l M
Revised: G l oba l M

Original: SA M L P
Revised: SA M L P

Original: M L P
Revised: M L P

Original: Lo c a l M SA
Revised: Lo c a l M SA

Original: Global Token Gen.
Revised: Global Token Gen.

Original: G l oba l M
Revised: G l oba l M

Original: SA M L P 2 D
Revised: SA M L P 2 D

Original: A v g P oo l
Revised: A v g P oo l

Original: H ead D
Revised: H ead D

Original: o w n s a m p l e
Revised: o w n s a m p l e

Original: M L P  which we refer to asmodified Fused-MBConv blocks, that encompass global contextual information from different image regions.
Revised: M L P blocks, which we refer to as Fused-MBConv blocks, that encompass global contextual information from different image regions.

Original: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and value representations.
Revised: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values.

Revised: As we demonstrate in Sec. 5, sharing global queries allow for reducing number of parameters and FLOPs and increasing generalizability.

Original: The design of our proposed framework for global query generator and self-attention is intuitive and simple and can be efficiently implemented using major deep learning framework.
Revised: The design of our proposed framework for global query generation and self-attention is intuitive and simple and can be efficiently implemented using major deep learning framework.

Original: Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied to high-resolution images.
Revised: Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied to high-resolution images.

Original: In addition, we propose a novel downsampling block with a parameter-efficient fused-MBConv layer to address the lack of inductive bias in ViTs and enhancing the modeling of inter-channel dependencies.

Original: We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks.
Revised: We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks.

Original: For image classification using ImageNet-1K dataset,GC
Revised: For image classification using ImageNet-1K dataset,

Original:  ViT with 28 M, 51M, 90 M and 201 M parameters, referred to as tiny, small, base and large variants, achieve new SOTA benchmarks of 83.4% , 83.9%, 84.4% and 84.6% Top-1 accuracy.
Revised: GC ViT with 28 M, 51 M and 90 M parameters, referred to as tiny, small and base variants, achieve new SOTA benchmarks of 83.4% , 83.9% and 84.5% Top-1 accuracy.

Original: Hence, GC ViT consistently outperforms both ConvNeXt (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1).
Revised: Hence, GC ViT consistently outperforms both ConvNeXt (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1).

Original: Using a pre-trained GC ViT base backbone witha Cascade Mask RCNN (He et al., 2017) head, our model achieves a box mAP of 52.9 for object detection and a mask mAP of 45.8 for instance segmentation on the MS COCO dataset.
Revised: Using a pre-trained GC ViT base backbone with Cascade Mask RCNN (He et al., 2017) head, our model achieves a box mAP of 52.2 for object detection and a mask mAP of 45.2 for instance segmentation on the MS COCO test-dev.

Original: In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves a mIoU of 49.0 on ADE20K for semantic segmentation.
Revised: In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves a mIoU of 49.0 on ADE20K for semantic segmentation.

Original: Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sized models on both MS COCO and ADE20K datasets.
Revised: Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sized models on both MS COCO and ADE20K datasets.

Original: Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.
Revised: Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.

Original: The main contributions of our work are summarized as follows:
Revised: The main contributions of our work are summarized as follows:

Original: • We introduce a compute and parameter-optimized hierarchical ViT with reparametrization of the design space ( e.g. , embedding dimension, number of heads, MLP ratio). • We design an efficient CNN-like token generator that encodes spatial features at different resolutions for  globalquery representations. • We propose global query tokens that can effectively capture contextual informationin an efficient manner and model both local and global interactions.
Revised: • A novel hierarchical Transformer model, called GC ViT, that can be employed as a general backbone in various computer vision tasks such as classification, detection, instance segmentation down-stream tasks such as detection, instance and semantic segmentation. • A novel and yet simple design comprising of global self-attention and token generation modules that allow for modeling long-range dependencies by capturing global contextual information and hence eliminating the need for highly sophisticated or complex operations.

Original: • We introduce a parameter-efficient downsampling module with modified Fused MB-Conv blocks that not only integrates inductive bias but also enables the modeling of inter-channel dependencies.
Revised: • As demonstrated in Fig.

Original: • We demonstrate new SOTA benchmarksfor : (1) ImageNet classification with Pareto fronts on ImageNet-1K  for  model size and FLOPs (see Fig. 1), and (2) downstream tasks such as detection, instance segmentation and semantic segmentation on MS COCO and ADE20K, respectively.
Revised: 1, the proposed GC ViT achieves new SOTA benchmarks on ImageNet-1K dataset for a variety of model sizes and FLOPs, outperforming both CNN and ViT-based models by a significant margin. Using GC ViT as the backbone yields SOTA or competitive performance for object detection and semantic segmentation on MS COCO and ADE20K datasets, respectively.

Original:  GC V I T ARCHITECTURE
Revised:  GC V I T ARCHITECTURE

Original: Architecture.
Revised: Architecture.

Original: Fig.
Revised: Fig.

Original: 2 depicts the architecture of GC ViT. 
Revised: 2 depicts the architecture of GC ViT. Similar to previous efforts (Liu et al., 2021;

Original: We propose a hierarchical framework to obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension, both by factors of 2 .
Revised: Yang et al., 2021b), we use a hierarchical framework to obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension by factors of 2 and 2 , respectively.

Original: At first, given an input image with resolution of x ∈ R H × W × 3 ,  we obtain overlapping patches by applying a 3 ×convolutional layer with a stride of 2 and appropriate padding.
Revised: At first, given an input image with resolution of x ∈ R H × W × 3 , following Wang et al. (2022), we obtain overlapping patches by applying a 3 ×convolutional layer with a stride of 2 and appropriate padding.

Original: Then patches are projected  into a C -dimensional embedding space with another 3 × 3 convolutional layer with stride 2 .
Revised: Then patches are projected it into a C -dimensional embedding space. We show the benefit of using overlapping as opposed to non-overlapping patches in Sec. 5.

Original: Every GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features.
Revised: Every GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features.

Original: Both operate in local windows like Swin Transformer (Liu et al., 2021), however, the global self-attention has access to global features extracted by the global token generator.
Revised: Both operate in local windows like Swin Transformer (Liu et al., 2021), however, the global self-attention accesses global features extracted by the Global Token Generator (GTG).

Original: The token generator is a CNN-like module that extracts features from the entire image only once at every stage.
Revised: The GTG is a CNN-like module that extracts features from the entire image only once at every stage.

Original: After each stage, the spatial resolution is decreased by 2 while  the number of channelsis increased by2 via a downsampling block.
Revised: The spatial resolution is decreased by 2 while increasing the number of channels by a downsampling block after every stage.

Original: Resulting features are passed through average pooling and linear layers to create an embedding for a downstream task.
Revised: Resulting features are passed through average pooling and linear layers to create an embedding for a downstream task.

Original: The GC ViT architecture benefits from novel blocks such as a downsampling operator , a global query generator and a global self-attention module described in the next sections. 
Revised: The GC ViT architecture relies on novel blocks such as a downsampling operator, a global query generator and a global self-attention module described in the next sections. Downsamplingblock

Original: [Figure]
Revised: F u s ed

Original: [Figure]
Revised: M B C on v C on v 3 x 3 , s t r i de La y e r N o r m

Original: Downsampler.
Revised: Downsampling.

Original: We leverage an idea of spatial feature contraction from
Revised: We borrow an idea of spatial feature contraction from

Original: CNN models that imposes locality bias and cross channel interaction while reducing dimensions.
Revised: CNN models that imposes locality bias and cross channel communication while reducing dimensions.

Original: We utilize a modified Fused-MBConv block, followed by a max pooling layer with a kernel size of 3 and stride ofas a downsampling operator, see Fig 3.
Revised: We use a modified Fused-MBConv block, followed by a max pooling layer with a kernel size of 3 and stride ofas a downsampling operator, see Fig 3.

Original: The Fused-MBConv block in our work is similar to the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in
Revised: The Fused-MBConv block in our work is similar to the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in

Original: [Equation]
Revised: [Equation]

Original: where SE, GELU and DW-Conv 3 × 3 denote Squeeze and Excitation block (Hu et al., 2018), Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016) and 3 × 3 depth-wise convolution, respectively.
Revised: where SE, GELU and DW-Conv 3 × 3 denote Squeeze and Excitation block (Hu et al., 2018b), Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016) and 3 ×depth-wise convolution, respectively.

Original: In our proposed architecture, the Fused-MBConv blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies.
Revised: In our proposed architecture, the Fused-MBConv blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies.

Original: It is ablated in Table 5.

Original: Attention.
Revised: Attention.

Original: Multi-head self-attention is the the core computational operator in the GC ViT architecture, it extracts semantic information from the image.
Revised: Multi-head self-attention is the the core computational operator in the GC ViT architecture to extract semantic information from the image.

Original: GC ViT is composed of local and global selfattention modules illustrated in Fig 4.
Revised: GC ViT is composed of local and global selfattention modules illustrated in Fig 4.

Original: Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size.
Revised: Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size.

Original: The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross-patch communication with those far beyond the local window.
Revised: The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross-patch communication with those far beyond the local window.

Original: Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module.[Figure] [Figure]
Revised: Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module.

Original: 2. G LOBAL Q UERY G ENERATOR
Revised: 2. G LOBAL Q UERY G ENERATOR

Original: We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value feature pairs.
Revised: We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value features.

Original: Specifically, as shown in Fig.
Revised: Specifically, as shown in Fig.

Original: 5, a layer f in the the generator consists of a Fused-MBConv block followed by a max pooling layer, similar to the one described in Sec.
Revised: 5, a layer f in [Figure]

Original: [Figure]
Revised: [Figure]

Original:  and the final global query q g , i at stage i ( i ∈ { 1 , 2 , 3 , 4 } ) of GC ViT is computed according to
Revised: the the generator consists of a Fused-MBConv block followed by a max pooling layer, similar to the one described in Sec. 2, and the final global query q g , i at stage i ( i ∈ { 1 , 2 , 3 , 4 } ) of GC ViT is computed according to

Original: [Equation]
Revised: [Equation]

Original: These query tokens are computed once at every stage of the model and shared across all global attention blocks, hence decreasing number of parameters and FLOPs and improving the generalizability.
Revised: These query tokens are computed once at every stage of GC ViT model and shared across all global attention blocks, hence decreasing number of parameters and FLOPs and improving the generalizability.

Original: In addition, the global attention layers only learn local key and value features which will be used for interaction with global query tokens.
Revised: In addition, the Global attention layers only learn local key and value features which will be used for interaction with global query tokens

Original:  2. G LOBAL S ELF -A TTENTION
Revised: . 2. G LOBAL S ELF -A TTENTION

Original: Fig.
Revised: Fig.

Original: 4 demonstrates the main idea behind our contribution.
Revised: 4 demonstrates the main idea of our contribution.

Original: Local self-attention can only query patches within a local window, whereas the global attention  can querydifferent image regions while still operating within the window.
Revised: Local self-attention can only query patches within a local window, whereas with global attention attention can query image globally while still operating in the window.

Original: At each stage, the global query component is pre-computed  asdescribed in Sec.2.1.
Revised: The only difference in implementation is that query component is pre-computed in case of global attention and the rest is the same as in local attention. In each stage, GC ViT employs alternating local and global self-attention blocks to effectively capture both local and global spatial information.

Original: The global self-attention utilizesthe extracted global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with the local key and value representations.
Revised: The global self-attention utilizes global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with extracted local key and value features. Fig.

Original: In addition, GC ViT employs alternating local and global self-attention blocks to effectively capture
Revised: 6 illustrates the difference between local and global self-attention. The global attention query q g has a size of B × C × h × w , wherein B , C , h and w denote batch size, embedding dimension, local window height and width, respectively. Moreover, q g

Original: Local MSA
Revised: [Figure] MSA

Original: La y e r N o r m
Revised: La y e r N o r m

Original: MLP MLP
Revised: MLP MLP

Original: MLP
Revised: MLP

Original: C on c a t
Revised: C on c a t

Original: P r o j e c t i on multi head
Revised: P r o j e c t i on multi head

Original: Global MSA
Revised: Global MSA

Original: La y e r N o r m
Revised: La y e r N o r m

Original: MLP
Revised: MLP

Original: MLP
Revised: MLP

Original: C on c a t
Revised: C on c a t

Original: P r o j e c t i onglobal token both local and global spatial information.
Revised: P r o j e c t i on

Original: Fig.
Revised: [Figure]

Original: 6 illustrates the difference between local and global self-attention.

Original: The global attention query q g has a size of B × C × h × w , wherein B , C , h and w denote batch size, embedding dimension, local window height and width, respectively.

Original: Moreover, q g is repeated along the batch dimension to compensate for the overall number of windows and batch size B ∗ =
Revised: is repeated along the batch dimension to compensate for the overall number of windows and batch size B ∗ =

Original: B × N where N is the number of local windows.
Revised: B × N where N is the number of local windows.

Original: q g is further reshaped into multiple heads.
Revised: q g is further reshaped into multiple head.

Original: The value and key are computed within each local window using a linear layer.
Revised: The value and key are computed within each local window using a linear layer.

Original: The global self-attention query, key and value features are computed as follows [Equation] [Equation] [Equation]
Revised: # Input/output shape: (B*, N, C)

Original: [Table] [Table] [Table] [Table]
Revised: # B*: Batchsize*Num Windows; H: Height; # W: Width; C: dim; q_g: Global Token; # F: Num Attention Head; N: Num Windows; def init():f=nn.

Original: [Table] [Table] [Table] [Table]
Revised: Linear(C, 2*C) softmax = nn.Softmax(dim=-1) def forward(x, q_g): B*, N, C = x.shape

Original: ; H: Height; # [Table] : Width; C
Revised: B, C, h, w = q_global.shape kv =

Original: [Table] [Table] Token ; #
Revised: f(x).reshape(B*, N, 2, F, C //

Original: [Table] [Table] Windows; def [Table]
Revised: F) kv = kv.permute(2, 0, 3, 1, 4) k, v = split(kv, (1, 1), 0) q_g

Original: [Table] = [Table] .Linear
Revised: = q_g.repeat(1, B* // B, 1, 1) q_g

Original: [Table] [Table] *C) softmax = nn. Softmax(dim=
Revised: = q_g.reshape(B*, F, N, C // F) qk =

Original: [Table]

Original: 1) def forward(x, [Table] g): [Table]
Revised: matmul(q_g,k.transpose(-2, -1))

Original: [Table]

Original: [Table]

Original: =

Original: [Table]

Original: .shape

Original: B, C, h, w = q_g.shape kv =

Original: f(x).reshape(B*, N, 2, F, C // F) kv =

Original: kv.permute(2, 0, 3, 1, 4) k, v = split(kv, (1, 1), 0) q_g = q_g.repeat(1, B* // B, 1, 1) q_g = q_g.reshape(B*, F, N, C // F ) qk = matmul(q_g,k.transpose(-2,-1)) attn =
Revised: attn =

Original: softmax(qk) return matmul(attn, v).reshape(B*, N, C)
Revised: softmax(qk) return matmul(attn, v).reshape(B*, N, C)

Revised: The global self-attention query, key and value features are computed as follows

Original: Since the partitioned windows only contain local information, interaction with rich contextual information embedded in the global query tokens provides an effective way of enlarging the receptive field and attending to various regions in the input feature maps.
Revised: Since the partitioned windows only contain local information, interaction with rich contextual information embedded in the global query tokens provides an effective way of enlarging the receptive field and attending to various regions in the input feature maps.

Original: The self-attention module is computed as in
Revised: The self-attention module is computed as in

Original: Attention

Original: [Equation]
Revised: [Equation]

Original: where d is scaling factor and b is a learnable relative position bias term.
Revised: where d is scaling factor and b is a learnable relative position bias term.

Original: Assuming position change between [ − p + 1 , p − 1] along horizontal and vertical axes, b is sampled from the grid ˆ b ∈ R (2 p − 1) × (2 p − 1) .
Revised: Following previous efforts (Hu et al., 2018a; Liu et al., 2021), assuming position change between [ − p + 1 , p − 1] along horizontal and vertical axes, b is sampled from the grid ˆ b ∈ R (2 p − 1) × (2 p − 1) .

Original: As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks.
Revised: As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks.

Original: In Algorithm 1, we present a PyTorch-like pseudocode for computing global selfattention in GC ViT. A complexity analysis of the global self-attention is presented in the supplementary materials. 
Revised: In Algorithm 1, we present a PyTorch-like pseudocode for computing global self-attention in GC ViT. A complexity analysis of the global selfattention is presented in the supplementary materials. 2. GC V I T MODEL CONFIGURATIONS

Revised: GC ViT model configurations are presented in Table 1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number.

Revised: [Equation]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: ViT .
Revised: ViT .

Original: The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advantage of enlarged receptive field, due to its self-attention layers.
Revised: The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advantage of enlarged receptive field, due to its self-attention layers.

Original: However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance.
Revised: However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance.

Original: Data-efficient Image Transformers (DeiT) Touvron et al.
Revised: Data-efficient Image Transformers (DeiT) Touvron et al.

Original: (2021b)  introduced a distillation-based training strategy which significantly improved the classification accuracy.
Revised: (2021b) [Table] introduced a distillation-based training strategy which significantly improved the classification accuracy.

Original: LeViT (Graham et al., 2021) proposed a hybrid model with re-designed multi-layer perceptron (MLP) and self-attention modules that are highly-optimized for fast inference.
Revised: LeViT (Graham et al., 2021) proposed a hybrid model with re-designed multi-layer perceptron (MLP) and self-attention modules that are highly-optimized for fast inference.

Original: Cross-covariance
Revised: Cross-covariance

Original: Image Transformer (XCiT) (Ali et al., 2021) introduced a transposed self-attention module for modeling the interactions of feature channels.
Revised: Image Transformer (XCiT) (Ali et al., 2021) introduced a transposed self-attention module for modeling the interactions of feature channels.

Original: Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs.
Revised: Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs.

Original: Conditional Position encoding Vision
Revised: Conditional Position encoding Vision

Original: Transformer (CPVT)
Revised: Transformer (CPVT)

Original: (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token.
Revised: (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token.

Original: Tokens-To-Token Vision Transformer (T2T-ViT)
Revised: Tokens-To-Token Vision Transformer (T2T-ViT)

Original: (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations.
Revised: (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations.

Original: Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the computational efficiency.
Revised: Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the computational efficiency.

Original: Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction.
Revised: Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction.

Original: Twins Transformer
Revised: Twins Transformer

Original: (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency.
Revised: (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency.

Original: Focal Transformer (Yang et al., 2021b) introduced the Focal self-attention to capture long-range spatial interactions. PVTv2 (Wang et al., 2022) improved performance and efficiency comparing to PVT (Wang et al., 2021) by introducing overlapping patch embedding, convolutional feed-forward network and linear attention.
Revised: EdgeViT (Pan et al., 2022) introduced a lightweight ViT model, with global sparse attention and local aggregations and propagation modules for capturing short and long-range information. Please see the supplementary materials for further discussion.

Original: ConvNet .
Revised: ConvNet .

Original: Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017;
Revised: Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017;

Original: He et al., 2016; Szegedy et al., 2016; Huang et al., 2017; Hu et al., 2018) have dominated computer vision benchmarks with SOTA performance.
Revised: He et al., 2016; Szegedy et al., 2016; Huang et al., 2017; Hu et al., 2018b) have dominated computer vision benchmarks with SOTA performance.

Original: Recently, inspired by ViTs, ConvMixer (Trockman & Kolter, 2022) introduced a simple architecture with large-kernel depth-wise and point-wise convolutional layers and global pooling with competitive performance for classification.

Original: Furthermore, ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.
Revised: Recently, ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Image classification.
Revised: Image classification.

Original: For image classification, we trained and tested our model on ImageNet-1K dataset (Deng et al., 2009).
Revised: For image classification, we trained and tested our model on ImageNet-1K dataset (Deng et al., 2009).

Original: To allow for a fair comparison, all GC ViT variants are trained by following training configurations of previous efforts (Liu et al., 2021;
Revised: To allow for a fair comparison, all GC ViT variants are trained by following training configurations of previous efforts (Liu et al., 2021;

Original: Yang et al., 2021b;
Revised: Yang et al., 2021b;

Original: Chu et al., 2021a).
Revised: Chu et al., 2021a).

Original: Specifically, all models are trainedon 4 nodes ( 32 A100 GPUs) with the AdamW (Loshchilov & Hutter, 2017) optimizer for 300 epochs with an initial learning rate of 0 .
Revised: Specifically, all models are trained with the AdamW (Kingma & Ba, 2014) optimizer forepochs with an initial learning rate of 0 .

Original: 001 , weight decay of 0 . 05 , cosine decay scheduler and 20 warm-up and cool-down epochs, respectively. 
Revised: 001 , weight decay of 0 . 05 , cosine decay scheduler andwarm-up and cool-down epochs, respectively. [Table] PVT [Table]

Revised: Large

Revised: [Table]

Original: We use total batch sizes of 4096 for GC ViT-XXT, GC ViT-XT and GC ViT-T models and 1028 for all other variants.
Revised: [Table] [Table] DeiT [Table]

Revised: Base

Original: See supplementary materials for more training details.
Revised: [Table] al [Table] 2021b

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: T2T

Revised: [Table]

Original:  Object detection and semantic segmentation For object detection and instance segmentation, we trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using × 3 LR schedule with an initial learning rate of 0 . 0001 , a batch size of 16 and weight decay of 0 .
Revised: ViT [Table] [Table] Object detection and semantic segmentation For object detection and instance segmentation, we trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using × 3 LR schedule with an initial learning rate of 0 . 0001 , a batch size of 16 and weight decay of 0 .

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2022), we compared against Tiny, Small and Base model variants using Cascade
Revised: (2022), we compared against Tiny, Small and Base model variants using Cascade

Original: Mask-RCNN but only compared against Tiny variant using Mask-RCNN.
Revised: Mask-RCNN but only compared against Tiny variant using Mask-RCNN.

Original: For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head.
Revised: For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head.

Original: Following previous efforts, we used a random crop size of 512 × 512 for the input images.
Revised: Following previous efforts, we used a random crop size of 512 × 512 for the input images.

Original: For fair assessment, we only compare against models with a pre-trained ImageNet-1K backbone.
Revised: [Table]

Original: 4. C LASSIFICATION
Revised: 4. C LASSIFICATION

Original:  We present the ImageNet-1K classification benchmarks in Table 1 and compare against CNN, ViTbased models across different model sizes.
Revised: [Table] We present the ImageNet-1K classification benchmarks in Table 2 and compare against CNN, ViT and MLP-based models across different model sizes.

Original: Our model achieves new SOTA benchmarks in all categories.
Revised: Our model achieves new SOTA benchmarks in all categories by a large margin.

Original: Specifically, the proposed GC ViT surpasses similar-sized counterpart models by +0.5% for GC ViT-XT (82.0%) compared to [-T2T-ViT-
-]
Revised: Specifically, the proposed GC ViT surpasses similar-sized counterpart models by +0.5% for GC ViT-XT (82.0%) compared to [-T2T-ViT-
-]CPVT-SmallGAP

Original: (Yuan et al., 2021) (81.5%), +0.7% forGC
Revised: (Chu et al., 2021b) (81.5%), +1.2% for

Original:  ViT-T (83.4%) over CSwin-T (Dong et al., 2022) (82.7%), +0.3% for GC ViT-S (83.9%) over CSwinS (Dong et al., 2022) (83.6%), +0.2% for GC ViT-B (84.4%) compared to CSwin-B (Dong et al., 2022) (84.2%) and +0.1% for GC ViT-L (84.6%) over CoAtNet-3 (Dai et al., 2021) (84.5%), respectively.
Revised: GC ViT-T (83.4%) over Focal-T (Yang et al., 2021b) (82.2%), +0.4% for GC ViT-S (83.9%) over Focal-S (Yang et al., 2021b) (83.5%) and +0.7% for GC ViT-B (84.5%) compared to FocalB (Yang et al., 2021b) (83.8%), respectively. In addition, our models demonstrate similar or better performance gap compared to other established benchmarks such as ConvNeXt-T (Liu et al., 2022).

Original: Furthermore, as shown in Fig. 1, GC ViT models have better or comparable computational efficiency in terms of number FLOPs over the competing counterpart models.[Table] [Table] [Table]
Revised: Furthermore, as shown in Fig. 1, GC ViT models have better or comparable computational efficiency in terms of number FLOPs over the competing counterpart models.

Original: [Table]

Original: [Table]

Original: 4. D ETECTION
Revised: 4. D ETECTION

Original: [Table] In Table 2, we present object detection and instance segmentation benchmarks  on MS COCO dataset.
Revised: In Table 3, we present object detection and instance segmentation benchmarks using on MS COCO dataset.

Original: Using a Mask-RCNN head,the model with pre-trained GC ViTT (47.9/43.2) backbone outperforms counterparts with pre-trained ConvNeXt-T (Liu et al., 2022) (46.2/41.7) by +1.7 and +1.5 and SwinT (Liu et al., 2021) (46.0/41.6) by +1.9 and +1.6 in terms of box AP and mask AP, respectively.
Revised: Using a Mask-RCNN head, model with pre-trained GC ViT-T (46.5/41.8) backbone outperforms counterparts with pre-trained ConvNeXt-T (Liu et al., 2022) (46.2/41.7) by +0.3 and +0.1 and Swin-T (Liu et al., 2021) (46.0/41.6) by +0.5 and +0.2 in terms of box AP and mask AP, respectively.

Original: Using a Cascade Mask-RCNN head,the models with pre-trained GC ViT-T (51.6/44.6) and GC ViT-S (52.4/45.4) backbones outperform ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +1.2 and +0.9 and ConvNeXt-S (Liu et al., 2022) (51.9/45.0) by +0.5 and +0.4 in terms of box AP and mask
Revised: Using a Cascade Mask-RCNN head, models with pre-trained GC ViT-T (51.1/44.4) and GC ViT-S (52.4/45.4) backbones surpass all competing counterparts. Specifically, it outperforms ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +0.7 and +0.7 and ConvNeXt-S (Liu et al., 2022) (51.9/45.0) by +0.

Original:  AP, respectively.Furthermore, the model with
Revised: and +0.4 in terms of box AP and mask AP, respectively.

Original: GC ViT-B (52.9/45.8) backbone outperforms the counterpart with ConvNeXt-B (Liu et al., 2022) (52.7/45.6)  by +0.2 and +0.2 in terms of box AP and mask AP, respectively.
Revised: However, ConvNeXt-B (Liu et al., 2022) (52.7/45.6) demonstrates better performance compared to GC ViT-B (52.2/45.2) by +0.5 and +0.4 in terms of box AP and mask AP, respectively.

Original: 4. ADE20K S EMANTIC S EGMENTATION R ESULTS
Revised: 4. ADE20K S EMANTIC S EGMENTATION R ESULTS

Original: We present  semantic segmentation benchmarks on ADE20Kdataset in Table 3.
Revised: We present the semantic segmentation benchmarks on ADE20K in Table 4.

Original: The models using pretrained GC ViT-T (47.0), GC ViT-S (48.3)and GC ViT-B (49.0) backbones outperform counterpart models with pre-trained Twins-SVT-S
Revised: Models using pre-trained GC ViT-T (46.5), GC ViT-T (48.3) backbones surpass competing models with pre-trained Twins-

Original:  (Chu et al., 2021a) (46.2), Twins-SVT-B (Chu et al., 2021a) (47.7) and Twins-SVT-L (Chu et al., 2021a) (48.8) by +0.8, +0.6 and +0.2 in terms of mIoU, respectively.
Revised: [Figure] SVT-S (Chu et al., 2021a) (46.2) and Focal-S (Yang et al., 2021b) (48.0) by +0.3 and +0.3 in terms of mIoU, respectively.

Original: In addition, models with GC  ViTbackbones significantly outperform counterparts with
Revised: Models pre-trained with GC ViT-B (49.0) and Focal-B (Yang et al., 2021b) (49.0) both achieve the same segmentation performance. GC ViT significantly outperforms Swin

Original: Swin Transformer backbones, hence demonstrating the effectiveness of the global self-attention.
Revised: Transformer counterparts, hence demonstrating the effectiveness of the global self-attention.

Original:  A BLATION5. C OMPONENT - WISE A NALYSIS
Revised:  A BLATION

Original: [Table]
Revised: [Table]

Original:  As shown in Table 4, we study the role of each component in
Revised: Component-wise Analysis. As shown in Table 5, replacing global self-attention blocks with their local counterparts significantly impacts the performance. Specifically, segmentation shows the most significant degradation. Sharing global context query features consistently improves the performance for all tasks, according to

Original: GC ViT model for classification, detection, instance and semantic segmentation.
Revised: Table 5, and decreases the number of parameters and operations. Such improvements may imply that model generalizability improves by re-using the same tokens throughout the local and global attention blocks. Our experiments also show the benefits of employing relative position bias in both local and global self-attention blocks.  I NTERPRETABILITY

Original: For simplicity, we start with Swin Transformer as the base model and progressively re-design the components to demonstrate their importance in improving the performance.
Revised: To provide further insights on interpretability of the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017)maps in Fig. The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act as an informative source for local attention modules.  C ONCLUSION

Original: Firstly, we remove the window shifting and predictably observe significant  performance degradation across all tasks.
Revised: In this work, we introduced a novel hierarchical ViT, referred to as GC ViT, which can efficiently capture global context by utilizing global query tokens and interact with local regions. We have extensively validated the effectiveness of our model on various tasks. The proposed GC ViT model achieves new SOTA benchmarks for image classification across various model sizes on ImageNet-1K dataset, and surpasses both CNN and ViT-based counterparts by a significant margin. We have also achieved SOTA or competitive performance for downstream tasks of detection and semantic segmentation on high-resolution images using MS COCO and ADE20K datasets.

Original: Changing distribution of parameters to our design improves the performance by +1.7, +2.8, +2.2 and +1.7 in terms of accuracy, box AP, mask AP and mIoU. Such reparametrization includes changing the window size, MLP ratio, number of layers to name but a few.
Revised: We present GC ViT-XXT architecture configuration in Table S.1. For comparison, we also include architecture configurations of GC ViT-T, GC ViT-S and GC ViT-B models. GC ViT-XXT is a lightweight model which achieves a competitive Top-1 classification accuracy of 79.8% on ImageNet1K dataset with only 12 M parameters.

Revised: [Table]

Original: Adding the CNN-based stem of GC ViTto the model provides additional improvements of +0.3, +0.2, +0.2 and +0.2 in terms of accuracy, box AP, mask AP and mIoU.
Revised: Other efforts such as EdgeViT (Pan et al., 2022) in computer vision and BigBird (Zaheer et al., 2020) in NLP have proposed global self-attention in their respective applications. In this section, we discuss the differences between the proposed global self-attention in GC ViT and these efforts. EdgeViT : EdgeViT and GC ViT use completely different self-attention blocks.

Original: In addition, using our proposed downsampler further improves the accuracy, box AP, mask AP and mIoU by +0.4, +0.1, +0.1 and +0.3, respectively.
Revised: The EdgeViT uses a series of local aggregation (convolution), sparse attention and local propagation(depthwise convolution), whereas GC ViT only uses an interleaved pattern of local and global self-attention layers without convolution in order to compute self-attention. The proposed global sparse attention in EdgeViT and GCViT are competently different. EdgeViT samples representative tokens and only computes sparse self-attention between these representative tokens with reduced feature size.

Original: The last two changes demonstrate the importance of convolutional inductive bias and capturing the inter-channel dependencies in our model.
Revised: On the contrary, GC ViT computes self-attention between the global queries (not just the token) and local keys and values without any subsampling in their respective local regions. Furthermore, in EdgeViT, only subsampled representative tokens per region interact In the self-attention module; however, in GC ViT, the global queries interact with the entire local regions, instead of interacting with each other, and hence provide an effective mechanism for capturing both short and long-range spatial dependencies.

Original: Finally, leveraging the proposed global self-attention improves the performance by by +0.8, +0.8, +0.6 and +1.2 in terms of accuracy, box AP, mask AP and mIoU.
Revised: In addition, GC ViT generates global query tokens by using a series of modified Fused MB-Conv from the entire image and without subsampling. Note that the resolution of global query tokens are the same as local query and values. However, in EdgeViT: (A) the representative tokens are obtained per local window, not the entire image, and by subsampling and reducing the feature resolution. Hence, since generated tokens have a lower resolution compared to their respective local windows, this could result in loss of spatial information and impact the effectiveness of self-attention.

Original: Hence, this validates the effectiveness of the 
Revised: Unlike EdgeViT, the downsampler in GCViT also benefits from modified Fused-MBConv blocks which allows for modeling cross channel interactions and impose more locality and convolutional inductive bias. [Table] [Table]

Original: [Figure]
Revised: [Table]

Original:  proposedglobal self-attention, in particular for downstream tasks with high resolution images such as semantic segmentation in whichmodeling long-range spatial dependencies is critical.
Revised: BigBird : Bigbird, which is primarily introduced for NLP applications with 1D inputs, has significant differences compared to GC ViT, which is proposed for computer vision with mainly 2D inputs. Firstly, BigBird uses a combination of random, window and global attention mechanisms, which is different from the proposed local and global self-attention scheme in GC ViT. In addition, BigBird does not have any specific mechanisms for extracting global tokens as the existing tokens or additional special tokens can be specified as global tokens.

Revised: On the contrary, the global tokens in

Revised: GC ViT are extracted by the proposed global query generator module which consists of a series of modified Fused MB-Conv blocks to extract contextual information from the entire input features.

Revised: Lastly, BigBird employs a set of global tokens which attend to the entire input sequence; in this case, select global query, key and values attend to local query, key and value tensors.

Revised: However, as opposed to this formulation, in GC ViT, the global query tokens attend to local key and value tokens in partitioned windows.

Revised: This is due to the fact that attending to the entire input sequence, as done in

Revised: BigBird, is not feasible considering the larger size of input features in computer vision.

Original: 5. D OWNSAMPLER D ESIGN
Revised: D A BLATION D. I MPORTANCE OF D OWN - SAMPLER

Original: We studied the effectiveness of various downsampler blocks in Table 5.
Revised: We studied the effectiveness of various down-sampler blocks in Table S.2.

Original: The simplest alternative to our design is a pair of convolutional and maxpooling layers.
Revised: The simplest alternative to our design is a pair of convolutional and maxpooling layers.

Original: However, it results in a reduction of
Revised: However, it results in a reduction of

Original: ImageNet Top-1 accuracy by -0.7.
Revised: ImageNet Top-1 accuracy by 0 .

Original: Patch merging is another variant which was introduced in Swin
Revised: Patch merging is another variant which was introduced in Swin

Original: Transformers (Liu et al., 2021).
Revised: Transformers (Liu et al., 2021).

Original: [Table]
Revised: However, it reduces the accuracy by 0 . Another variant is the down-sampler introduced in ConvNeXt (Liu et al., 2022) which is a 3 × 3 strided convolution (s= 2 ).

Original: However, it  reduces the accuracy by -0.5.
Revised: However, it also reduces the accuracy by 0 .

Original:  Finally, our down-sampler which consists of a modified Fused-MBConv block and strided convolution and shows the best result.
Revised: 3% when used in our setting. Finally, our down-sampler which consists of a modified Fused-MBConv block and strided convolution and shows the best result.

Original: Importance of the former component is explained by the SE operation which boosts cross channel interaction while keeping number of parameters and FLOPs low.
Revised: Importance of the former component is explained by the SE operation which boosts cross channel interaction while keeping number of parameters and FLOPs low.

Original: We conclude that our proposed down-sampler is essential to achieve high accuracy as it introduces convolutional inductive bias.
Revised: We conclude that our proposed down-sampler is essential to achieve high accuracy as it introduces convolutional inductive bias.

Original: I NTERPRETABILITY
Revised: D. C OMPARISON TO S WIN T RANSFORMER

Original: To provide further insights on interpretability of the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017) maps in 
Revised: The most comparable architecture to GC ViT is the Swin Transformer (Liu et al., 2021) with local window splitting. As a result, we start with a Swin Transformer base design and progressively re-design the components to demonstrate their importance in improving the accuracy.

Original: Fig. The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act as an informative source for local attention modules. In addition, corresponding Grad-CAM maps demonstrate accurate object localization with most intricate details.  C ONCLUSION
Revised: Firstly, we remove window shifting and predictably observe 1 . 1% drop in accuracy.

Original: We introduced a new vision transformer architecture named GC ViT that can efficiently capture global context by utilizing global query tokens and interact with local regions. Through extensive experiments we show SOTA benchmarks for image classification on ImageNet-1K dataset, surpassing CNN/ViT-based counterparts by large margin. We also consistently achieved SOTA for downstream tasks of detection, instance and semantic segmentation on MS COCO and ADE20K datasets.
Revised: Changing distribution of parameters to our design improves the accuracy to 81 .

Original: GC ViT model configurations are presented in Table S.1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number
Revised: Such reparametrization includes changing the window size, MLP ratio, number of layers to name but a few, as shown in Table 1. Adding the

Original: [Table]
Revised: [Table]

Original: C A BLATION C. G LOBAL Q UERY

Original: We performed ablation studies to validate the effectiveness of the proposed global query.

Original: Using the same architecture, instead of global query, we compute: (1) global key and value features and interact them with local query (2) global value features and interact it with local query and key.

Original: As shown in

Original: Table S.2, replacing global query may significantly impact the performance for image segmentation and downstream tasks such as object detection, instance segmentation and semantic segmentation

Original: [Table]
Revised: [Table]

Original: We also used used Exponential Moving Averages (EMA) and observed slight improvement in terms of ImageNet TOp-1 accuracy.
Revised: CNN-based stem of GC ViT to the model provides an additional 0 . 3% improvement in terms of ImageNet Top-1 accuracy.

Original: Furthermore, the performance of the model across different batch sizes were stable as we did not observe significant changes.
Revised: In addition, changing the down-sampling block to the one proposed for

Original: Table S.3 demonstrates the effect of EMA and batch size on the accuracy of a GCViT-T model [Table]
Revised: GC ViT improves the performance to 82 . The last two changes demonstrate the importance of convolutional inductive bias in our model.

Original: [Table]
Revised: Finally, adding the proposed global self-attention improves the accuracy to 83 . Note that number of layers is not changed to the previous experiment where all layers benefit from only local self-attention.

Original: [Figure] [Figure] D I NTERPRETABILITY
Revised: These experiments validate the effectiveness of GC

Original: In Fig. S.1, we illustrate the learned global query token maps and demonstrate their effectiveness in capturing long-range contextual representations from different image regions.
Revised: ViT components and demonstrate their improvement over Swin Transformers (Liu et al., 2021) as the total number of parameters are kept the same.

Original: E T RAINING D ETAILS
Revised: E T RAINING D ETAILS

Original: For image classification, GC ViT models were trained using four computational nodes with
Revised: For image classification, GC ViT models were trained using four computational nodes with

Original: NVIDIA A100 GPUs.
Revised: Nvidia A100 GPUs.

Original: The total training batch size is 1024 ( 32 per GPU) for GC ViT-S, GC ViT-B, GC ViT-L and 4096 ( 128 per GPU) for GC ViT-XXT, GC ViT-XT and GC ViT-T.
Revised: The total training batch size is 1024 ( 32 per GPU) for GC ViT-S, GC ViT-B and 4096 ( 128 per GPU) for GC ViT-XXT, GC ViT-XT and GC ViT-T.

Original: On average, each model required 32 hours of training with the specified hyper-parameters as indicated in the paper.
Revised: On average, each model required 32 hours of training with the specified hyper-parameters as indicated in the paper.

Original: All classification models were trained using the timm package (Wightman, 2019).
Revised: All classification models were trained using the timm package Wightman (2019).

Original: Object detection and instance segmentation models as well as semantic segmentation models were trained using one computational node with 8 NVIDIA A40 GPUs using a total batch size of 16 , hence a batch size ofper GPU.
Revised: Object detection and instance segmentation models as well as semantic segmentation models were trained using one computational node with 8 Nvidia A40 GPUs using a total batch size of 16 , hence a batch size of 2 per GPU.

Original: Detection and instance segmentation models were trained using mmdetection (Chen et al., 2019) package and on average required 56 hours of training.
Revised: Detection and instance segmentation models were trained using mmdetection Chen et al. (2019) package and on average required 56 hours of training.

Original: Semantic segmentation models were trained using mmsegmentation (Contributors, 2020) package, and on average requiredhours of training.
Revised: Semantic segmentation models were trained using mmsegmentation Contributors (2020) package, and on average requiredhours of training.

Original: F C OMPLEXITY A NALYSIS
Revised: F C OMPLEXITY A NALYSIS

Original: Given an input feature map of x ∈ R H × W × C at each stage with a window size of h × w , the computational complexity of GC ViT is as follows
Revised: Given an input feature map of x ∈ R H × W × C at each stage with a window size of h × w , the computational complexity of GC ViT is as follows

Original: [Equation]
Revised: [Equation]

Original: The efficient design of global query token generator and other components allows to maintain a similar computational complexity in comparison to Swin Transformer Liu et al.
Revised: The efficient design of global query token generator and other components allows to maintain a similar computational complexity in comparison to Swin Transformer Liu et al.

Original: (2021) while being able to capture long-range information and achieve better higher accuracy for classification and downstream tasks such as detection and segmentation.
Revised: (2021) while being able to capture long-range information and achieve better higher accuracy for classification and downstream tasks such as detection and segmentation.

Original: G I MAGE N ET CLASSIFICATION B ENCHMARKS
Revised: G I MAGE N ET V2 B ENCHMARKS

Original: In Table S.4, we provide a comprehensive benchmark in terms of Top-1 accuracy for the models that are only trained on ImageNet-1K (Deng et al., 2009) dataset, andwithout additional data.
Revised: In Table S.4, we have evaluated the performance of GC ViT on ImageNetV2 dataset (Recht et al., 2019) to further measure its robustness. Specifically, we have used different sampling strategies of Matched Frequency and

Original: [Table] [Table] [Table] [Table]
Revised: Threshold-0.7.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Medium

Original: [Table] Wang [Table] [Table]
Revised: These benchmarks demonstrate the competetive performance of GC ViT on ImageNetV2 dataset and validates its effectiveness in robustness and generalizability.

Original: [Table]

Original: [Table] [Table] [Table] -B
Revised: H E FFECT OF G LOBAL C ONTEXT M ODULE

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: B

Original: [Table]

Original: 2021a

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Dosovitskiy

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Touvron

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: PVT-Large (Wang et al.,

Original: [Table]

Original: [Table] [Table] [Table] Base/16 (Touvron et al., 2021b)
Revised: In order to demonstrate the effectiveness of Global Context (GC) module, we use Swin Transformers as the base model and add our propoped GC module.

Original: [Table] CrossViT-B (Chen et al., 2021) 21. T2T-ViT-24 (Yuan et al., 2021) 14. CPVT-B (Chu et al., 2021b) 17.
Revised: In this analysis, we remove the window shifting operation from Swin Transformers, since GC module is capable of modeling cross-region interactions.

Original: Twins-PCPVT-L (Chu et al., 2021a) 9. Swin-B (Liu et al., 2021) 15. CoAtNet-2 (Dai et al., 2021) 8. PVT-v2-B4 (Wang et al., 2022) 10.
Revised: As shown in Table S.5, addition of GC module improves the ImageNet Top-1 accuracy by +0 . 9% and +0 .

Original: Twins-SVT-L (Chu et al., 2021a) 15.

Original: ConvNeXt-B (Liu et al., 2022) 15. Focal-B (Yang et al., 2021b) 16. PVT-v2-B5 (Wang et al., 2022) 11. CSwin-B (Dong et al., 2022) 15.
Revised: 7% for Swin Transformers Tiny and Small variants respectively.

Original: BoTNet (Dong et al., 2022) 19.

Original: GC ViT-B 14.

Original: ConvNeXt-L (Liu et al., 2022) 34.

Original: CoAtNet-3 (Dai et al., 2021) 34.

Original: GC ViT-L 32.