Original: G LOBAL C ONTEXT V ISION T RANSFORMERS
Revised: G LOBAL C ONTEXT V ISION T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision.
Revised: We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision.

Original: Our method leverages global context self-attention modules, joint with standard local selfattention, to effectively yet efficiently model both long and short-range spatial interactions, without the need for expensive operations such as computing attention masks or shifting local windows.
Revised: Our method leverages global context self-attention modules, joint with standard local selfattention, to effectively yet efficiently model both long and short-range spatial interactions, without the need for expensive operations such as computing attention masks or shifting local windows.

Original: In addition, we address the lack of the inductive bias in ViTs, and propose to leverage a modified fused inverted residual blocks in our architecture.
Revised: In addition, we address the lack of the inductive bias in ViTs, and propose to leverage a modified fused inverted residual blocks in our architecture.

Original: Our proposed GC ViT achieves state-of-the-art results across image classification, object detection and semantic segmentation tasks.
Revised: Our proposed GC ViT achieves state-of-the-art results across image classification, object detection and semantic segmentation tasks.

Original: On
Revised: On

Original: ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28 M, 51 M and 90 M parameters achieve 83.4% , 83.9% and 84.5% Topaccuracy, respectively, surpassing comparably-sized prior art such as CNN-based
Revised: ImageNet-1K dataset for classification, the tiny, small and base variants of GC ViT with 28 M, 51 M and 90 M parameters achieve 83.4% , 83.9% and 84.5% Topaccuracy, respectively, surpassing comparably-sized prior art such as CNN-based

Original: ConvNeXt and ViT-based Swin Transformer by a large margin.
Revised: ConvNeXt and ViT-based Swin Transformer by a large margin.

Original: Pre-trained GC
Revised: Pre-trained GC

Original: ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins.
Revised: ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently, sometimes by large margins.

Original: Code and pre-trained models available at https://anonymous.4open.science/r/GCViTs-8852/.
Revised: Code and pre-trained models available at https://anonymous.4open.science/r/GCViTs-8852/.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: During the recent years, Transformers (Vaswani et al., 2017) have achieved State-Of-The-Art (SOTA) performance in Natural Language Processing (NLP) benchmarks and became the de facto model for various tasks.
Revised: During the recent years, Transformers (Vaswani et al., 2017) have achieved State-Of-The-Art (SOTA) performance in Natural Language Processing (NLP) benchmarks and became the de facto model for various tasks.

Original: A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021).
Revised: A key element in the success of Transformers is the self-attention mechanism which allows for capturing contextual representations via attending to both distant and nearby tokens (Yin et al., 2021).

Original: Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in a monolithic architecture with minor differences comparing to encoder of the original Transformer.
Revised: Following this trend, Vision Transformer (ViT) (Dosovitskiy et al., 2020) proposed to utilize image patches as tokens in a monolithic architecture with minor differences comparing to encoder of the original Transformer.

Original: Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA or competitive performance in various computer vision tasks.
Revised: Despite the historic dominance of Convolutional Neural Network (CNN) in computer vision, ViT-based models have achieved SOTA or competitive performance in various computer vision tasks.

Original: In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison to CNN.
Revised: In essence, the self-attention mechanism in ViT allows for learning more uniform short and long-range information (Raghu et al., 2021) in comparison to CNN.

Original: However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) in which capturing multi-scale long-range information is crucial for accurate representation modeling.
Revised: However, the monolithic architecture of ViT and quadratic computational complexity of self-attention baffle their swift application to high resolution images (Yang et al., 2021a) in which capturing multi-scale long-range information is crucial for accurate representation modeling.

Original: Several efforts (Liu et al., 2021; Dong et al., 2021;
Revised: Several efforts (Liu et al., 2021; Dong et al., 2021;

Original: Chu et al., 2021a;
Revised: Chu et al., 2021a;

Original: Tu et al., 2022), most notably
Revised: Tu et al., 2022), most notably

Original: Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and longrange spatial dependencies by proposing multi-resolution architectures in which the self-attention is computed in local windows and cross-window connections such as window shifting are used for modeling the interactions across different regions.
Revised: Swin Transformer (Liu et al., 2021), have attempted to address the balance between short- and longrange spatial dependencies by proposing multi-resolution architectures in which the self-attention is computed in local windows and cross-window connections such as window shifting are used for modeling the interactions across different regions.

Original: Despite this progress, the limited receptive field of local windows baffles the capability of self-attention to capture long-range information, and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window.
Revised: Despite this progress, the limited receptive field of local windows baffles the capability of self-attention to capture long-range information, and window-connection schemes such as shifting only cover a small neighborhood in the vicinity of each window.

Original: Subsequent efforts such as Focal Transformer (Yang et al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.
Revised: Subsequent efforts such as Focal Transformer (Yang et al., 2021b) attempted to address this issue by designing highly sophisticated self-attention modules with increased model complexity.

Original: The burden of computationally expensive operations such as token unfolding and rolling and additional computations of key and values can significantly affect the run-time and throughput.
Revised: The burden of computationally expensive operations such as token unfolding and rolling and additional computations of key and values can significantly affect the run-time and throughput.

Original: In this work, we introduce the Global Context (GC) ViT network to address these limitations.
Revised: In this work, we introduce the Global Context (GC) ViT network to address these limitations.

Original: Specifically, we propose a hierarchical ViT architecture consisting of local and global self-attention modules.
Revised: Specifically, we propose a hierarchical ViT architecture consisting of local and global self-attention modules.

Original: At each stage, we compute global query tokens, using modified fused inverted residual
Revised: At each stage, we compute global query tokens, using modified fused inverted residual

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: Swin
Revised: Swin

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: ConvNext
Revised: ConvNext

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: DeiT-S
Revised: DeiT-S

Original: DeiT
Revised: DeiT

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: PVT-S
Revised: PVT-S

Original: PVT-M
Revised: PVT-M

Original: PVT-L
Revised: PVT-L

Original: Focal-T
Revised: Focal-T

Original: Focal-S
Revised: Focal-S

Original: Focal
Revised: Focal

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: GC ViT-XT
Revised: GC ViT-XT

Original: GC ViT-T
Revised: GC ViT-T

Original: GC ViT-S
Revised: GC ViT-S

Original: GC ViT
Revised: GC ViT

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ConvNext
Revised: ConvNext

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: ConvNext
Revised: ConvNext

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Revised: ConvNext

Revised: [Figure]

Revised: ResNet

Revised: [Figure]

Original: ConvNext [Figure] ResNet 
Revised: ResNet [Figure]

Original: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: ResNet

Original: [Figure]

Original: DeiT
Revised: DeiT

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: DeiT
Revised: DeiT

Original: [Figure]
Revised: [Figure]

Original: PVT-S
Revised: PVT-S

Original: PVT-M
Revised: PVT-M

Original: PVT-L
Revised: PVT-L

Original: Focal-T
Revised: Focal-T

Original: Focal-S
Revised: Focal-S

Original: Focal
Revised: Focal

Original: [Figure]
Revised: [Figure]

Original: GC ViT-XT
Revised: GC ViT-XT

Original: GC ViT-T GC ViT-
Revised: GC ViT-T GC ViT-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: P
Revised: P

Original: D o w n s a m p l e
Revised: D o w n s a m p l e

Original: Lo c a l M SA
Revised: Lo c a l M SA

Original: Global Token Gen.
Revised: Global Token Gen.

Original: G l oba l M
Revised: G l oba l M

Original: SA M L P
Revised: SA M L P

Original: M L P
Revised: M L P

Original: Lo c a l M SA
Revised: Lo c a l M SA

Original: Global Token Gen.
Revised: Global Token Gen.

Original: G l oba l M
Revised: G l oba l M

Original: SA M L P 2 D
Revised: SA M L P 2 D

Original: A v g P oo l
Revised: A v g P oo l

Original: H ead D
Revised: H ead D

Original: o w n s a m p l e
Revised: o w n s a m p l e

Original: M L P blocks, which we refer to as Fused-MBConv blocks, that encompass global contextual information from different image regions.
Revised: M L P blocks, which we refer to as Fused-MBConv blocks, that encompass global contextual information from different image regions.

Original: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values.
Revised: While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values.

Original: As we demonstrate in Sec. 5, sharing global queries allow for reducing number of parameters and FLOPs and increasing generalizability.
Revised: As we demonstrate in Sec. 5, sharing global queries allow for reducing number of parameters and FLOPs and increasing generalizability.

Original: The design of our proposed framework for global query generation and self-attention is intuitive and simple and can be efficiently implemented using major deep learning framework.
Revised: The design of our proposed framework for global query generation and self-attention is intuitive and simple and can be efficiently implemented using major deep learning framework.

Original: Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied to high-resolution images.
Revised: Hence, it eliminates sophisticated and computationally expensive operations and ensures the effectiveness of self-attention when applied to high-resolution images.

Original: We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks.
Revised: We have extensively validated the effectiveness of the proposed GC ViT using three publicly available datasets for various computer vision tasks.

Original: For image classification using ImageNet-1K dataset,
Revised: For image classification using ImageNet-1K dataset,

Original: GC ViT with 28 M, 51 M and 90 M parameters, referred to as tiny, small and base variants, achieve new SOTA benchmarks of 83.4% , 83.9% and 84.5% Top-1 accuracy.
Revised: GC ViT with 28 M, 51 M and 90 M parameters, referred to as tiny, small and base variants, achieve new SOTA benchmarks of 83.4% , 83.9% and 84.5% Top-1 accuracy.

Original: Hence, GC ViT consistently outperforms both ConvNeXt (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1).
Revised: Hence, GC ViT consistently outperforms both ConvNeXt (Liu et al., 2022) and Swin Transformer (Liu et al., 2021) models by a significant margin (see Fig. 1).

Original: Using a pre-trained GC ViT base backbone with Cascade Mask
Revised: Using a pre-trained GC ViT base backbone with Cascade Mask

Original: RCNN (He et al., 2017) head, our model achieves a box mAP of 52.2 for object detection and a mask mAP of 45.2 for instance segmentation on the MS COCO test-dev.
Revised: RCNN (He et al., 2017) head, our model achieves a box mAP of 52.2 for object detection and a mask mAP of 45.2 for instance segmentation on the MS COCO test-dev.

Original: In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves a mIoU of 49.0 on ADE20K for semantic segmentation.
Revised: In addition, using an UPerNet (Xiao et al., 2018) head, our model achieves a mIoU of 49.0 on ADE20K for semantic segmentation.

Original: Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sized models on both MS COCO and ADE20K datasets.
Revised: Other variants of GC ViT with different learning capacities also demonstrate SOTA results when compared to similarly-sized models on both MS COCO and ADE20K datasets.

Original: Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.
Revised: Hence, GC ViT demonstrates great scalability for high-resolution images on various downstream tasks, validating the effectiveness of the proposed framework in capturing both short and long-range information.

Original: The main contributions of our work are summarized as follows:
Revised: The main contributions of our work are summarized as follows:

Original: • A novel hierarchical Transformer model, called GC ViT, that can be employed as a general backbone in various computer vision tasks such as classification, detection, instance segmentation down-stream tasks such as detection, instance and semantic segmentation.
Revised: • A novel hierarchical Transformer model, called GC ViT, that can be employed as a general backbone in various computer vision tasks such as classification, detection, instance segmentation down-stream tasks such as detection, instance and semantic segmentation.

Original: • A novel and yet simple design comprising of global self-attention and token generation modules that allow for modeling long-range dependencies by capturing global contextual information and hence eliminating the need for highly sophisticated or complex operations.
Revised: • A novel and yet simple design comprising of global self-attention and token generation modules that allow for modeling long-range dependencies by capturing global contextual information and hence eliminating the need for highly sophisticated or complex operations.

Original: • As demonstrated in Fig.
Revised: • As demonstrated in Fig.

Original: 1, the proposed GC ViT achieves new SOTA benchmarks on ImageNet-1K dataset for a variety of model sizes and FLOPs, outperforming both CNN and ViT-based models by a significant margin.
Revised: 1, the proposed GC ViT achieves new SOTA benchmarks on ImageNet-1K dataset for a variety of model sizes and FLOPs, outperforming both CNN and ViT-based models by a significant margin.

Original: Using GC ViT as the backbone yields SOTA or competitive performance for object detection and semantic segmentation on MS COCO and ADE20K datasets, respectively.
Revised: Using GC ViT as the backbone yields SOTA or competitive performance for object detection and semantic segmentation on MS COCO and ADE20K datasets, respectively.

Original:  GC V I T ARCHITECTURE
Revised:  GC V I T ARCHITECTURE

Original: Architecture.
Revised: Architecture.

Original: Fig.
Revised: Fig.

Original: 2 depicts the architecture of GC ViT. Similar to previous efforts (Liu et al., 2021;
Revised: 2 depicts the architecture of GC ViT. Similar to previous efforts (Liu et al., 2021;

Original: Yang et al., 2021b), we use a hierarchical framework to obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension by factors of 2 and 2 , respectively.
Revised: Yang et al., 2021b), we use a hierarchical framework to obtain feature representations at several resolutions (called stages) by decreasing the spatial dimensions while expanding the embedding dimension by factors of 2 and 2 , respectively.

Original: At first, given an input image with resolution of x ∈ R H × W × 3 , following Wang et al. (2022), we obtain overlapping patches by applying a 3 ×convolutional layer with a stride of 2 and appropriate padding.
Revised: At first, given an input image with resolution of x ∈ R H × W × 3 , following Wang et al. (2022), we obtain overlapping patches by applying a 3 ×convolutional layer with a stride of 2 and appropriate padding.

Original: Then patches are projected it into a C -dimensional embedding space.
Revised: Then patches are projected it into a C -dimensional embedding space.

Original: We show the benefit of using overlapping as opposed to non-overlapping patches in Sec. 5.
Revised: We show the benefit of using overlapping as opposed to non-overlapping patches in Sec. 5.

Original: Every GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features.
Revised: Every GC ViT stage is composed of alternating local and global self-attention modules to extract spatial features.

Original: Both operate in local windows like Swin
Revised: Both operate in local windows like Swin

Original: Transformer (Liu et al., 2021), however, the global self-attention accesses global features extracted by the Global Token Generator (GTG).
Revised: Transformer (Liu et al., 2021), however, the global self-attention accesses global features extracted by the Global Token Generator (GTG).

Original: The GTG is a CNN-like module that extracts features from the entire image only once at every stage.
Revised: The GTG is a CNN-like module that extracts features from the entire image only once at every stage.

Original: The spatial resolution is decreased by 2 while increasing the number of channels by a downsampling block after every stage.
Revised: The spatial resolution is decreased by 2 while increasing the number of channels by a downsampling block after every stage.

Original: Resulting features are passed through average pooling and linear layers to create an embedding for a downstream task.
Revised: Resulting features are passed through average pooling and linear layers to create an embedding for a downstream task.

Original: The GC ViT architecture relies on novel blocks such as a downsampling operator, a global query generator and a global self-attention module described in the next sections.
Revised: The GC ViT architecture relies on novel blocks such as a downsampling operator, a global query generator and a global self-attention module described in the next sections.

Original: Downsamplingblock
Revised: Downsamplingblock

Original: F u s ed
Revised: F u s ed

Original: M B C on v
Revised: M B C on v

Original: C on v 3 x 3 , s t r i de
Revised: C on v 3 x 3 , s t r i de

Original: La y e r N o r m
Revised: La y e r N o r m

Original: Downsampling.
Revised: Downsampling.

Original: We borrow an idea of spatial feature contraction from
Revised: We borrow an idea of spatial feature contraction from

Original: CNN models that imposes locality bias and cross channel communication while reducing dimensions.
Revised: CNN models that imposes locality bias and cross channel communication while reducing dimensions.

Original: We use a modified Fused-MBConv block, followed by a max pooling layer with a kernel size of 3 and stride ofas a downsampling operator, see Fig 3.
Revised: We use a modified Fused-MBConv block, followed by a max pooling layer with a kernel size of 3 and stride ofas a downsampling operator, see Fig 3.

Original: The Fused-MBConv block in our work is similar to the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in
Revised: The Fused-MBConv block in our work is similar to the one in EfficientNetV2 (Tan & Le, 2021) with modifications as in

Original: [Equation]
Revised: [Equation]

Original: where SE, GELU and DW-Conv 3 × 3 denote Squeeze and Excitation block (Hu et al., 2018b), Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016) and 3 ×depth-wise convolution, respectively.
Revised: where SE, GELU and DW-Conv 3 × 3 denote Squeeze and Excitation block (Hu et al., 2018b), Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016) and 3 ×depth-wise convolution, respectively.

Original: In our proposed architecture, the Fused-MBConv blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies.
Revised: In our proposed architecture, the Fused-MBConv blocks provide desirable properties such as inductive bias and modeling of inter-channel dependencies.

Original: Attention.
Revised: Attention.

Original: Multi-head self-attention is the the core computational operator in the GC ViT architecture to extract semantic information from the image.
Revised: Multi-head self-attention is the the core computational operator in the GC ViT architecture to extract semantic information from the image.

Original: GC ViT is composed of local and global selfattention modules illustrated in Fig 4.
Revised: GC ViT is composed of local and global selfattention modules illustrated in Fig 4.

Original: Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size.
Revised: Similar to Swin Transformer (Liu et al., 2021), we benefit from splitting the image into windows and performing local self-attention within them, this leads to linear complexity scaling with image size.

Original: The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross-patch communication with those far beyond the local window.
Revised: The local self-attention extracts local, short-range, information and in order to facilitate long range dependencies we propose to use a novel global self attention to allow cross-patch communication with those far beyond the local window.

Original: Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module.
Revised: Global self-attention attends other regions in the image via global query token that represents image embedding extracted with CNN-like module.

Original: 2. G LOBAL Q UERY G ENERATOR
Revised: 2. G LOBAL Q UERY G ENERATOR

Original: We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value features.
Revised: We propose to generate global query tokens that encompass information across the entire input feature maps for interaction with local key and value features.

Original: Specifically, as shown in Fig.
Revised: Specifically, as shown in Fig.

Original: 5, a layer f in
Revised: 5, a layer f in

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the the generator consists of a Fused-MBConv block followed by a max pooling layer, similar to the one described in Sec. 2, and the final global query q g , i at stage i ( i ∈ { 1 , 2 , 3 , 4 } ) of GC ViT is computed according to
Revised: the the generator consists of a Fused-MBConv block followed by a max pooling layer, similar to the one described in Sec. 2, and the final global query q g , i at stage i ( i ∈ { 1 , 2 , 3 , 4 } ) of GC ViT is computed according to

Original: [Equation]
Revised: [Equation]

Original: These query tokens are computed once at every stage of GC ViT model and shared across all global attention blocks, hence decreasing number of parameters and FLOPs and improving the generalizability.
Revised: These query tokens are computed once at every stage of GC ViT model and shared across all global attention blocks, hence decreasing number of parameters and FLOPs and improving the generalizability.

Original: In addition, the Global attention layers only learn local key and value features which will be used for interaction with global query tokens
Revised: In addition, the Global attention layers only learn local key and value features which will be used for interaction with global query tokens

Original: . 2. G LOBAL S ELF -A TTENTION
Revised: . 2. G LOBAL S ELF -A TTENTION

Original: Fig.
Revised: Fig.

Original: 4 demonstrates the main idea of our contribution.
Revised: 4 demonstrates the main idea of our contribution.

Original: Local self-attention can only query patches within a local window, whereas with global attention attention can query image globally while still operating in the window.
Revised: Local self-attention can only query patches within a local window, whereas with global attention attention can query image globally while still operating in the window.

Original: The only difference in implementation is that query component is pre-computed in case of global attention and the rest is the same as in local attention.
Revised: The only difference in implementation is that query component is pre-computed in case of global attention and the rest is the same as in local attention.

Original: In each stage, GC ViT employs alternating local and global self-attention blocks to effectively capture both local and global spatial information.
Revised: In each stage, GC ViT employs alternating local and global self-attention blocks to effectively capture both local and global spatial information.

Original: The global self-attention utilizes global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with extracted local key and value features.
Revised: The global self-attention utilizes global query tokens, obtained according to Eq. 2 and shared across all blocks, to interact with extracted local key and value features.

Original: Fig.
Revised: Fig.

Original: 6 illustrates the difference between local and global self-attention.
Revised: 6 illustrates the difference between local and global self-attention.

Original: The global attention query q g has a size of B × C × h × w , wherein B , C , h and w denote batch size, embedding dimension, local window height and width, respectively.
Revised: The global attention query q g has a size of B × C × h × w , wherein B , C , h and w denote batch size, embedding dimension, local window height and width, respectively.

Original: Moreover, q g
Revised: Moreover, q g

Original: [Figure]
Revised: [Figure]

Original: MSA
Revised: MSA

Original: La y e r N o r m
Revised: La y e r N o r m

Original: MLP MLP
Revised: MLP MLP

Original: MLP
Revised: MLP

Original: C on c a t
Revised: C on c a t

Original: P r o j e c t i on multi head
Revised: P r o j e c t i on multi head

Original: Global MSA
Revised: Global MSA

Original: La y e r N o r m
Revised: La y e r N o r m

Original: MLP
Revised: MLP

Original: MLP
Revised: MLP

Original: C on c a t
Revised: C on c a t

Original: P r o j e c t i on
Revised: P r o j e c t i on

Original: [Figure]
Revised: [Figure]

Original: is repeated along the batch dimension to compensate for the overall number of windows and batch size B ∗ =
Revised: is repeated along the batch dimension to compensate for the overall number of windows and batch size B ∗ =

Original: B × N where N is the number of local windows.
Revised: B × N where N is the number of local windows.

Original: q g is further reshaped into multiple head.
Revised: q g is further reshaped into multiple head.

Original: The value and key are computed within each local window using a linear layer.
Revised: The value and key are computed within each local window using a linear layer.

Original: # Input/output shape: (B*, N, C)
Revised: # Input/output shape: (B*, N, C)

Original: # B*: Batchsize*Num Windows; H: Height; # W: Width; C: dim; q_g: Global Token; # F: Num Attention Head; N: Num Windows; def init():f=nn.
Revised: # B*: Batchsize*Num Windows; H: Height; # W: Width; C: dim; q_g: Global Token; # F: Num Attention Head; N: Num Windows; def init():f=nn.

Original: Linear(C, 2*C) softmax = nn.Softmax(dim=-1) def forward(x, q_g): B*, N, C = x.shape
Revised: Linear(C, 2*C) softmax = nn.Softmax(dim=-1) def forward(x, q_g): B*, N, C = x.shape

Original: B, C, h, w = q_global.shape kv =
Revised: B, C, h, w = q_global.shape kv =

Original: f(x).reshape(B*, N, 2, F, C //
Revised: f(x).reshape(B*, N, 2, F, C //

Original: F) kv = kv.permute(2, 0, 3, 1, 4) k, v = split(kv, (1, 1), 0) q_g
Revised: F) kv = kv.permute(2, 0, 3, 1, 4) k, v = split(kv, (1, 1), 0) q_g

Original: = q_g.repeat(1, B* // B, 1, 1) q_g
Revised: = q_g.repeat(1, B* // B, 1, 1) q_g

Original: = q_g.reshape(B*, F, N, C // F) qk =
Revised: = q_g.reshape(B*, F, N, C // F) qk =

Original: matmul(q_g,k.transpose(-2, -1))
Revised: matmul(q_g,k.transpose(-2, -1))

Original: attn =
Revised: attn =

Original: softmax(qk) return matmul(attn, v).reshape(B*, N, C)
Revised: softmax(qk) return matmul(attn, v).reshape(B*, N, C)

Original: The global self-attention query, key and value features are computed as follows
Revised: The global self-attention query, key and value features are computed as follows

Original: Since the partitioned windows only contain local information, interaction with rich contextual information embedded in the global query tokens provides an effective way of enlarging the receptive field and attending to various regions in the input feature maps.
Revised: Since the partitioned windows only contain local information, interaction with rich contextual information embedded in the global query tokens provides an effective way of enlarging the receptive field and attending to various regions in the input feature maps.

Original: The self-attention module is computed as in
Revised: The self-attention module is computed as in

Original: [Equation]
Revised: [Equation]

Original: where d is scaling factor and b is a learnable relative position bias term.
Revised: where d is scaling factor and b is a learnable relative position bias term.

Original: Following previous efforts (Hu et al., 2018a; Liu et al., 2021), assuming position change between [ − p + 1 , p − 1] along horizontal and vertical axes, b is sampled from the grid ˆ b ∈ R (2 p − 1) × (2 p − 1) .
Revised: Following previous efforts (Hu et al., 2018a; Liu et al., 2021), assuming position change between [ − p + 1 , p − 1] along horizontal and vertical axes, b is sampled from the grid ˆ b ∈ R (2 p − 1) × (2 p − 1) .

Original: As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks.
Revised: As shown in Sec. 5, relative position bias improves the performance, especially for dense prediction downstream tasks.

Original: In Algorithm 1, we present a PyTorch-like pseudocode for computing global self-attention in GC ViT. A complexity analysis of the global selfattention is presented in the supplementary materials.
Revised: In Algorithm 1, we present a PyTorch-like pseudocode for computing global self-attention in GC ViT. A complexity analysis of the global selfattention is presented in the supplementary materials.

Original: 2. GC V I T MODEL CONFIGURATIONS
Revised: 2. GC V I T MODEL CONFIGURATIONS

Original: GC ViT model configurations are presented in Table 1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number.
Revised: GC ViT model configurations are presented in Table 1 describing the choice of internal hyper parameters to obtain models with various compute load and parameter number.

Original: [Equation]
Revised: [Equation]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: ViT .
Revised: ViT .

Original: The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advantage of enlarged receptive field, due to its self-attention layers.
Revised: The ViT (Dosovitskiy et al., 2020) was first proposed as an alternative to CNNs with the advantage of enlarged receptive field, due to its self-attention layers.

Original: However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance.
Revised: However, it lacked desirable properties of CNNs such as inductive biases and translation invariance and required large-scale training datasets to achieve competitive performance.

Original: Data-efficient Image Transformers (DeiT) Touvron et al.
Revised: Data-efficient Image Transformers (DeiT) Touvron et al.

Original: (2021b)
Revised: (2021b)

Original: [Table]
Revised: [Table]

Original: introduced a distillation-based training strategy which significantly improved the classification accuracy.
Revised: introduced a distillation-based training strategy which significantly improved the classification accuracy.

Original: LeViT (Graham et al., 2021) proposed a hybrid model with re-designed multi-layer perceptron (MLP) and self-attention modules that are highly-optimized for fast inference.
Revised: LeViT (Graham et al., 2021) proposed a hybrid model with re-designed multi-layer perceptron (MLP) and self-attention modules that are highly-optimized for fast inference.

Original: Cross-covariance
Revised: Cross-covariance

Original: Image Transformer (XCiT) (Ali et al., 2021) introduced a transposed self-attention module for modeling the interactions of feature channels.
Revised: Image Transformer (XCiT) (Ali et al., 2021) introduced a transposed self-attention module for modeling the interactions of feature channels.

Original: Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs.
Revised: Convolutional vision Transformer (CvT) (Wu et al., 2021) introduced convolutional token embedding layer and Transformer block in a hierarchical architecture to improve the efficiency and accuracy of ViTs.

Original: Conditional Position encoding Vision
Revised: Conditional Position encoding Vision

Original: Transformer (CPVT)
Revised: Transformer (CPVT)

Original: (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token.
Revised: (Chu et al., 2021b) demonstrated improved performance on various tasks such as image classification and object detection by conditioning the position encoding on localized patch token.

Original: Tokens-To-Token Vision Transformer (T2T-ViT)
Revised: Tokens-To-Token Vision Transformer (T2T-ViT)

Original: (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations.
Revised: (Yuan et al., 2021) proposed a transformation layer for aggregating adjacent tokens and establishing image prior by exploiting spatial correlations.

Original: Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the computational efficiency.
Revised: Pyramid Vision Transformer (PVT) (Wang et al., 2021) proposed a hierarchical architecture with patch embedding at the beginning of each stage and spatial dimension reduction to improve the computational efficiency.

Original: Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction.
Revised: Independently, Swin Transformers (Liu et al., 2021) also proposed a hierarchical architecture in which self-attention is computed within local windows which are shifted for region interaction.

Original: Twins Transformer
Revised: Twins Transformer

Original: (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency.
Revised: (Chu et al., 2021a) proposed a spatially separable self-attention with locally-grouped and global sub-sampling modules to improve the efficiency.

Original: EdgeViT (Pan et al., 2022) introduced a lightweight ViT model, with global sparse attention and local aggregations and propagation modules for capturing short and long-range information.
Revised: EdgeViT (Pan et al., 2022) introduced a lightweight ViT model, with global sparse attention and local aggregations and propagation modules for capturing short and long-range information.

Original: Please see the supplementary materials for further discussion.
Revised: Please see the supplementary materials for further discussion.

Original: ConvNet .
Revised: ConvNet .

Original: Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017;
Revised: Since the advent of deep learning, CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Howard et al., 2017;

Original: He et al., 2016; Szegedy et al., 2016; Huang et al., 2017; Hu et al., 2018b) have dominated computer vision benchmarks with SOTA performance.
Revised: He et al., 2016; Szegedy et al., 2016; Huang et al., 2017; Hu et al., 2018b) have dominated computer vision benchmarks with SOTA performance.

Original: Recently, ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.
Revised: Recently, ConvNeXt (Liu et al., 2022) proposed modifications to the architecture of ResNet (He et al., 2016), and achieved competitive benchmarks for classification, detection and segmentation tasks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Image classification.
Revised: Image classification.

Original: For image classification, we trained and tested our model on ImageNet-1K dataset (Deng et al., 2009).
Revised: For image classification, we trained and tested our model on ImageNet-1K dataset (Deng et al., 2009).

Original: To allow for a fair comparison, all GC ViT variants are trained by following training configurations of previous efforts (Liu et al., 2021;
Revised: To allow for a fair comparison, all GC ViT variants are trained by following training configurations of previous efforts (Liu et al., 2021;

Original: Yang et al., 2021b;
Revised: Yang et al., 2021b;

Original: Chu et al., 2021a).
Revised: Chu et al., 2021a).

Original: Specifically, all models are trained with the AdamW (Kingma & Ba, 2014) optimizer forepochs with an initial learning rate of 0 .
Revised: Specifically, all models are trained with the AdamW (Kingma & Ba, 2014) optimizer forepochs with an initial learning rate of 0 .

Original: 001 , weight decay of 0 . 05 , cosine decay scheduler andwarm-up and cool-down epochs, respectively.
Revised: 001 , weight decay of 0 . 05 , cosine decay scheduler andwarm-up and cool-down epochs, respectively.

Original: [Table]
Revised: [Table]

Original: PVT
Revised: PVT

Original: [Table]
Revised: [Table]

Original: Large
Revised: Large

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: DeiT
Revised: DeiT

Original: [Table]
Revised: [Table]

Original: Base
Revised: Base

Original: [Table]
Revised: [Table]

Original: al
Revised: al

Original: [Table]
Revised: [Table]

Original: 2021b
Revised: 2021b

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: T2T
Revised: T2T

Original: [Table]
Revised: [Table]

Original: ViT
Revised: ViT

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Object detection and semantic segmentation For object detection and instance segmentation, we trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using × 3 LR schedule with an initial learning rate of 0 . 0001 , a batch size of 16 and weight decay of 0 .
Revised: Object detection and semantic segmentation For object detection and instance segmentation, we trained our model on MS COCO (Lin et al., 2014) with a Mask-RCNN (He et al., 2017) head, using × 3 LR schedule with an initial learning rate of 0 . 0001 , a batch size of 16 and weight decay of 0 .

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2022), we compared against Tiny, Small and Base model variants using Cascade
Revised: (2022), we compared against Tiny, Small and Base model variants using Cascade

Original: Mask-RCNN but only compared against Tiny variant using Mask-RCNN.
Revised: Mask-RCNN but only compared against Tiny variant using Mask-RCNN.

Original: For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head.
Revised: For semantic segmentation, we used the ADE20K dataset (Zhou et al., 2017) with a UPerNet (Xiao et al., 2018) segmentation head.

Original: Following previous efforts, we used a random crop size of 512 × 512 for the input images.
Revised: Following previous efforts, we used a random crop size of 512 × 512 for the input images.

Original: [Table]
Revised: [Table]

Original: 4. C LASSIFICATION
Revised: 4. C LASSIFICATION

Original: [Table]
Revised: [Table]

Original: We present the ImageNet-1K classification benchmarks in Table 2 and compare against
Revised: We present the ImageNet-1K classification benchmarks in Table 2 and compare against

Original: CNN, ViT and MLP-based models across different model sizes.
Revised: CNN, ViT and MLP-based models across different model sizes.

Original: Our model achieves new SOTA benchmarks in all categories by a large margin.
Revised: Our model achieves new SOTA benchmarks in all categories by a large margin.

Original: Specifically, the proposed GC ViT surpasses similar-sized counterpart models by +0.5% for
Revised: Specifically, the proposed GC ViT surpasses similar-sized counterpart models by +0.5% for

Original: GC ViT-XT (82.0%) compared to CPVT-SmallGAP
Revised: GC ViT-XT (82.0%) compared to CPVT-SmallGAP

Original: (Chu et al., 2021b) (81.5%), +1.2% for
Revised: (Chu et al., 2021b) (81.5%), +1.2% for

Original: GC ViT-T (83.4%) over Focal-T (Yang et al., 2021b) (82.2%), +0.4% for GC ViT-S (83.9%) over Focal-S (Yang et al., 2021b) (83.5%) and +0.7% for GC ViT-B (84.5%) compared to FocalB (Yang et al., 2021b)
Revised: GC ViT-T (83.4%) over Focal-T (Yang et al., 2021b) (82.2%), +0.4% for GC ViT-S (83.9%) over Focal-S (Yang et al., 2021b) (83.5%) and +0.7% for GC ViT-B (84.5%) compared to FocalB (Yang et al., 2021b)

Original: (83.8%), respectively.
Revised: (83.8%), respectively.

Original: In addition, our models demonstrate similar or better performance gap compared to other established benchmarks such as ConvNeXt-T (Liu et al., 2022).
Revised: In addition, our models demonstrate similar or better performance gap compared to other established benchmarks such as ConvNeXt-T (Liu et al., 2022).

Original: Furthermore, as shown in Fig. 1, GC ViT models have better or comparable computational efficiency in terms of number FLOPs over the competing counterpart models.
Revised: Furthermore, as shown in Fig. 1, GC ViT models have better or comparable computational efficiency in terms of number FLOPs over the competing counterpart models.

Original: 4. D ETECTION
Revised: 4. D ETECTION

Original: In Table 3, we present object detection and instance segmentation benchmarks using on MS COCO dataset.
Revised: In Table 3, we present object detection and instance segmentation benchmarks using on MS COCO dataset.

Original: Using a Mask-RCNN head, model with pre-trained GC ViT-T (46.5/41.8) backbone outperforms counterparts with pre-trained ConvNeXt-T (Liu et al., 2022) (46.2/41.7) by +0.3 and +0.1 and
Revised: Using a Mask-RCNN head, model with pre-trained GC ViT-T (46.5/41.8) backbone outperforms counterparts with pre-trained ConvNeXt-T (Liu et al., 2022) (46.2/41.7) by +0.3 and +0.1 and

Original: Swin-T (Liu et al., 2021) (46.0/41.6) by +0.5 and +0.2 in terms of box AP and mask AP, respectively.
Revised: Swin-T (Liu et al., 2021) (46.0/41.6) by +0.5 and +0.2 in terms of box AP and mask AP, respectively.

Original: Using a Cascade Mask-RCNN head, models with pre-trained GC ViT-T (51.1/44.4) and GC ViT-S (52.4/45.4) backbones surpass all competing counterparts.
Revised: Using a Cascade Mask-RCNN head, models with pre-trained GC ViT-T (51.1/44.4) and GC ViT-S (52.4/45.4) backbones surpass all competing counterparts.

Original: Specifically, it outperforms ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +0.7 and +0.7 and ConvNeXt-S (Liu et al., 2022) (51.9/45.0) by +0.
Revised: Specifically, it outperforms ConvNeXt-T (Liu et al., 2022) (50.4/43.7) by +0.7 and +0.7 and ConvNeXt-S (Liu et al., 2022) (51.9/45.0) by +0.

Original: and +0.4 in terms of box AP and mask AP, respectively.
Revised: and +0.4 in terms of box AP and mask AP, respectively.

Original: However, ConvNeXt-B (Liu et al., 2022) (52.7/45.6) demonstrates better performance compared to GC ViT-B (52.2/45.2) by +0.5 and +0.4 in terms of box AP and mask AP, respectively.
Revised: However, ConvNeXt-B (Liu et al., 2022) (52.7/45.6) demonstrates better performance compared to GC ViT-B (52.2/45.2) by +0.5 and +0.4 in terms of box AP and mask AP, respectively.

Original: 4. ADE20K S EMANTIC S EGMENTATION R ESULTS
Revised: 4. ADE20K S EMANTIC S EGMENTATION R ESULTS

Original: We present the semantic segmentation benchmarks on ADE20K in Table 4.
Revised: We present the semantic segmentation benchmarks on ADE20K in Table 4.

Original: Models using pre-trained
Revised: Models using pre-trained

Original: GC ViT-T (46.5), GC ViT-T (48.3) backbones surpass competing models with pre-trained Twins-
Revised: GC ViT-T (46.5), GC ViT-T (48.3) backbones surpass competing models with pre-trained Twins-

Original: [Figure]
Revised: [Figure]

Original: SVT-S (Chu et al., 2021a) (46.2) and Focal-S
Revised: SVT-S (Chu et al., 2021a) (46.2) and Focal-S

Original: (Yang et al., 2021b) (48.0) by +0.3 and +0.3 in terms of mIoU, respectively.
Revised: (Yang et al., 2021b) (48.0) by +0.3 and +0.3 in terms of mIoU, respectively.

Original: Models pre-trained with GC ViT-B (49.0) and Focal-B
Revised: Models pre-trained with GC ViT-B (49.0) and Focal-B

Original: (Yang et al., 2021b) (49.0) both achieve the same segmentation performance.
Revised: (Yang et al., 2021b) (49.0) both achieve the same segmentation performance.

Original: GC ViT significantly outperforms Swin
Revised: GC ViT significantly outperforms Swin

Original: Transformer counterparts, hence demonstrating the effectiveness of the global self-attention.
Revised: Transformer counterparts, hence demonstrating the effectiveness of the global self-attention.

Original:  A BLATION
Revised:  A BLATION

Original: [Table]
Revised: [Table]

Original: Component-wise Analysis.
Revised: Component-wise Analysis.

Original: As shown in Table 5, replacing global self-attention blocks with their local counterparts significantly impacts the performance.
Revised: As shown in Table 5, replacing global self-attention blocks with their local counterparts significantly impacts the performance.

Original: Specifically, segmentation shows the most significant degradation.
Revised: Specifically, segmentation shows the most significant degradation.

Original: Sharing global context query features consistently improves the performance for all tasks, according to
Revised: Sharing global context query features consistently improves the performance for all tasks, according to

Original: Table 5, and decreases the number of parameters and operations.
Revised: Table 5, and decreases the number of parameters and operations.

Original: Such improvements may imply that model generalizability improves by re-using the same tokens throughout the local and global attention blocks.
Revised: Such improvements may imply that model generalizability improves by re-using the same tokens throughout the local and global attention blocks.

Original: Our experiments also show the benefits of employing relative position bias in both local and global self-attention blocks.
Revised: Our experiments also show the benefits of employing relative position bias in both local and global self-attention blocks.

Original:  I NTERPRETABILITY
Revised:  I NTERPRETABILITY

Original: To provide further insights on interpretability of the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017)maps in
Revised: To provide further insights on interpretability of the proposed global self-attention and query tokens, we demonstrate visualization of the learned attention and Grad-CAM (Selvaraju et al., 2017)maps in

Original: Fig.
Revised: Fig.

Original: The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act as an informative source for local attention modules.
Revised: The associated attention distributions uncovered by the global self-attention modules align with image semantics, and hence act as an informative source for local attention modules.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced a novel hierarchical ViT, referred to as GC ViT, which can efficiently capture global context by utilizing global query tokens and interact with local regions.
Revised: In this work, we introduced a novel hierarchical ViT, referred to as GC ViT, which can efficiently capture global context by utilizing global query tokens and interact with local regions.

Original: We have extensively validated the effectiveness of our model on various tasks.
Revised: We have extensively validated the effectiveness of our model on various tasks.

Original: The proposed GC ViT model achieves new SOTA benchmarks for image classification across various model sizes on ImageNet-1K dataset, and surpasses both CNN and ViT-based counterparts by a significant margin.

Original: We have also achieved SOTA or competitive performance for downstream tasks of detection and semantic segmentation on high-resolution images using MS COCO and ADE20K datasets.

Original: A A PPENDIX B GC V I T MODEL CONFIGURATIONS

Original: We present GC ViT-XXT architecture configuration in Table S.1. For comparison, we also include architecture configurations of GC ViT-T, GC ViT-S and GC ViT-B models. GC ViT-XXT is a lightweight modelwhich achieves a competitive Top-1 classification accuracy of 79.8% on ImageNet1K dataset with only 12 M parameters. [Table]
Revised: The proposed GC ViT model achieves new SOTA benchmarks for image classification across various model sizes on ImageNet-1K dataset, and surpasses both CNN and ViT-based counterparts by a significant margin.

Original: C C OMPARISON TO O THER G LOBAL S ELF - ATTENTION M ODULES

Original: Other efforts such as EdgeViT (Pan et al., 2022) in computer vision and BigBird (Zaheer et al., 2020) in NLP have proposed global self-attention in their respective applications.

Original: In this section, we discuss the differences between the proposed global self-attention in GC ViT and these efforts.

Original: EdgeViT : EdgeViT and GC ViT use completely different self-attention blocks.

Original: The EdgeViT uses a series of local aggregation (convolution), sparse attention and local propagation(depthwise convolution), whereas GC ViT only uses an interleaved pattern of local and global self-attention layers without convolution in order to compute self-attention.

Original: The proposed global sparse attention in

Original: EdgeViT and GCViT are competently different.

Original: EdgeViT samples representative tokens and only computes sparse self-attention between these representative tokens with reduced feature size.

Original: On the contrary, GC ViT computes self-attention between the global queries (not just the token) and local keys and values without any subsampling in their respective local regions.

Original: Furthermore, in EdgeViT, only subsampled representative tokens per region interact

Original: In the self-attention module; however, in

Original: GC ViT, the global queries interact with the entire local regions, instead of interacting with each other, and hence provide an effective mechanism for capturing both short and long-range spatial dependencies.

Original: In addition, GC ViT generates global query tokens by using a series of modified Fused MB-Conv from the entire image and without subsampling.

Original: Note that the resolution of global query tokens are the same as local query and values.

Original: However, in EdgeViT: (A) the representative tokens are obtained per local window, not the entire image, and by subsampling and reducing the feature resolution.

Original: Hence, since generated tokens have a lower resolution compared to their respective local windows, this could result in loss of spatial information and impact the effectiveness of self-attention.

Original: Unlike

Original: EdgeViT, the downsampler in GCViT also benefits from modified Fused-MBConv blocks which allows for modeling cross channel interactions and impose more locality and convolutional inductive bias.

Original: [Table]

Original: [Table]

Original: [Table]

Original: BigBird : Bigbird, which is primarily introduced for NLP applications with 1D inputs, has significant differences compared to GC ViT, which is proposed for computer vision with mainly 2D inputs.

Original: Firstly, BigBird uses a combination of random, window and global attention mechanisms, which is different from the proposed local and global self-attention scheme in GC ViT.

Original: In addition,

Original: BigBird does not have any specific mechanisms for extracting global tokens as the existing tokens or additional special tokens can be specified as global tokens.

Original: On the contrary, the global tokens in

Original: GC ViT are extracted by the proposed global query generator module which consists of a series of modified Fused MB-Conv blocks to extract contextual information from the entire input features.

Original: Lastly, BigBird employs a set of global tokens which attend to the entire input sequence; in this case, select global query, key and values attend to local query, key and value tensors.

Original: However, as opposed to this formulation, in GC ViT, the global query tokens attend to local key and value tokens in partitioned windows.

Original: This is due to the fact that attending to the entire input sequence, as done in

Original: BigBird, is not feasible considering the larger size of input features in computer vision.

Original: D A BLATION D. I MPORTANCE OF D OWN - SAMPLER

Original: We studied the effectiveness of various down-sampler blocks in Table S.2. The simplest alternative to our design is a pair of convolutional and maxpooling layers. However, it results in a reduction of ImageNet Top-1 accuracy by 0 .
Revised: We have also achieved SOTA or competitive performance for downstream tasks of detection and semantic segmentation on high-resolution images using MS COCO and ADE20K datasets.

Original: Patch merging is another variant which was introduced in Swin

Original: Transformers (Liu et al., 2021).

Original: However, it reduces the accuracy by 0 .

Original: Another variant is the down-sampler introduced in ConvNeXt (Liu et al., 2022) which is a 3 × 3 strided convolution (s= 2 ).

Original: However, it also reduces the accuracy by 0 .

Original: 3% when used in our setting.

Original: Finally, our down-sampler which consists of a modified Fused-MBConv block and strided convolution and shows the best result.

Original: Importance of the former component is explained by the SE operation which boosts cross channel interaction while keeping number of parameters and FLOPs low.

Original: We conclude that our proposed down-sampler is essential to achieve high accuracy as it introduces convolutional inductive bias.

Original: D. C OMPARISON TO S WIN T RANSFORMER

Original: The most comparable architecture to GC ViT is the Swin Transformer (Liu et al., 2021) with local window splitting.

Original: As a result, we start with a Swin Transformer base design and progressively re-design the components to demonstrate their importance in improving the accuracy.

Original: Firstly, we remove window shifting and predictably observe 1 . 1% drop in accuracy.

Original: Changing distribution of parameters to our design improves the accuracy to 81 .

Original: Such reparametrization includes changing the window size, MLP ratio, number of layers to name but a few, as shown in Table 1.

Original: Adding the

Original: [Table]

Original: [Table]

Original: CNN-based stem of GC ViT to the model provides an additional 0 .

Original: 3% improvement in terms of

Original: ImageNet Top-1 accuracy.

Original: In addition, changing the down-sampling block to the one proposed for

Original: GC ViT improves the performance to 82 .

Original: The last two changes demonstrate the importance of convolutional inductive bias in our model.

Original: Finally, adding the proposed global self-attention improves the accuracy to 83 .

Original: Note that number of layers is not changed to the previous experiment where all layers benefit from only local self-attention.

Original: These experiments validate the effectiveness of GC

Original: ViT components and demonstrate their improvement over Swin Transformers (Liu et al., 2021) as the total number of parameters are kept the same.

Original: E T RAINING D ETAILS

Original: For image classification, GC ViT models were trained using four computational nodes with

Original: Nvidia A100 GPUs.

Original: The total training batch size is 1024 ( 32 per GPU) for GC ViT-S, GC ViT-B and 4096 ( 128 per GPU) for GC ViT-XXT, GC ViT-XT and GC ViT-T.

Original: On average, each model required 32 hours of training with the specified hyper-parameters as indicated in the paper.

Original: All classification models were trained using the timm package Wightman (2019).

Original: Object detection and instance segmentation models as well as semantic segmentation models were trained using one computational node with 8 Nvidia A40 GPUs using a total batch size of 16 , hence a batch size of 2 per GPU.

Original: Detection and instance segmentation models were trained using mmdetection Chen et al.

Original: (2019) package and on average required 56 hours of training.

Original: Semantic segmentation models were trained using mmsegmentation Contributors (2020) package, and on average requiredhours of training.

Original: F C OMPLEXITY A NALYSIS

Original: Given an input feature map of x ∈ R H × W × C at each stage with a window size of h × w , the computational complexity of GC ViT is as follows

Original: [Equation]

Original: The efficient design of global query token generator and other components allows to maintain a similar computational complexity in comparison to Swin Transformer Liu et al.

Original: (2021) while being able to capture long-range information and achieve better higher accuracy for classification and downstream tasks such as detection and segmentation.

Original: G I MAGE N ET V2 B ENCHMARKS

Original: In Table S.4, we have evaluated the performance of GC ViT on ImageNetV2 dataset (Recht et al., 2019) to further measure its robustness.

Original: Specifically, we have used different sampling strategies of

Original: Matched Frequency

Original: and

Original: Threshold-0.7.

Original: These benchmarks demonstrate the competetive performance of GC ViT on ImageNetV2 dataset and validates its effectiveness in robustness and generalizability.

Original: H E FFECT OF G LOBAL C ONTEXT M ODULE

Original: In order to demonstrate the effectiveness of Global Context (GC) module, we use Swin Transformers as the base model and add our propoped GC module.

Original: In this analysis, we remove the window shifting operation from Swin Transformers, since GC module is capable of modeling cross-region interactions.

Original: As shown in Table S.5, addition of GC module improves the ImageNet Top-1 accuracy by +0 . 9% and +0 .

Original: 7% for Swin Transformers Tiny and Small variants respectively.