Original: Inception Transformer
Revised: Inception Transformer

Original: Abstract
Revised: Abstract

Original: Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information.
Revised: Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information.

Original: To tackle this issue, we present a novel and general-purpose Inception Transformer , or iFormer for short, that effectively learns comprehensive features with both high- and low-frequency information in visual data.
Revised: To tackle this issue, we present a novel and general-purpose Inception Transformer , or iFormer for short, that effectively learns comprehensive features with both high- and low-frequency information in visual data.

Original: Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to Transformers.
Revised: Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to Transformers.

Original: Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high- and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range.
Revised: Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high- and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range.

Original: Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e. , gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high- and lowfrequency components across different layers.
Revised: Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e. , gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high- and lowfrequency components across different layers.

Original: We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation.
Revised: We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation.

Original: For example, our iFormer-S hits the top-1 accuracy of 83 .
Revised: For example, our iFormer-S hits the top-1 accuracy of 83 .

Original: 4% on ImageNet-1K, much higher than DeiT-S by 3 . 6% , and even slightly better than much bigger model Swin-B ( 83 .
Revised: 4% on ImageNet-1K, much higher than DeiT-S by 3 . 6% , and even slightly better than much bigger model Swin-B ( 83 .

Original: with only 1/4 parameters and 1/3 FLOPs.
Revised: with only 1/4 parameters and 1/3 FLOPs.

Original: Code and models are released at https://github.com/sail-sg/iFormer .
Revised: Code and models are released at https://github.com/sail-sg/iFormer .

Original:  Introduction
Revised:  Introduction

Original: Transformer [1] has taken the natural language processing (NLP) domain by storm, achieving surprisingly high performance in many NLP tasks, e.g. , machine translation [2] and question-answering [3].
Revised: Transformer [1] has taken the natural language processing (NLP) domain by storm, achieving surprisingly high performance in many NLP tasks, e.g. , machine translation [2] and question-answering [3].

Original: This is largely attributed to its strong capability of modeling long-range dependencies in the data with self-attention mechanism.
Revised: This is largely attributed to its strong capability of modeling long-range dependencies in the data with self-attention mechanism.

Original: Its success has led researchers to investigate its adaptation to the computer vision field, and Vision Transformer (ViT) [4] is a pioneer.
Revised: Its success has led researchers to investigate its adaptation to the computer vision field, and Vision Transformer (ViT) [4] is a pioneer.

Original: This architecture is directly inherited from NLP [1], but applied to image classification with raw image patches as input.
Revised: This architecture is directly inherited from NLP [1], but applied to image classification with raw image patches as input.

Original: Later, many ViT variants [5–13] have been developed to boost performance or scale to a wider range of vision tasks, e.g. , object detection [10, 11] and segmentation [12, 13].
Revised: Later, many ViT variants [5–13] have been developed to boost performance or scale to a wider range of vision tasks, e.g. , object detection [10, 11] and segmentation [12, 13].

Original: ViT and its variants are highly capable of capturing low-frequencies in the visual data [14], mainly including global shapes and structures of a scene or object, but are not very powerful for learning high-frequencies, mainly including local edges and textures.
Revised: ViT and its variants are highly capable of capturing low-frequencies in the visual data [14], mainly including global shapes and structures of a scene or object, but are not very powerful for learning high-frequencies, mainly including local edges and textures.

Original: This can be intuitively explained: selfattention, the main operation used in ViTs to exchange information among non-overlap patch tokens, is a global operation and much more capable of capturing global information (low frequencies) in the
Revised: This can be intuitively explained: selfattention, the main operation used in ViTs to exchange information among non-overlap patch tokens, is a global operation and much more capable of capturing global information (low frequencies) in the

Original: ViT iFormer (a)
Revised: ViT iFormer (a)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Frequency -
Revised: Frequency -

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: L o g a m p li t u d e iFormer
Revised: L o g a m p li t u d e iFormer

Original: ViT (b)
Revised: ViT (b)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: PVT
Revised: PVT

Original: [Figure]
Revised: [Figure]

Original: (c) show data than local information (high frequencies).
Revised: (c) show data than local information (high frequencies).

Original: As shown in Fig. 1(a) and 1(b), the Fourier spectrum and relative log amplitudes of the Fourier show that ViT tends to well capture low-frequency signals but few high-frequency signals.
Revised: As shown in Fig. 1(a) and 1(b), the Fourier spectrum and relative log amplitudes of the Fourier show that ViT tends to well capture low-frequency signals but few high-frequency signals.

Original: This observation also accords with the empirical results in [14], which shows ViT presents the characteristics of low-pass filters.
Revised: This observation also accords with the empirical results in [14], which shows ViT presents the characteristics of low-pass filters.

Original: This low-frequency preferability impairs the performance of ViTs, as 1) low-frequency information filling in all the layers may deteriorate high-frequency components, e.g. , local textures, and weakens modeling capability of
Revised: This low-frequency preferability impairs the performance of ViTs, as 1) low-frequency information filling in all the layers may deteriorate high-frequency components, e.g. , local textures, and weakens modeling capability of

Original: ViTs; 2) high-frequency information is also discriminative and can benefit many tasks, e.g. , (finegrained) classification.
Revised: ViTs; 2) high-frequency information is also discriminative and can benefit many tasks, e.g. , (finegrained) classification.

Original: Actually, human visual system extracts visual elementary features at different frequencies [15–17]: low frequency provides global information about a visual stimulus, and high frequency conveys local spatial changes in the image ( e.g. , local edges/textures).
Revised: Actually, human visual system extracts visual elementary features at different frequencies [15–17]: low frequency provides global information about a visual stimulus, and high frequency conveys local spatial changes in the image ( e.g. , local edges/textures).

Original: Hence, it is necessary to develop a new ViT architecture for capturing both high and low frequencies in the visual data.
Revised: Hence, it is necessary to develop a new ViT architecture for capturing both high and low frequencies in the visual data.

Original: CNNs are the most fundamental backbone for general vision tasks.
Revised: CNNs are the most fundamental backbone for general vision tasks.

Original: Unlike ViTs, they cover more local information through local convolution within the receptive fields, thus effectively extracting high-frequency representations [19, 20].
Revised: Unlike ViTs, they cover more local information through local convolution within the receptive fields, thus effectively extracting high-frequency representations [19, 20].

Original: Recent studies [21–25] have integrated CNNs and ViTs considering their complementary advantages.
Revised: Recent studies [21–25] have integrated CNNs and ViTs considering their complementary advantages.

Original: Some methods [21, 22, 24, 25] stack convolution and attention layers in a serial manner to inject the local information into global context.
Revised: Some methods [21, 22, 24, 25] stack convolution and attention layers in a serial manner to inject the local information into global context.

Original: Unfortunately, this serial manner only models one type of dependency, either global or local, in one layer, and discards the global information during locality modeling, or vice versa.
Revised: Unfortunately, this serial manner only models one type of dependency, either global or local, in one layer, and discards the global information during locality modeling, or vice versa.

Original: Other works [23, 26] adopt parallel attention and convolution to learn global and local dependencies of the input at the same time.
Revised: Other works [23, 26] adopt parallel attention and convolution to learn global and local dependencies of the input at the same time.

Original: However, it is found in [27] that part of the channels are for processing local information and the other for global modeling, meaning current parallel structures have information redundancy if processing all channels in each branch.
Revised: However, it is found in [27] that part of the channels are for processing local information and the other for global modeling, meaning current parallel structures have information redundancy if processing all channels in each branch.

Original: To address this issue, we propose a simple and efficient Inception Transformer (iFormer) , as shown in Fig. 2, which grafts the merit of CNNs for capturing high-frequencies to ViTs.
Revised: To address this issue, we propose a simple and efficient Inception Transformer (iFormer) , as shown in Fig. 2, which grafts the merit of CNNs for capturing high-frequencies to ViTs.

Original: The key component in iFormer is an Inception token mixer as shown in Fig.
Revised: The key component in iFormer is an Inception token mixer as shown in Fig.

Original: This Inception mixer aims to augment the perception capability of ViTs in the frequency spectrum by capturing both high and low frequencies in the data.
Revised: This Inception mixer aims to augment the perception capability of ViTs in the frequency spectrum by capturing both high and low frequencies in the data.

Original: To this end, the Inception mixer first splits the input feature along the channel dimension, and then feeds the split components into high-frequency mixer and low-frequency mixer respectively.
Revised: To this end, the Inception mixer first splits the input feature along the channel dimension, and then feeds the split components into high-frequency mixer and low-frequency mixer respectively.

Original: Here the high-frequency mixer consists of a max-pooling operation and a parallel convolution operation, while the low-frequency mixer is implemented by a vanilla self-attention in ViTs.
Revised: Here the high-frequency mixer consists of a max-pooling operation and a parallel convolution operation, while the low-frequency mixer is implemented by a vanilla self-attention in ViTs.

Original: In this way, our iFormer can effectively capture particular frequency information on the corresponding channel, and thus learn more comprehensive features within a wide frequency range compared with vanilla ViTs, which can be clearly observed in Fig. 1(a) and 1(b).
Revised: In this way, our iFormer can effectively capture particular frequency information on the corresponding channel, and thus learn more comprehensive features within a wide frequency range compared with vanilla ViTs, which can be clearly observed in Fig. 1(a) and 1(b).

Original: Moreover, we find that lower layers often need more local information, while higher layers desire more global information, which also accords with the observations in [27].
Revised: Moreover, we find that lower layers often need more local information, while higher layers desire more global information, which also accords with the observations in [27].

Original: This is because, like in human visual system, the details in high frequency components help lower layers to capture visual elementary features and also to gradually gather local information for having a global understanding of the input.
Revised: This is because, like in human visual system, the details in high frequency components help lower layers to capture visual elementary features and also to gradually gather local information for having a global understanding of the input.

Original: Inspired by this, we design a frequency ramp structure.
Revised: Inspired by this, we design a frequency ramp structure.

Original: In particular, from lower to higher layers, we gradually feed more channel dimensions to low-frequency mixer and fewer channel dimensions to high-frequency mixer.
Revised: In particular, from lower to higher layers, we gradually feed more channel dimensions to low-frequency mixer and fewer channel dimensions to high-frequency mixer.

Original: This structure can trade-off high-frequency and low-frequency components across all layers.
Revised: This structure can trade-off high-frequency and low-frequency components across all layers.

Original: Its effectiveness has been verified by experimental results in Sec. 4.
Revised: Its effectiveness has been verified by experimental results in Sec. 4.

Original: Experimental results show that iFormer surpasses state-of-the-art ViTs and CNNs on several vision tasks, including image classification, object detection and segmentation.
Revised: Experimental results show that iFormer surpasses state-of-the-art ViTs and CNNs on several vision tasks, including image classification, object detection and segmentation.

Original: For example, as shown in Fig.
Revised: For example, as shown in Fig.

Original: 1(c), with different model sizes, iFormer makes consistent improvements over popular frameworks on ImageNet-1K [28], e.g. , DeiT [29], Swin [5] and ConvNeXt [30].
Revised: 1(c), with different model sizes, iFormer makes consistent improvements over popular frameworks on ImageNet-1K [28], e.g. , DeiT [29], Swin [5] and ConvNeXt [30].

Original: Meanwhile, iFormer outperforms recent frameworks on COCO [31] detection and ADE20K [32] segmentation.
Revised: Meanwhile, iFormer outperforms recent frameworks on COCO [31] detection and ADE20K [32] segmentation.

Original:  Related work
Revised:  Related work

Original: Transformers [1] are firstly proposed for machine translation tasks and then become popular in other tasks like natural language understanding [33–35] and generation [36, 37] in NLP domain, as well as image classification [18, 29, 38], object detection [6, 39, 40] and semantic segmentation [41, 42] in computer vision.
Revised: Transformers [1] are firstly proposed for machine translation tasks and then become popular in other tasks like natural language understanding [33–35] and generation [36, 37] in NLP domain, as well as image classification [18, 29, 38], object detection [6, 39, 40] and semantic segmentation [41, 42] in computer vision.

Original: The attention module in Transformers has an outstanding ability to capture global dependency, but it makes the models produce similar representations across layers [27].
Revised: The attention module in Transformers has an outstanding ability to capture global dependency, but it makes the models produce similar representations across layers [27].

Original: Moreover, self-attention mainly captures low-frequency information and tends to neglect high-frequency components related to the detailed information [14].
Revised: Moreover, self-attention mainly captures low-frequency information and tends to neglect high-frequency components related to the detailed information [14].

Original: CNNs [43–47] are the de-facto model for vision tasks due to their outstanding ability to model local dependency [47–49] as well as extract high-frequency [19, 50].
Revised: CNNs [43–47] are the de-facto model for vision tasks due to their outstanding ability to model local dependency [47–49] as well as extract high-frequency [19, 50].

Original: With these advantages, CNNs are rapidly introduced into Transformers in a serial or parallel manner [23–26, 51–53].
Revised: With these advantages, CNNs are rapidly introduced into Transformers in a serial or parallel manner [23–26, 51–53].

Original: For serial methods, convolutions are applied at different positions of the Transformer.
Revised: For serial methods, convolutions are applied at different positions of the Transformer.

Original: CvT [25] and PVT-v [54] replace the hard patch embedding with a layer of overlapping convolution.
Revised: CvT [25] and PVT-v [54] replace the hard patch embedding with a layer of overlapping convolution.

Original: LV-ViT [51], LeViT [55] and ViT C [21] further stack several layers of convolutions as the stem for models, which is found helpful in training and achieving better performance.
Revised: LV-ViT [51], LeViT [55] and ViT C [21] further stack several layers of convolutions as the stem for models, which is found helpful in training and achieving better performance.

Original: Besides the stem, ViT-hybrid [18],
Revised: Besides the stem, ViT-hybrid [18],

Original: CoAtNet [24], Hybrid-MS [56] and UniFormer [22] design early stages with convolution layers.
Revised: CoAtNet [24], Hybrid-MS [56] and UniFormer [22] design early stages with convolution layers.

Original: However, the combination of convolution and attention in a serial order means each layer can only process either high or low frequency and neglects the other part.
Revised: However, the combination of convolution and attention in a serial order means each layer can only process either high or low frequency and neglects the other part.

Original: To enable each layer to process different frequencies, we adopt the parallel manner to combine convolution and attention in a token mixer.
Revised: To enable each layer to process different frequencies, we adopt the parallel manner to combine convolution and attention in a token mixer.

Original: Compared with serial methods, there are not many works combining attention and convolution in a parallel manner in literature.
Revised: Compared with serial methods, there are not many works combining attention and convolution in a parallel manner in literature.

Original: CoaT [26] and ViTAE [23] introduce convolution as a branch parallel to attention and utilize elementwise sum to merge the output of the two branches.
Revised: CoaT [26] and ViTAE [23] introduce convolution as a branch parallel to attention and utilize elementwise sum to merge the output of the two branches.

Original: However, Raghu et al.
Revised: However, Raghu et al.

Original: find that some channels tend to extract local dependency while others are for modeling global information [27], indicating redundancy for the current parallel mechanism to process all channels in different branches.
Revised: find that some channels tend to extract local dependency while others are for modeling global information [27], indicating redundancy for the current parallel mechanism to process all channels in different branches.

Original: In contrast, we split channels into branches of high and low frequencies.
Revised: In contrast, we split channels into branches of high and low frequencies.

Original: GLiT [53] also adopt parallel manner but it directly concatenate the features from convolution and attention branches as the mixer output, lacking the fusion of features in different frequencies.
Revised: GLiT [53] also adopt parallel manner but it directly concatenate the features from convolution and attention branches as the mixer output, lacking the fusion of features in different frequencies.

Original: Instead, we design a explicit fusion module to merge the outputs from low- and high-frequency branches.
Revised: Instead, we design a explicit fusion module to merge the outputs from low- and high-frequency branches.

Original:  Method 3.1 Revisit Vision Transformer
Revised:  Method 3.1 Revisit Vision Transformer

Original: We first revisit the Vision Transformer.
Revised: We first revisit the Vision Transformer.

Original: For vision tasks, Transformers first split the input image into a sequence of tokens, and each patch token is projected into a hidden representation vector with a leaner layer, denoted as { x 1 , x 2 , ..., x N } or X ∈ R N × C , where N is the number of patch tokens and C indicates the dimension of features.
Revised: For vision tasks, Transformers first split the input image into a sequence of tokens, and each patch token is projected into a hidden representation vector with a leaner layer, denoted as { x 1 , x 2 , ..., x N } or X ∈ R N × C , where N is the number of patch tokens and C indicates the dimension of features.

Original: Then, all of the tokens are combined with a positional embedding and fed into the Transformer layers that contain multi-head self-attention (MSA) and a feed-forward network (FFN).
Revised: Then, all of the tokens are combined with a positional embedding and fed into the Transformer layers that contain multi-head self-attention (MSA) and a feed-forward network (FFN).

Original: In MSA, the attention-based mixer exchanges information between all patch tokens so that it strongly focuses on aggregating the global dependency across all layers.
Revised: In MSA, the attention-based mixer exchanges information between all patch tokens so that it strongly focuses on aggregating the global dependency across all layers.

Original: However, excessive propagation of global information would strengthen the low-frequency representation.
Revised: However, excessive propagation of global information would strengthen the low-frequency representation.

Original: It can be seen from the visualization of Fourier spectrum in Fig.
Revised: It can be seen from the visualization of Fourier spectrum in Fig.

Original: 1(a) that low-frequency information dominates the representations of ViT [18].
Revised: 1(a) that low-frequency information dominates the representations of ViT [18].

Original: This actually impairs the performance of ViTs, as it may deteriorate the high-frequency components, e.g. , local textures, and weakens the modeling capability of ViTs [14].
Revised: This actually impairs the performance of ViTs, as it may deteriorate the high-frequency components, e.g. , local textures, and weakens the modeling capability of ViTs [14].

Original: In the visual data, high-frequency information is also discriminative and can benefit many tasks iFormerBlocks iFormerBlocks iFormerBlocks iFormerBlocks
Revised: In the visual data, high-frequency information is also discriminative and can benefit many tasks iFormerBlocks iFormerBlocks iFormerBlocks iFormerBlocks

Original: P a t c
Revised: P a t c

Original: h e m b e d i n g
Revised: h e m b e d i n g

Original: P a t c
Revised: P a t c

Original: h e m b e d i n g
Revised: h e m b e d i n g

Original: P a t c
Revised: P a t c

Original: h e m b e d i n g
Revised: h e m b e d i n g

Original: P a t c
Revised: P a t c

Original: h e m b e d i n g
Revised: h e m b e d i n g

Original: Input 3×𝐻×𝑊
Revised: Input 3×𝐻×𝑊

Original: Stage 𝐶 !
Revised: Stage 𝐶 !

Original: ×𝐻4×𝑊
Revised: ×𝐻4×𝑊

Original: Stage 𝐶 " ×𝐻8×𝑊
Revised: Stage 𝐶 " ×𝐻8×𝑊

Original: Stage 𝐶 # × 𝐻 16×𝑊
Revised: Stage 𝐶 # × 𝐻 16×𝑊

Original: Stage 𝐶 $ ×
Revised: Stage 𝐶 $ ×

Original: 𝐻 32×𝑊
Revised: 𝐻 32×𝑊

Original: Norm
Revised: Norm

Original: InceptionMixer
Revised: InceptionMixer

Original: Norm
Revised: Norm

Original: FFN
Revised: FFN

Original: Hence, to address the issue, we propose a simple and efficient Inception Transformer, as shown in Fig.
Revised: Hence, to address the issue, we propose a simple and efficient Inception Transformer, as shown in Fig.

Original: 2, with two key novelties, i.e. , Inception mixer and frequency ramp structure.
Revised: 2, with two key novelties, i.e. , Inception mixer and frequency ramp structure.

Original: 3.2 Inception token mixer
Revised: 3.2 Inception token mixer

Original: [Figure]
Revised: [Figure]

Original: Linear Linear
Revised: Linear Linear

Original: AvePool
Revised: AvePool

Original: Attention
Revised: Attention

Original: Upsample
Revised: Upsample

Original: DWConv 𝑋 !"
Revised: DWConv 𝑋 !"

Original: 𝑋 !
Revised: 𝑋 !

Original: # 𝑋 $ 𝑌 $ 𝑌 !
Revised: # 𝑋 $ 𝑌 $ 𝑌 !

Original: # 𝑌 !"
Revised: # 𝑌 !"

Original: We propose an Inception mixer to graft the powerful capability of
Revised: We propose an Inception mixer to graft the powerful capability of

Original: CNNs for extracting high-frequency representation to Transformers.
Revised: CNNs for extracting high-frequency representation to Transformers.

Original: Its detailed architecture is depicted in Fig.
Revised: Its detailed architecture is depicted in Fig.

Original: We use the name of “Inception" since the token mixer is highly inspired by the Inception module [46, 57–59] with multiple branches.
Revised: We use the name of “Inception" since the token mixer is highly inspired by the Inception module [46, 57–59] with multiple branches.

Original: Instead of directly feeding image tokens into the MSA mixer, the Inception mixer first splits the input feature along the channel dimension, and then respectively feeds the split components into high-frequency mixer and low-frequency mixer.
Revised: Instead of directly feeding image tokens into the MSA mixer, the Inception mixer first splits the input feature along the channel dimension, and then respectively feeds the split components into high-frequency mixer and low-frequency mixer.

Original: Here the high-frequency mixer consists of a max-pooling operation and a parallel convolution operation, while the low-frequency mixer is implemented by a self-attention.
Revised: Here the high-frequency mixer consists of a max-pooling operation and a parallel convolution operation, while the low-frequency mixer is implemented by a self-attention.

Original: Technically, given the input feature map X ∈ R N × C , it is factorized X into X h ∈ R N × C h and X l ∈ R N × C l along the channel dimension, where C h + C l = C .
Revised: Technically, given the input feature map X ∈ R N × C , it is factorized X into X h ∈ R N × C h and X l ∈ R N × C l along the channel dimension, where C h + C l = C .

Original: Then, X h and X l are assigned to high-frequency mixer and low-frequency mixer respectively.
Revised: Then, X h and X l are assigned to high-frequency mixer and low-frequency mixer respectively.

Original: High-frequency mixer.
Revised: High-frequency mixer.

Original: Considering the sharp sensitiveness of the maximum filter and the detail perception of convolution operation, we propose a parallel structure to learn the high-frequency components.
Revised: Considering the sharp sensitiveness of the maximum filter and the detail perception of convolution operation, we propose a parallel structure to learn the high-frequency components.

Original: We divide the input X h into
Revised: We divide the input X h into

Original: X h 1 ∈ R N ×
Revised: X h 1 ∈ R N ×

Original: Ch 2 and X h 2 ∈ R N ×
Revised: Ch 2 and X h 2 ∈ R N ×

Original: Ch 2 along the channel.
Revised: Ch 2 along the channel.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, X h 1 is embedded with a max-pooling and a linear layer [46], and X h 2 is fed into a linear and a depthwise convolution layer [60–62]:
Revised: 3, X h 1 is embedded with a max-pooling and a linear layer [46], and X h 2 is fed into a linear and a depthwise convolution layer [60–62]:

Original: [Equation]
Revised: [Equation]

Original: where Y h 1 and Y h 2 denote the outputs of high-frequency mixers.
Revised: where Y h 1 and Y h 2 denote the outputs of high-frequency mixers.

Original: Finally, the outputs of low- and high-frequency mixers are concatenated along the channel dimension:
Revised: Finally, the outputs of low- and high-frequency mixers are concatenated along the channel dimension:

Original: [Equation]
Revised: [Equation]

Original: The upsample operation in Eq.
Revised: The upsample operation in Eq.

Original: (7) selects the value of the nearest point for each position to be interpolated regardless of any other points, which results in excessive smoothness between adjacent tokens.
Revised: (7) selects the value of the nearest point for each position to be interpolated regardless of any other points, which results in excessive smoothness between adjacent tokens.

Original: We design a fusion module to elegantly overcome this issue, i.e. , a depthwise convolution exchanging information between patches, while keeping a cross-channel linear layer that works per location like in previous Transformers.
Revised: We design a fusion module to elegantly overcome this issue, i.e. , a depthwise convolution exchanging information between patches, while keeping a cross-channel linear layer that works per location like in previous Transformers.

Original: The final output can be expressed as
Revised: The final output can be expressed as

Original: [Equation]
Revised: [Equation]

Original: Like the vanilla Transformer, our iFormer is equipped with a feed-forward network (FFN), and differently it also incorporates the above Inception token mixer (ITM); LayerNorm (LN) is applied before ITM and FFN.
Revised: Like the vanilla Transformer, our iFormer is equipped with a feed-forward network (FFN), and differently it also incorporates the above Inception token mixer (ITM); LayerNorm (LN) is applied before ITM and FFN.

Original: Hence the Inception Transformer block is formally defined as
Revised: Hence the Inception Transformer block is formally defined as

Original: [Equation]
Revised: [Equation]

Original: Low-frequency mixer.
Revised: Low-frequency mixer.

Original: We use the vanilla multi-head self-attention to communicate information among all tokens for the low-frequency mixer.
Revised: We use the vanilla multi-head self-attention to communicate information among all tokens for the low-frequency mixer.

Original: Despite the strong capability of the attention for learning global representation, the large resolution of feature maps would bring large computation cost in lower layers.
Revised: Despite the strong capability of the attention for learning global representation, the large resolution of feature maps would bring large computation cost in lower layers.

Original: We therefore simply utilize an average pooling layer to reduce the spatial scale of X l before the attention operation and an upsample layer to recover the original spatial dimension after the attention.
Revised: We therefore simply utilize an average pooling layer to reduce the spatial scale of X l before the attention operation and an upsample layer to recover the original spatial dimension after the attention.

Original: This design largely reduces the computational overhead and makes the attention operation focus on embedding global information.
Revised: This design largely reduces the computational overhead and makes the attention operation focus on embedding global information.

Original: This branch can be defined as
Revised: This branch can be defined as

Original: [Equation]
Revised: [Equation]

Original: where Y l is the output of low-frequency mixer.
Revised: where Y l is the output of low-frequency mixer.

Original: Note that the kernel size and stride for the pooling and upsample layers are set to 2 only at the first two stages.
Revised: Note that the kernel size and stride for the pooling and upsample layers are set to 2 only at the first two stages.

Original: 3.3 Frequency ramp structure
Revised: 3.3 Frequency ramp structure

Original: In the general visual frameworks, bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, i.e. , the hierarchical representations of ResNet [47].
Revised: In the general visual frameworks, bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, i.e. , the hierarchical representations of ResNet [47].

Original: Like humans, by capturing the details in high frequency components, lower layers can capture visual elementary features, and also gradually gather local information to achieve a global understanding of the input.
Revised: Like humans, by capturing the details in high frequency components, lower layers can capture visual elementary features, and also gradually gather local information to achieve a global understanding of the input.

Original: We are inspired to design a frequency ramp structure which gradually splits more channel dimensions from lower to higher layers to low-frequency mixer and thus leave fewer channel dimensions to high-frequency mixer.
Revised: We are inspired to design a frequency ramp structure which gradually splits more channel dimensions from lower to higher layers to low-frequency mixer and thus leave fewer channel dimensions to high-frequency mixer.

Original: Specifically, as shown in Fig.
Revised: Specifically, as shown in Fig.

Original: 2, our backbone has four stages with different channel and spatial dimensions.
Revised: 2, our backbone has four stages with different channel and spatial dimensions.

Original: For each blocks, we define a channel ratio to better balance the high-frequency and low frequency components, i.e. , C h C and C l C , where C h C + C l C
Revised: For each blocks, we define a channel ratio to better balance the high-frequency and low frequency components, i.e. , C h C and C l C , where C h C + C l C

Original: =
Revised: =

Original: In the proposed frequency ramp structure, C h C gradually decreases from shallow to deep layers, while C l C gradually increases.
Revised: In the proposed frequency ramp structure, C h C gradually decreases from shallow to deep layers, while C l C gradually increases.

Original: Hence, with the flexible frequency ramp structure, iFormer can effectively trade-off high- and low-frequency components across all layers.
Revised: Hence, with the flexible frequency ramp structure, iFormer can effectively trade-off high- and low-frequency components across all layers.

Original: The configuration of different iFormer models will be described in the appendix.
Revised: The configuration of different iFormer models will be described in the appendix.

Original:  Experiments
Revised:  Experiments

Original: We evaluate our iFormer on several vision benchmark tasks, i.e. , image classification, object detection and semantic segmentation, by comparing it with representative ViTs, CNNs and their hybrid variants.
Revised: We evaluate our iFormer on several vision benchmark tasks, i.e. , image classification, object detection and semantic segmentation, by comparing it with representative ViTs, CNNs and their hybrid variants.

Original: Ablation analysis is also conducted to show the contribution of each novelty in our method.
Revised: Ablation analysis is also conducted to show the contribution of each novelty in our method.

Original: More results will be reported in the appendix.
Revised: More results will be reported in the appendix.

Original: 4.1 Results on image classification
Revised: 4.1 Results on image classification

Original: Setup.
Revised: Setup.

Original: For image classification, we evaluate iFormer on the ImageNet dataset [28].
Revised: For image classification, we evaluate iFormer on the ImageNet dataset [28].

Original: We train the iFormer model with the standard procedure in [6, 22, 29].
Revised: We train the iFormer model with the standard procedure in [6, 22, 29].

Original: Specifically, we use AdamW optimizer with an initial learning rate 1 × 10 − 3 via cosine decay [70], a momentum of 0.9, and a weight decay of 0.05.
Revised: Specifically, we use AdamW optimizer with an initial learning rate 1 × 10 − 3 via cosine decay [70], a momentum of 0.9, and a weight decay of 0.05.

Original: We set the training epoch number as 300 and the input size as 224 × 224.
Revised: We set the training epoch number as 300 and the input size as 224 × 224.

Original: We adopt the same data augmentations and regularization methods in DeiT [29] for fair comparison.
Revised: We adopt the same data augmentations and regularization methods in DeiT [29] for fair comparison.

Original: We also use LayerScale [71] to train deep models.
Revised: We also use LayerScale [71] to train deep models.

Original: Like previous studies [5, 67], we further fine tune iFormer on the input size of 384 × 384 , with the weight decay of 1 × 10 − 8 , learning rate of 1 × 10 − 5 , batch size of 512.
Revised: Like previous studies [5, 67], we further fine tune iFormer on the input size of 384 × 384 , with the weight decay of 1 × 10 − 8 , learning rate of 1 × 10 − 5 , batch size of 512.

Original: For fairness, we adopt Timm [72] to implement and train iFormer.
Revised: For fairness, we adopt Timm [72] to implement and train iFormer.

Original: Results.
Revised: Results.

Original: Table 1 summarizes the image classification accuracy of all compared methods on ImageNet.
Revised: Table 1 summarizes the image classification accuracy of all compared methods on ImageNet.

Original: For the small model size ( ∼ 20M), our iFormer surpasses both the SoTA ViTs and hybrid ViTs, although some ViTs, e.g. , Swin [5], Focal [64] and CSwin [65], actually already introduce convolutionlike inductive bias into their architectures, and hybrid ViTs directly integrate convolution into ViTs.
Revised: For the small model size ( ∼ 20M), our iFormer surpasses both the SoTA ViTs and hybrid ViTs, although some ViTs, e.g. , Swin [5], Focal [64] and CSwin [65], actually already introduce convolutionlike inductive bias into their architectures, and hybrid ViTs directly integrate convolution into ViTs.

Original: Specifically, our iFormer-S respectively gains 0 .
Revised: Specifically, our iFormer-S respectively gains 0 .

Original: 7% and 0 .
Revised: 7% and 0 .

Original: 5% top-1 accuracy advantage over SoTA
Revised: 5% top-1 accuracy advantage over SoTA

Original: [Table]
Revised: [Table]

Original: l a r g e m od e l s i ze ( ∼
Revised: l a r g e m od e l s i ze ( ∼

Original: M )
Revised: M )

Original: [Table]
Revised: [Table]

Original: ViTs ( i.e. , CSwin-T) and hybrid ViTs ( i.e. , UniFormer-S), while enjoying the same or smaller model size.
Revised: ViTs ( i.e. , CSwin-T) and hybrid ViTs ( i.e. , UniFormer-S), while enjoying the same or smaller model size.

Original: For the medium model size ( ∼ 50M), iFormer-B achieves 84.6% top-1 accuracy, and improves over the SoTA ViTs and hybrid ViTs with similar model sizes by significant margins 1.0% and 0.7% respectively.
Revised: For the medium model size ( ∼ 50M), iFormer-B achieves 84.6% top-1 accuracy, and improves over the SoTA ViTs and hybrid ViTs with similar model sizes by significant margins 1.0% and 0.7% respectively.

Original: For CNNs, similar to comparison results on medium model size, our iFormer-B outperforms ConvNeXt-S by 1 . 5% .
Revised: For CNNs, similar to comparison results on medium model size, our iFormer-B outperforms ConvNeXt-S by 1 . 5% .

Original: As for the large mode ( ∼ 100M), one can observe similar results on small and medium model sizes.
Revised: As for the large mode ( ∼ 100M), one can observe similar results on small and medium model sizes.

Original: Table 2 reports the fine-tuning accuracy on the larger resolution, i.e. , 384 × 384 .
Revised: Table 2 reports the fine-tuning accuracy on the larger resolution, i.e. , 384 × 384 .

Original: One can observe that iFormer consistently outperforms the counterparts by a significant margin across different computation settings.
Revised: One can observe that iFormer consistently outperforms the counterparts by a significant margin across different computation settings.

Original: These results clearly demonstrate the advantages of iFormer on image classifications.
Revised: These results clearly demonstrate the advantages of iFormer on image classifications.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.2 Results on object detection and instance segmentation
Revised: 4.2 Results on object detection and instance segmentation

Original: Setup.
Revised: Setup.

Original: We evaluate iFormer on the COCO object detection and instance segmentation tasks [31], where the models are trained on 118K images and evaluated on validation set with 5K images.
Revised: We evaluate iFormer on the COCO object detection and instance segmentation tasks [31], where the models are trained on 118K images and evaluated on validation set with 5K images.

Original: Here, we use iFormer as the backbone in Mask R-CNN [75].
Revised: Here, we use iFormer as the backbone in Mask R-CNN [75].

Original: In the training phase, we use iFormer pretrained on ImageNet to initialize the detector, and adopt AdamW to train with an initial learning rate of 1 × 10 − 4 , a batch size of 16, and 1 × training schedule with 12 epochs.
Revised: In the training phase, we use iFormer pretrained on ImageNet to initialize the detector, and adopt AdamW to train with an initial learning rate of 1 × 10 − 4 , a batch size of 16, and 1 × training schedule with 12 epochs.

Original: For training, the input images are resized to be 800 pixels on the shorter side an no more than 1,333 pixels on the longer side.
Revised: For training, the input images are resized to be 800 pixels on the shorter side an no more than 1,333 pixels on the longer side.

Original: For the test image, its shorter side is fixed to 800 pixels.
Revised: For the test image, its shorter side is fixed to 800 pixels.

Original: All experiments are implemented on mmdetection [78] codebase.
Revised: All experiments are implemented on mmdetection [78] codebase.

Original: Results.
Revised: Results.

Original: Table 3 reports the box mAP (AP b ) and mask mAP (AP m ) of the compared models.
Revised: Table 3 reports the box mAP (AP b ) and mask mAP (AP m ) of the compared models.

Original: Under similar computation configurations, iFormers outperforms all previous backbones.
Revised: Under similar computation configurations, iFormers outperforms all previous backbones.

Original: Specifically, compared with popular ResNet [47] backbones, our iFormer-S brings 8 .
Revised: Specifically, compared with popular ResNet [47] backbones, our iFormer-S brings 8 .

Original: 2 points of AP b and 7 .points AP m improvements over ResNet50.
Revised: 2 points of AP b and 7 .points AP m improvements over ResNet50.

Original: Compared with various Transformer backbones, our iFormers still maintain the performance superiority over their results.
Revised: Compared with various Transformer backbones, our iFormers still maintain the performance superiority over their results.

Original: For example, our iFormer-B surpasses UniFormer-B [22], Swin-S [5] by 0 .
Revised: For example, our iFormer-B surpasses UniFormer-B [22], Swin-S [5] by 0 .

Original: 9 points of AP b and 3 .
Revised: 9 points of AP b and 3 .

Original: 5 points of AP b respectively.
Revised: 5 points of AP b respectively.

Original: [Table]
Revised: [Table]

Original: 4.3 Results on semantic segmentation
Revised: 4.3 Results on semantic segmentation

Original: Setup.
Revised: Setup.

Original: We further evaluate the generality of iFormer through a challenging scene parsing benchmark on semantic segmentation, i.e. ,
Revised: We further evaluate the generality of iFormer through a challenging scene parsing benchmark on semantic segmentation, i.e. ,

Original: ADE20K [32].
Revised: ADE20K [32].

Original: The dataset contains 20K training images and 2K validation images.
Revised: The dataset contains 20K training images and 2K validation images.

Original: We adopt iFormer pretrained on ImageNet as the backbone of the Semantic FPN [79] framework.
Revised: We adopt iFormer pretrained on ImageNet as the backbone of the Semantic FPN [79] framework.

Original: Following PVT [6] and UniFormer [22], we use
Revised: Following PVT [6] and UniFormer [22], we use

Original: AdamW with an initial learning rate of 2 × 10 −with cosine learning rate schedule to train 80k iterations.
Revised: AdamW with an initial learning rate of 2 × 10 −with cosine learning rate schedule to train 80k iterations.

Original: All experiments are implemented on mmsegmentation [80] codebase.
Revised: All experiments are implemented on mmsegmentation [80] codebase.

Original: Results.
Revised: Results.

Original: In Table 4, we report the mIoU results of different backbones.
Revised: In Table 4, we report the mIoU results of different backbones.

Original: On the Semantic
Revised: On the Semantic

Original: FPN [79] framework, our iFormer consistently outperforms previous backbones on this task, including CNNs and (hybrid) ViTs.
Revised: FPN [79] framework, our iFormer consistently outperforms previous backbones on this task, including CNNs and (hybrid) ViTs.

Original: For instance, iFormer-S achieves 48 .
Revised: For instance, iFormer-S achieves 48 .

Original: 6 mIoU, surpassing UniFormerS [22] by 2 . 

Revised: 6 mIoU, surpassing UniFormerS [22] by 2 .

Original: mIoU, while using less computation complexity.
Revised: mIoU, while using less computation complexity.

Original: Moreover, compared with UniFormerB [22], our iFormer-S still achieves 0 .
Revised: Moreover, compared with UniFormerB [22], our iFormer-S still achieves 0 .

Original: 6 mIoU improvement with only 1 / 2 parameters and nearly 1 /
Revised: 6 mIoU improvement with only 1 / 2 parameters and nearly 1 /

Original: FLOPs.
Revised: FLOPs.

Original: 4.4 Ablation study and visualization
Revised: 4.4 Ablation study and visualization

Original: In this section, we conduct experiments to better understand iFormer.
Revised: In this section, we conduct experiments to better understand iFormer.

Original: All the models are trained for 100 epochs on ImageNet, with the same training setting as described in Sec. 4.1.
Revised: All the models are trained for 100 epochs on ImageNet, with the same training setting as described in Sec. 4.1.

Original: Inception token mixer.
Revised: Inception token mixer.

Original: The Inception mixer is proposed to augment the perception capability of
Revised: The Inception mixer is proposed to augment the perception capability of

Original: ViTs in the frequency spectrum.
Revised: ViTs in the frequency spectrum.

Original: To evaluate the effects of the components in the Inception mixer, we remove the max-pooling or convolution from the full model and then report the results in Table 5, where (cid:33) and (cid:37) denote whether or not the corresponding branch is enabled.
Revised: To evaluate the effects of the components in the Inception mixer, we remove the max-pooling or convolution from the full model and then report the results in Table 5, where (cid:33) and (cid:37) denote whether or not the corresponding branch is enabled.

Original: Observably, combining attention with convolution and max-pooling can the highest classification accuracy.
Revised: Observably, combining attention with convolution and max-pooling can the highest classification accuracy.

Original: To further explore this scheme, Fig.
Revised: To further explore this scheme, Fig.

Original: 4 visualizes the Fourier spectrum of the Attention, MaxPool and DwConv branches
Revised: 4 visualizes the Fourier spectrum of the Attention, MaxPool and DwConv branches

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in Inception mixer.
Revised: in Inception mixer.

Original: We can see the attention mixer has higher concentrations on low frequencies; with the high-frequency mixer, i.e. , convolution and max-pooling, the model is encouraged to learn high frequency information.
Revised: We can see the attention mixer has higher concentrations on low frequencies; with the high-frequency mixer, i.e. , convolution and max-pooling, the model is encouraged to learn high frequency information.

Original: Overall, these results prove the effectiveness of the Inception mixer for expanding the perception capability of the Transformer in the frequency spectrum.
Revised: Overall, these results prove the effectiveness of the Inception mixer for expanding the perception capability of the Transformer in the frequency spectrum.

Original: Frequency ramp structure.
Revised: Frequency ramp structure.

Original: Previous investigations [27] show requirement of more local information at lower layers of the Transformer and more global information at higher layers.
Revised: Previous investigations [27] show requirement of more local information at lower layers of the Transformer and more global information at higher layers.

Original: We accordingly assume that a frequency ramp structure, i.e. , decreasing dimensions at high-frequency components and increasing dimensions at low-frequency components from lower to higher layers, has a better trade-off between high-frequency and low-frequency components across all layers.
Revised: We accordingly assume that a frequency ramp structure, i.e. , decreasing dimensions at high-frequency components and increasing dimensions at low-frequency components from lower to higher layers, has a better trade-off between high-frequency and low-frequency components across all layers.

Original: In order to justify this hypothesis, we investigate the effects of the channel ratio ( C h C and C l C ) in Table 5.
Revised: In order to justify this hypothesis, we investigate the effects of the channel ratio ( C h C and C l C ) in Table 5.

Original: It can be clearly seen that the model with C l /C ↑ , C h /C ↓ outperforms the other two models, which is consistent with the previous investigations.
Revised: It can be clearly seen that the model with C l /C ↑ , C h /C ↓ outperforms the other two models, which is consistent with the previous investigations.

Original: Hence, this indicates the rationality of the frequency ramp structure and its potential for leaning discriminating vision representations.
Revised: Hence, this indicates the rationality of the frequency ramp structure and its potential for leaning discriminating vision representations.

Original: Visualization.
Revised: Visualization.

Original: We visualize the Grad-CAM [81] activation maps of iFormer-S as well as Swin-T [5] models trained on ImageNet-1K in Fig.
Revised: We visualize the Grad-CAM [81] activation maps of iFormer-S as well as Swin-T [5] models trained on ImageNet-1K in Fig.

Original: It can be seen that compared with Swin, iFormer can more accurately and completely locate the objects.
Revised: It can be seen that compared with Swin, iFormer can more accurately and completely locate the objects.

Original: For example, in the hummingbird image, iFormer skips the branch and accurately attends to the whole bird including the tail.
Revised: For example, in the hummingbird image, iFormer skips the branch and accurately attends to the whole bird including the tail.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present an Inception Transformer (iFormer), a novel and general Transformer backbone.
Revised: In this paper, we present an Inception Transformer (iFormer), a novel and general Transformer backbone.

Original: iFormer adopts a channel splitting mechanism to simply and efficiently couple convolution/maxpooling and self-attention, giving more concentrations on high frequencies and expanding the perception capability of the Transformer in the frequency spectrum.
Revised: iFormer adopts a channel splitting mechanism to simply and efficiently couple convolution/maxpooling and self-attention, giving more concentrations on high frequencies and expanding the perception capability of the Transformer in the frequency spectrum.

Original: Based on the flexible Inception token mixer, we further design a frequency ramp structure, enabling effective trade-off between high-frequency and low-frequency components across all layers.
Revised: Based on the flexible Inception token mixer, we further design a frequency ramp structure, enabling effective trade-off between high-frequency and low-frequency components across all layers.

Original: Extensive experiments show that iFormer outperforms representative vision Transformers on image classification, object detection and semantic segmentation, demonstrating the great potential of our iFormer to serve as a general-purpose backbone for computer vision.
Revised: Extensive experiments show that iFormer outperforms representative vision Transformers on image classification, object detection and semantic segmentation, demonstrating the great potential of our iFormer to serve as a general-purpose backbone for computer vision.

Original: We hope this study will provide valuable insights for the community to design efficient and effective Transformer architectures.
Revised: We hope this study will provide valuable insights for the community to design efficient and effective Transformer architectures.

Original: Limitation.
Revised: Limitation.

Original: One obvious limitation of the proposed iFormer is that it requires manually defined channel ratio in the frequency ramp structure i e. , C h C and C l C for each iFormer block, which needs rich experience to define better on different tasks.
Revised: One obvious limitation of the proposed iFormer is that it requires manually defined channel ratio in the frequency ramp structure i e. , C h C and C l C for each iFormer block, which needs rich experience to define better on different tasks.

Original: it is not trained on large scale datasets, e g. ,
Revised: it is not trained on large scale datasets, e g. ,

Original: ImageNet-21K [48], due to computational constraint, which will be explored in further.
Revised: ImageNet-21K [48], due to computational constraint, which will be explored in further.

Original: Also, iFormer requires manually defined channel ratio in the frequency ramp structure i.e. , C h C and C l C for each iFormer block, which needs rich experience to define better on different tasks.
Revised: Also, iFormer requires manually defined channel ratio in the frequency ramp structure i.e. , C h C and C l C for each iFormer block, which needs rich experience to define better on different tasks.

Original: A straightforward solution would be to use neural architecture search
Revised: A straightforward solution would be to use neural architecture search

Original: Acknowledgement
Revised: Acknowledgement

Original: Weihao Yu would like to thank TRC program and GCP research credits for the support of partial computational resources.
Revised: Weihao Yu would like to thank TRC program and GCP research credits for the support of partial computational resources.

Original: This project is in part supported by the National Research Foundation
Revised: This project is in part supported by the National Research Foundation

Original: Singapore under its AI Singapore Programme (Award Number: AISG2-RP-2021-023)
Revised: Singapore under its AI Singapore Programme (Award Number: AISG2-RP-2021-023)

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The code will be released in future.
Revised: [No] The code will be released in future.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] 5.
Revised: [Yes] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Appendix Potential Impacts.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: The study introduces a general vision Transformer, i.e. , iFormer, which can be used on different vision tasks, e.g. , image classification, object detection and semantic segmentation.

Original: iFormer has no direct negative societal impact.

Original: However, we will realize that iFormer as a generalpurpose backbone can be used for harmful applications such as illegal face recognition.

Original: A.1 Results on semantic segmentation

Original: Setup.

Original: We further evaluate the generality of iFormer on semantic segmentation with the Upernet [82] framework.

Original: Following the training settings in Swin [5], the model is trained for 160K iterations with a batch size of 16.

Original: For training, we use AdamW optimizer with an initial learning rate 6 × 10 − 5 .

Original: All experiments are implemented on mmsegmentation [80] codebase.

Original: Results.

Original: Table 6 shows the mIoU and MS mIoU results of different backbones based on the

Original: UperNet [82] framework.

Original: From these results, it can be seen that our iFormer achieves 48.4 mIoU and 48.8 MS mIoU, consistently surpassing previous backbones on this task.

Original: For instance, our iFormer-S outperforms the Swin-T [5] by 3.9 mIoU while using fewer parameters.

Original: Compered with

Original: UniFormer-S [22], iFormer-S still achieves 0.8 mIoU improvement.

Original: [Table]

Original: A.2 Ablation study A.2.1 Down-sample and up-sample for self-attention

Original: In this work

Original: [Table]

Original: we propose the down- and up-sample structure to reduce the computational cost.

Original: Tableshows that, when removing this structure, iFormer-S has 7.0G FLOPs

Original: [Table]
Revised: [N/A]

Original: and achieves 83.6% topaccuracy.

Original: However, by adopting the down- and up-sample structure, iFormer-S gets a similar accuracy (83.4%) but has much less FLOPs (4.8 G)

Original: [Table]

Original: [Table]

Original: [Figure]

Original: We use large kernel (5 × 5) to replace 3 × 3 kernel of DWConv.

Original: The results are summarized in

Original: Tables 7.

Original: We can see that the model with 3 × 3 kernel achieves better results.

Original: We believe that this is because the self-attention has learned the information that is learnt by large-kernel DWConv.

Original: Besides, the smaller kernel is more conducive and effective for capturing high-frequency information.

Original: A.3 Visualization A.3.1 Fourier spectrum of different layers

Original: Frequency ramp structure plays an important role in iFormer, which is designed to learn hierarchical representations, i.e. , more high-frequency signals at lower layers of the Transformer and more low-frequency signals at higher layers.

Original: In order to justify this hypothesis, we visualize the Fourier spectrum of feature maps for different iFormer layers in Fig 6.

Original: We can see that iFormer captures more high-frequency components at 6- th layer and more low-frequency information at 18- th layer.

Original: Moreover, the high-frequency information gradually decreases from 6- th layer to 18- th layer, and low-frequency information does the opposite.

Original: Hence, these results indicate our iFormer can effectively trade-off high- and low-frequency components across all layers.

Original: A.3.2 CAM

Original: We futher show more examples of the Grad-CAM [81] activation maps of iFormer-S as well as

Original: Swin-T [5] models trained on ImageNet-1K in Fig.

Original: These examples indicate that compared with

Original: Swin, iFormer can more accurately and completely attend to the key objects.

Original: Taking the hog picture as example, iFormer locates the hog accurately but Swin also locates irrelevant part.

Original: A.4 Configurations of iFormers

Original: In this work, three variants of iFormer are used for a fair comparison under computation configurations, i.e. , iFormer-S, iFormer-B and iFormer-L. Table 8 shows their detailed configurations.

Original: Following

Original: Swin [5], iFormer adopts 4-stage architecture with H 4 × W 4 , H 8 × W 8 , H 16 × W 16 , H 32 × W 32 input sizes, where H and W are the width and height of the input image.

Original: In each iFormer block, C h /C and

Original: C l /C are used to balance the high-frequency and low frequency components.

Original: As shown in Table 8,

Original: C h /C gradually decreases from shallow to deep layers, while C l /C gradually increases.

Original: iFormer block uses depthwise convolution and max-pooling as high-frequency mixers.

Original: We set the kernel sizes of depthwise convolution and max-pooling to 3 × 3 .

Original: chihuahua drake pole cock harp hog brambling

Original: Input

Original: Swin-T iFormer-S

Original: [Table]