Original: AS-MLP: A N A XIAL S HIFTED MLP A RCHITECTURE FOR V ISION
Revised: AS-MLP: A N A XIAL S HIFTED MLP A RCHITECTURE FOR V ISION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: An Axial Shifted MLP architecture (AS-MLP) is proposed in this paper.
Revised: An Axial Shifted MLP architecture (AS-MLP) is proposed in this paper.

Original: Different from MLP-Mixer, where the global spatial feature is encoded forthe information ﬂow through matrix transposition and one token-mixing MLP, we pay more attention to the local features communication.
Revised: Different from MLP-Mixer, where the global spatial feature is encoded for information ﬂow through matrix transposition and one token-mixing MLP, we pay more attention to the local features communication.

Original: By axially shifting channels of the feature map, AS-MLP is able to obtain the information ﬂow from different axial directions, which captures the local dependencies.
Revised: By axially shifting channels of the feature map, AS-MLP is able to obtain the information ﬂow from different axial directions, which captures the local dependencies.

Original: Such an operation enables us to utilize a pure MLP architecture to achieve the same local receptive ﬁeld as
Revised: Such an operation enables us to utilize a pure MLP architecture to achieve the same local receptive ﬁeld as

Original: CNN-like architecture.
Revised: CNN-like architecture.

Original: We can also design the receptive ﬁeld size and dilation of blocks of AS-MLP, etc , just like designing those of convolution kernels.
Revised: We can also design the receptive ﬁeld size and dilation of blocks of AS-MLP, etc , just like designing those of convolution kernels.

Original: With the proposed AS-MLP architecture, our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset.
Revised: With the proposed AS-MLP architecture, our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset.

Original: Such a simple yet effective architecture outperforms all MLP-based architectures and achieves competitive performance compared to the transformer-based architectures ( e.g. ,
Revised: Such a simple yet effective architecture outperforms all MLP-based architectures and achieves competitive performance compared to the transformer-based architectures ( e.g. ,

Original: Swin Transformer) even with slightly lower FLOPs.
Revised: Swin Transformer) even with slightly lower FLOPs.

Original: In addition, AS-MLP is also the ﬁrst MLP-based architecture to be applied to the downstream tasks ( e.g. , object detection and semantic segmentation).
Revised: In addition, AS-MLP is also the ﬁrst MLP-based architecture to be applied to the downstream tasks ( e.g. , object detection and semantic segmentation).

Original: The experimental results are also impressive.
Revised: The experimental results are also impressive.

Original: Our proposed AS-MLP obtains 51.5 mAP on the COCO validation set and 49.5 MS mIoU on the ADE20K dataset, which is competitive compared to the transformer-based architectures.
Revised: Our proposed AS-MLP obtains 51.5 mAP on the COCO validation set and 49.5 MS mIoU on the ADE20K dataset, which is competitive compared to the transformer-based architectures.

Original: Our AS-MLP establishes a strong baseline of
Revised: Our AS-MLP establishes a strong baseline of

Original: MLP-based architecture.
Revised: MLP-based architecture.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the past decade, Convolutional Neural Networks (CNNs) (Krizhevsky et al., 2012; He et al., 2015) have received widespread attention and have become the de-facto standard for computer vision.
Revised: In the past decade, Convolutional Neural Networks (CNNs) (Krizhevsky et al., 2012; He et al., 2016) have received widespread attention and have become the de-facto standard for computer vision.

Original: Furthermore, with the in-depth exploration and research on self-attention, transformer-based architectures have also gradually emerged, and have surpassed CNN-based architectures in natural language processing ( e.g. , Bert (Devlin et al., 2018)) and vision understanding ( e.g. , ViT (Dosovitskiy et al., 2020), DeiT (Touvron et al., 2020)) with amounts of training data.
Revised: Furthermore, with the in-depth exploration and research on self-attention, transformer-based architectures have also gradually emerged, and have surpassed CNN-based architectures in natural language processing ( e.g. , Bert (Devlin et al., 2018)) and vision understanding ( e.g. , ViT (Dosovitskiy et al., 2021), DeiT (Touvron et al., 2021b)) with amounts of training data.

Original: Recently, Tolstikhin et al.
Revised: Recently, Tolstikhin et al.

Original: (2021) ﬁrstly propose MLP-based architecture, where almost all network parameters are learned from MLP (linear layer) and achieve amazing results, which is comparable with CNN-like models.
Revised: ﬁrst propose MLP-based architecture, where almost all network parameters are learned from MLP (linear layer) and achieve amazing results, which is comparable with CNN-like models.

Original: Such promising results drive our exploration of MLP-based architecture.
Revised: Such promising results drive our exploration of MLP-based architecture.

Original: In the MLP-Mixer (Tolstikhin et al., 2021), the model obtains the global receptive ﬁeld through matrix transposition and token-mixing projection such that the long-range dependencies are covered.
Revised: In the MLP-Mixer (Tolstikhin et al., 2021), the model obtains the global receptive ﬁeld through matrix transposition and token-mixing projection such that the long-range dependencies are covered.

Original: However, this rarely makes full use of the local information, which is very important in CNN-like architecture (Simonyan & Zisserman, 2014; He et al., 2015) because not all pixels need long-range dependencies, and the local information more focuses on extracting the low-level feature.
Revised: However, this rarely makes full use of the local information, which is very important in CNN-like architecture (Simonyan & Zisserman, 2015; He et al., 2016) because not all pixels need long-range dependencies, and the local information more focuses on extracting the low-level feature.

Original: In the transformer-based architectures, Swin Transformer (Liu et al., 2021b) computes the self-attention in a window ( 7 × 7 ) instead of the global receptive ﬁeld, which is similar to directly using a convolution layer with a large kernel size ( 7 × 7 ) to cover the local receptive ﬁeld.
Revised: In the transformer-based architectures, Swin Transformer (Liu et al., 2021b) computes the self-attention in a window ( 7 × 7 ) instead of the global receptive ﬁeld, which is similar to directly using a convolution layer with a large kernel size ( 7 × 7 ) to cover the local receptive ﬁeld.

Original: Some other papers have also already emphasized the advantages of local receptive ﬁelds, and introduced local information in the transformer, such as
Revised: Some other papers have also already emphasized the advantages of local receptive ﬁelds, and introduced local information in the transformer, such as

Original: Localvit (Li et al., 2021), NesT (Zhang et al., 2021), etc .
Revised: Localvit (Li et al., 2021), NesT (Zhang et al., 2021), etc .

Original: Driven by these ideas, we mainly explore the inﬂuence of locality on MLP-based architectures.
Revised: Driven by these ideas, we mainly explore the inﬂuence of locality on MLP-based architectures.

Original: In order to introduce locality into the MLP-based architecture, one of the simplest and most intuitive ideas is to add a window to the MLP-Mixer, and then perform a token-mixing projection of the local information on the features within the window, just as done in Swin Transformer (Liu et al., 2021b) and LINMAPPER (Fang et al., 2021).
Revised: In order to introduce locality into the MLP-based architecture, one of the simplest and most intuitive ideas is to add a window to the MLP-Mixer, and then perform a token-mixing projection of the local information on the features within the window, just as done in Swin Transformer (Liu et al., 2021b) and LINMAPPER (Fang et al., 2021).

Original: However, for the MLP-based architecture, if we divide the window ( e.g. , 7 × 7 ) and perform the token-mixing projection in the window, then only the 49 × 49  shared  1 linear layer is employed, which greatly limits the model capacity and thus affects the learning of parameters and the ﬁnal results.
Revised: However, for the MLP-based architecture, if we divide the window ( e.g. , 7 × 7 ) and perform the token-mixing projection in the window, then the linear layer has the 49 × 49 parameters shared between windows 1 , which greatly limits the model capacity and thus affects the learning of parameters and the ﬁnal results.

Original: Therefore,  we propose an axial shift strategy for MLP-based architecture, where we spatially shift features in both horizontal and vertical directions.
Revised: Therefore, a more ideal way to introduce locality is to directly model the relationship between a feature point and its surrounding feature points at any position, without the need to set a ﬁxed window (and window size) in advance. To aggregate the features of different spatial positions in the same position and model their relationships, inspired by (Wu et al., 2018; Lin et al., 2019; Wang et al., 2020; Ho et al., 2019), we propose an axial shift strategy for MLP-based architecture, where we spatially shift features in both horizontal and vertical directions.

Original: The axial shift can arrange the features of different spatial positions in the same position.
Revised: Such an approach not only aggregates features from different locations, but also makes the feature channel only need to be divided into k groups instead of k 2 groups to obtain a receptive ﬁeld of size k × k with the help of axial operation.

Original: After that, a channel-mixing MLP is used to combine these features, which is simple yet effective.
Revised: After that, a channel-mixing MLP combines these features, enabling the model to obtain local dependencies.

Original: Such a method enables the model to obtain more local dependencies, thus improve the performance.

Original: It also enables us to design MLP structure as the same as the convolutional kernel, for instance, to design the kernel size and dilation rate.
Revised: It also allows us to design MLP structure as the same as the convolutional kernel, for instance, to design the kernel size and dilation rate.

Original: Based on the axial shift strategy, we design Axial Shifted MLP architecture, named AS-MLP.
Revised: Based on the axial shift strategy, we design Axial Shifted MLP architecture, named AS-MLP.

Original: Our
Revised: Our

Original: AS-MLP obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs in the ImageNet1K dataset without any extra training data.
Revised: AS-MLP obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs in the ImageNet1K dataset without any extra training data.

Original: Such a simple yet effective method outperforms all
Revised: Such a simple yet effective method outperforms all

Original: MLP-based architectures and achieves competitive performance compared to the transformer-based architectures.
Revised: MLP-based architectures and achieves competitive performance compared to the transformer-based architectures.

Revised: It is also worth noting that the model weights in MLP-Mixer trained with ﬁxed image size cannot be adapted to downstream tasks with various input sizes because the token-mixing MLP has a ﬁxed dimension.

Original: The AS-MLP architecture canalso be transferred to downstream tasks ( e.g. , object detection).
Revised: On the contrary, the AS-MLP architecture can be transferred to downstream tasks ( e.g. , object detection) due to the design of axial shift.

Original: As far as we know, it is also the ﬁrst work to apply MLP-based architecture to the downstream task 2 .
Revised: As far as we know, it is also the ﬁrst work to apply MLP-based architecture to the downstream task.

Original: With the pre-trained model in the ImageNet-1K dataset, AS-MLP obtains 51. mAP on the COCO validation set and 49.5 MS mIoU on the ADE20K dataset, which is competitive compared to the transformer-based architectures.
Revised: With the pre-trained model in the ImageNet-1K dataset, AS-MLP obtains 51.5 mAP on the COCO validation set and 49.5 MS mIoU on the ADE20K dataset, which is competitive compared to the transformer-based architectures.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: CNN-based Architectures.
Revised: CNN-based Architectures.

Original: Since AlexNet (Krizhevsky et al., 2012) won the ImageNet competition in 2012, the CNN-based architectures have gradually been utilized to automatically extract image features instead of hand-crafted features.
Revised: Since AlexNet (Krizhevsky et al., 2012) won the ImageNet competition in 2012, the CNN-based architectures have gradually been utilized to automatically extract image features instead of hand-crafted features.

Original: Subsequently, the VGG network (Simonyan & Zisserman, 2014) is proposed, which purely uses a series of 3 × 3 convolution and fully connected layers, and obtains outstanding performance in image classiﬁcation.
Revised: Subsequently, the VGG network (Simonyan & Zisserman, 2015) is proposed, which purely uses a series of 3 × 3 convolution and fully connected layers, and obtains outstanding performance in image classiﬁcation.

Original: Furthermore, ResNet (He et al., 2015) is proposed, which utilizes the residual connection to transfer features in different layers, thereby alleviating the problem of gradient vanishing and obtaining superior performance.
Revised: Furthermore, ResNet (He et al., 2016) is proposed, which utilizes the residual connection to transfer features in different layers, thereby alleviating the problem of gradient vanishing and obtaining superior performance.

Original: After that, the residual module becomes an important component of the network design and is also employed in subsequent transformer-based architectures and MLP-based architectures.
Revised: After that, the residual module becomes an important component of the network design and is also employed in subsequent transformer-based architectures and MLP-based architectures.

Original: Some papers have made further improvements to the convolution operation in CNN-based architecture, such as dilated convolution (Yu & Koltun, 2015) and deformable convolution (Dai et al., 2017). 
Revised: Some papers have made further improvements to the convolution operation in CNN-based architecture, such as dilated convolution (Yu & Koltun, 2016) and deformable convolution (Dai et al., 2017). EfﬁcientNet (Tan &

Revised: Le, 2019; 2021) introduces neural architecture search into CNN to search for a suitable structure.

Original: These architectures build the CNN family and are used extensively in computer vision tasks.
Revised: These architectures build the CNN family and are used extensively in computer vision tasks.

Original: Transformer-based Architectures.
Revised: Transformer-based Architectures.

Original: Transformer is ﬁrstly proposed in (Vaswani et al., 2017), where the attention mechanism is utilized to model the relationship between features from the different spatial positions.
Revised: Transformer is ﬁrst proposed in (Vaswani et al., 2017), where the attention mechanism is utilized to model the relationship between features from the different spatial positions.

Original: Subsequently, the popularity of BERT (Devlin et al., 2018) in NLP also promotes the research on transformer in the ﬁeld of vision.
Revised: Subsequently, the popularity of BERT (Devlin et al., 2018) in NLP also promotes the research on transformer in the ﬁeld of vision.

Original: ViT (Dosovitskiy et al., 2020) uses a pure transformer framework to extract visual features, where an image is divided into 16 × 16 patches and the convolution layer is completely abandoned.
Revised: ViT (Dosovitskiy et al., 2021) uses a pure transformer framework to extract visual features, where an image is divided into 16 × 16 patches and the convolution layer is completely abandoned.

Original: It shows that the transformer-based architecture can perform well in large-scale datasets ( e.g. , JFT-300M).
Revised: It shows that the transformer-based architecture can perform well in large-scale datasets ( e.g. , JFT-300M).

Original: After that, DeiT (Touvron et al., 2020) carefully designs training strategies and data augmentation to further improve performance on the small datasets ( e.g. , ImageNet-1K).
Revised: After that, DeiT (Touvron et al., 2021b) carefully designs training strategies and data augmentation to further improve performance on the small datasets ( e.g. , ImageNet-1K).

Original: DeepViT (Zhou et al., 2021) and CaiT (Touvron et al., 2021b) consider the optimization problem when the network deepens, and train a deeper transformer network.
Revised: DeepViT (Zhou et al., 2021) and CaiT (Touvron et al., 2021c) consider the optimization problem when the network deepens, and train a deeper transformer network.

Original: CrossViT (Chen et al., 2021) combines the local patch and global patch using two vision transformers.
Revised: CrossViT (Chen et al., 2021a) combines the local patch and global patch using two vision transformers.

Original: CPVT (Chu et al., 2021b) uses a conditional position encoding to effectively encodes the spatial positions ofpatches.
Revised: CPVT (Chu et al., 2021b) uses a conditional position encoding to effectively encode the spatial positions of

Original: LeViT (Graham et al., 2021) improves ViT from many aspects, including the convolution embedding, extra non-linear projection and batch normalization, etc .

Original: Transformer-LS (Zhu et al., 2021) proposes a long-range attention and a short-term attention to model long sequences for both language and vision tasks.

Original: Some papers also design hierarchical backbone to extract spatial features at different scales, such as PVT (Wang et al., 2021),

Original: Swin Transformer (Liu et al., 2021b), Twins (Chu et al., 2021a) and NesT (Zhang et al., 2021), which can be applied to downstream tasks ( e.g. , object detection and semantic segmentation).

Original: #$%& ’() (* + # , - . / + # 0 , 1 , 1 4 1 3 %# 0 / 5 " 6 % 77 1 3 $ 891#:/;.1<,/=4+/ >:
Revised: #$%& ’() (* + # , - . / + # 0 , 1 , 1 4 1 3 %# 0 / 5 " 6 % 77 1 3 $ 891#:/;.1<,/=4+/ >:

Original: + # , - . /
Revised: + # , - . /

Original: = % 0 $ 1$ 891#:/;.1<,/=4+/ >:
Revised: = % 0 $ 1$ 891#:/;.1<,/=4+/ >:

Original: + # , - .
Revised: + # , - .

Original: = % 0 $ 1$ 891#:/;.1<,/=4+/ >:
Revised: = % 0 $ 1$ 891#:/;.1<,/=4+/ >:

Original: + # , - . /
Revised: + # , - . /

Original: = % 0 $ 1$ 891#:/;.1<,/=4+/ >:
Revised: = % 0 $ 1$ 891#:/;.1<,/=4+/ >:

Original: 2-? @A()@(*@
Revised: 2-? @A()@(*@

Original: B()@(*@
Revised: B()@(*@

Original: CB()A(*A @B( ) DE(*DE
Revised: CB()A(*A @B( ) DE(*DE

Original: AB( ) ’C(*’C (C (C (E (C ;,#$%D ;,#$%C ;,#$%’ ;,#$%@ 
Revised: AB( ) ’C(*’C (C (C (E (C ;,#$%D ;,#$%C ;,#$%’ ;,#$%@ patches.

Revised: LeViT (Graham et al., 2021) improves ViT from many aspects, including the convolution embedding, extra non-linear projection and batch normalization, etc .

Revised: Transformer-LS (Zhu et al., 2021) proposes a long-range attention and a short-term attention to model long sequences for both language and vision tasks.

Revised: Some papers also design hierarchical backbone to extract spatial features at different scales, such as PVT (Wang et al., 2021),

Revised: Swin Transformer (Liu et al., 2021b), Twins (Chu et al., 2021a) and NesT (Zhang et al., 2021), which can be applied to downstream tasks ( e.g. , object detection and semantic segmentation).

Original: MLP-based Architectures.
Revised: MLP-based Architectures.

Original: MLP-Mixer (Tolstikhin et al., 2021) has designed a very concise framework that utilizes matrix transposition and MLP to transmit information between spatial features.
Revised: MLP-Mixer (Tolstikhin et al., 2021) has designed a very concise framework that utilizes matrix transposition and MLP to transmit information between spatial features.

Original: Resort to MLP, skip connection between layers and normalization layer, MLP-Mixer obtains promising experimental results.
Revised: Resort to MLP, skip connection between layers and normalization layer, MLP-Mixer obtains promising experimental results.

Original: The concurrent work FF (Melas-Kyriazi, 2021) also applies a similar network architecture and reaches similar conclusions.
Revised: The concurrent work FF (Melas-Kyriazi, 2021) also applies a similar network architecture and reaches similar conclusions.

Original: Such experimental results are surprising, which shows that the MLP-based architecture also achieves comparable performance with CNN-based architectures and transformer-based architectures.
Revised: Such experimental results are surprising, which shows that the MLP-based architecture also achieves comparable performance with CNN-based architectures and transformer-based architectures.

Original: Subsequently, Res-MLP (Touvron et al., 2021a) is proposed, which also obtains impressive performance with residual MLP only trained on ImageNet1K. gMLP (Liu et al., 2021a) and EA (Guo et al., 2021) introduce Spatial Gating Unit (SGU) and the external attention to improve the performance of the pure MLP-based architecture, respectively.
Revised: Subsequently, Res-MLP (Touvron et al., 2021a) is proposed, which also obtains impressive performance with residual MLP only trained on ImageNet1K. gMLP (Liu et al., 2021a) and EA (Guo et al., 2021) introduce Spatial Gating Unit (SGU) and the external attention to improve the performance of the pure MLP-based architecture, respectively.

Original: Recently, Container (Gao et al., 2021) proposes a general network that uniﬁes convolution, transformer, and MLP-Mixer.
Revised: Recently, Container (Gao et al., 2021) proposes a general network that uniﬁes convolution, transformer, and MLP-Mixer.

Original: The closest concurrent methods to our work are S 2 -MLP (Yu et al., 2021) and ViP (Hou et al., 2021).
Revised: The closest concurrent methods to our work are S 2 -MLP (Yu et al., 2021) and ViP (Hou et al., 2021).

Original: S 2 -MLP uses spatial-shift MLP for feature exchange.
Revised: S 2 -MLP uses spatial-shift MLP for feature exchange.

Original: ViP proposes a
Revised: ViP proposes a

Original: Permute-MLP layer for spatial information encoding to capture long-range dependencies.
Revised: Permute-MLP layer for spatial information encoding to capture long-range dependencies.

Original: Different from their work, we focus on capturing the local dependencies with axially shifting features in the spatial dimension, which obtains better performance and can be applied to the downstream tasks.
Revised: Different from their work, we focus on capturing the local dependencies with axially shifting features in the spatial dimension, which obtains better performance and can be applied to the downstream tasks.

Revised: More recently, the concurrent work, CycleMLP (Chen et al., 2021b) and S 2 -MLPv2 (Yu et al., 2021) are proposed.

Revised: S 2 -MLPv2 improves S 2 -MLP and CycleMLP designs Cycle Fully-Connected Layer (Cycle FC) to obtain a larger receptive ﬁeld than Channel FC.

Revised: Our AS-MLP obtains comparable performance with these architectures.

Original:  T HE AS-MLP A RCHITECTURE 3. O VERALL A RCHITECTURE
Revised:  T HE AS-MLP A RCHITECTURE 3. O VERALL A RCHITECTURE

Original: Figure 1 shows our Axial Shifted MLP (AS-MLP) architecture.
Revised: Figure 1 shows our Axial Shifted MLP (AS-MLP) architecture 2 .

Original: Given an RGB image I ∈ R 3 × H × W , where H and W are the height and width of the image, respectively.
Revised: Given an RGB image I ∈ R 3 × H × W , where H and W are the height and width of the image, respectively.

Original: AS-MLP ﬁrstly performs the patch partition operation, which splits the original image into multiple patch tokens with the patch size of 4 × 4 , thus the combination of all tokens has the size of 48 × H 4 × W 4 .
Revised: AS-MLP ﬁrst performs the patch partition operation, which splits the original image into multiple patch tokens with the patch size of 4 × 4 , thus the combination of all tokens has the size of 48 × H 4 × W 4 .

Original: ASMLP has four stages in total and there are different numbers of AS-MLP blocks in different stages.
Revised: ASMLP has four stages in total and there are different numbers of AS-MLP blocks in different stages.

Original: Figure 1 only shows the tiny version of AS-MLP, and other variants will be discussed in Sec. 3.4.
Revised: Figure 1 only shows the tiny version of AS-MLP, and other variants will be discussed in Sec. 3.4.

Original: All the tokens in the previous step will go through these four stages, and the ﬁnal output feature will be used for image classiﬁcation.
Revised: All the tokens in the previous step will go through these four stages, and the ﬁnal output feature will be used for image classiﬁcation.

Original: In Stage 1, a linear embedding and the AS-MLP blocks are adopted for each token.
Revised: In Stage 1, a linear embedding and the AS-MLP blocks are adopted for each token.

Original: The output has the dimension of C × H 4 × W 4 , where C is the number of channels.
Revised: The output has the dimension of C × H 4 × W 4 , where C is the number of channels.

Original: Stage 2 ﬁrstly performs patch merging on the features outputted from the previous stage, which groups the neighbor 2 × 2 patches to obtain a feature with the size of 4 C × H 8 × W 8 and then a linear layer is adopted to warp the feature size to 2 C × H 8 × W 8 , followed by the cascaded AS-MLP blocks.
Revised: Stage 2 ﬁrst performs patch merging on the features outputted from the previous stage, which groups the neighbor 2 × 2 patches to obtain a feature with the size of 4 C × H 8 × W 8 and then a linear layer is adopted to warp the feature size to 2 C × H 8 × W 8 , followed by the cascaded AS-MLP blocks.

Original: Stage 3 and Stage 4 have similar structures to Stage 2, and the hierarchical representations will be generated in these stages.
Revised: Stage 3 and Stage 4 have similar structures to Stage 2, and the hierarchical representations will be generated in these stages.

Original: 3. AS-MLP B LOCK

Original: The core operation of AS-MLP architecture is the AS-MLP block, which is illustrated in Figure 2a.

Original: It mainly consists of the Norm layer, Axial Shift operation, MLP, and residual connection.

Original: In the Axial Shift operation, we utilize the channel projection, vertical shift, and horizontal shift to extract features, where the channel projection maps the feature with a linear layer.

Original: Vertical shift and horizontal shift are responsible for the feature translation along the spatial directions.

Original: As shown in Figure 2b, we take the horizontal shift as an example.

Original: The input has the dimension of

Original: C × h × w , for convenience, we omit h and assume C = 3 , w = 5 in this ﬁgure.

Original: When the shift size is 3, the input feature is split into three parts and each part is shifted by { -1, 0, 1 } units along the

Original: #$ !"
Revised: #$ !"

Original: #$ %&’ ()*+,-./* 2/+334,-’#"5461*" 2/+334,-’#"5461*" 74#1*6+,-./* 8"#*9"31+,-./* 2/+334,-’#"5461*" 2/+334,-’#"5461*" (a) ! "
Revised: #$ %&’ ()*+,-./* 2/+334,-’#"5461*" 2/+334,-’#"5461*" 74#1*6+,-./* 8"#*9"31+,-./* 2/+334,-’#"5461*" 2/+334,-’#"5461*" (a) ! "

Original: # $ % ! " # $ % ! "
Revised: # $ % ! " # $ % ! "

Original: # $ % & & !"
Revised: # $ % & & !"

Original: #$%"&’()*+$,’ ! " ! " !"
Revised: #$%"&’()*+$,’ ! " ! " !"

Original: #$%&’’()* !" #$%&’’()* ! "
Revised: #$%&’’()* !" #$%&’’()* ! "

Original: # $ % ! " # $ % ! "
Revised: # $ % ! " # $ % ! "

Original: # $ % & & ! " # $ % ! " 
Revised: # $ % & & ! " # $ % ! " # $ % ! "

Original: # $ % ! " # $ % & & " # $ % ! " 
Revised: # $ % & & " # $ % ! " # $ % ! " # $ (b)

Original: # $ % ! "
Revised: Figure 2: (a) shows the structure of the AS-MLP block; (b) shows the horizontal shift, where the arrows indicate the steps, and the number in each box is the index of the feature. 3. AS-MLP B LOCK The core operation of AS-MLP architecture is the AS-MLP block, which is illustrated in Figure 2a. It mainly consists of the Norm layer, Axial Shift operation, MLP, and residual connection.

Original: # $ (b) horizontal direction, respectively.
Revised: In the Axial Shift operation, we utilize the channel projection, vertical shift, and horizontal shift to extract features, where the channel projection maps the feature with a linear layer.

Original: In this operation, zero padding is performed (indicated by gray blocks), and  we also discuss the experimental results of using other padding methods in Sec. 4.
Revised: Vertical shift and horizontal shift are responsible for the feature translation along the spatial directions. As shown in Figure 2b, we take the horizontal shift as an example. The input has the dimension of C × h × w , for convenience, we omit h and assume C = 3 , w = 5 in this ﬁgure.

Revised: When the shift size is 3, the input feature is split into three parts and each part is shifted by { -1, 0, 1 } units along the horizontal direction, respectively.

Revised: In this operation, zero padding is performed (indicated by gray blocks), and we also discuss the experimental results of using other padding methods in Sec. 4.

Original: After that, the features in the dashed box will be taken out and used for the next channel projection.
Revised: After that, the features in the dashed box will be taken out and used for the next channel projection.

Original: The same operation is also performed in the vertical shift.
Revised: The same operation is also performed in the vertical shift.

Original: In the process of horizontal shift and vertical shift, since the feature performs different shift units in both directions, when the features are recombined (the feature in the dashed box), the information from different spatial positions can be combined together.
Revised: In the process of horizontal shift and vertical shift, since the feature performs different shift units in both directions, when the features are recombined (the feature in the dashed box), the information from different spatial positions can be combined together.

Original: In the next channel projection operation, information from different spatial locations can fully ﬂow and interact.
Revised: In the next channel projection operation, information from different spatial locations can fully ﬂow and interact.

Original: The code of AS-MLP block is listed in Alg. 1.
Revised: The code of AS-MLP block is listed in Alg. 1.

Original: Complexity.
Revised: Complexity.

Original: In the transformer-based architecture, the multi-head self-attention (MSA) is usually adopted, where the attention between tokens is computed.
Revised: In the transformer-based architecture, the multi-head self-attention (MSA) is usually adopted, where the attention between tokens is computed.

Original: Swin Transformer (Liu et al., 2021b) introduces a window to partition the image and propose window multi-head self-attention (W-MSA), which only considers the computation within this window.
Revised: Swin Transformer (Liu et al., 2021b) introduces a window to partition the image and propose window multi-head self-attention (W-MSA), which only considers the computation within this window.

Original: It signiﬁcantly reduces the computation complexity.
Revised: It signiﬁcantly reduces the computation complexity.

Original: However, in the AS-MLP block, without the concept of the window, we only Axially
Revised: However, in the AS-MLP block, without the concept of the window, we only Axially

Original: Shift (AS) the feature from the previous layer, which does not require any multiplication and addition operations.
Revised: Shift (AS) the feature from the previous layer, which does not require any multiplication and addition operations.

Original: Further, the time cost of Axial Shift is very low and almost irrelevant to the shift size.
Revised: Further, the time cost of Axial Shift is very low and almost irrelevant to the shift size.

Original: Given a feature map (is usually named patches in transformer) with the dimension of C × h × w , each Axial shift operation in Figure 2a only has four channel projection operations, which has the computation complexity 4 hwC 2 .
Revised: Given a feature map (is usually named patches in transformer) with the dimension of C × h × w , each Axial shift operation in Figure 2a only has four channel projection operations, which has the computation complexity 4 hwC 2 .

Original: If the window size in Swin Transformer (Liu et al., 2021b) is M , the complexities of MSA, W-MSA and AS are as follows:
Revised: If the window size in Swin Transformer (Liu et al., 2021b) is M , the complexities of MSA, W-MSA and AS are as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the AS-MLP architecture has slightly less complexity than Swin Transformer.
Revised: Therefore, the AS-MLP architecture has slightly less complexity than Swin Transformer.

Original: The speciﬁc complexity calculation of each layer is shown in Appendix A.2.
Revised: The speciﬁc complexity calculation of each layer is shown in Appendix A.2.

Original: 3. C OMPARISONS BETWEEN AS-MLP, C ONVOLUTION , T RANSFORMER AND MLP-M IXER
Revised: 3. C OMPARISONS BETWEEN AS-MLP, C ONVOLUTION , T RANSFORMER AND MLP-M IXER

Original: In this section, we show the comparisons between AS-MLP and the recent distinct building blocks used in computer vision, e.g. , the standard convolution, Swin Transformer, and MLP-Mixer.
Revised: In this section, we compare AS-MLP with the recent distinct building blocks used in computer vision, e.g. , the standard convolution, Swin Transformer, and MLP-Mixer.

Original: Although these modules are explored in completely different routes, from the perspective of calculation, they are all based on a given output location point, and its value depends on the weighted sum of different sampling location features (multiplication and addition operation).
Revised: Although these modules are explored in completely different routes, from the perspective of calculation, they are all based on a given output location point, and the output depends on the weighted sum of different sampling location features (multiplication and addition operation).

Original: These sampling location features include local dependencies ( e.g. , convolution) and long-range dependencies ( e.g. , MLP-Mixer).
Revised: These sampling location features include local dependencies ( e.g. , convolution) and long-range dependencies ( e.g. , MLP-Mixer).

Original: Figure 3 shows the main differences of these modules in the sampling location.Convolution.
Revised: Figureshows the main differences of these modules in the sampling location.

Original: Given a feature map X , a sliding kernel with the shape of k × k is performed on X to obtain the output Y conv :
Revised: Given an input feature map X ∈ R H × W × C , the outputs Y i,j with different operations in position ( i, j ) are as follows:

Original: [Equation]
Revised: [Figure]

Original: where W is the learnable weight. h and w are the height and width of X , respectively.
Revised: import torch import torch.nn.

Original: As shown in
Revised: [Figure] as

Original: Figure 3, the convolution operation has a local receptive ﬁeld, thus it is more suitable at extracting features with the local dependencies.

Original: [Figure]
Revised: [Figure]

Original: import torch.nn.functional as F def shift(x, dim): x = F.pad(x, "constant", 0) x
Revised: def shift(x, dim): x = F.pad(x, "constant", 0) x

Original: = torch.chunk(x, shift_size, 1) x 
Revised: = torch.chunk(x, shift_size, 1) x =

Original: = [ torch.roll(x_c, shift, dim) for x_s, shift in zip(x, range(-pad, pad+1))]
Revised: [ torch.roll(x_s, shift, dim) for x_s, shift in zip(x, range(-pad, pad+1))]

Original: x = torch.cat(x, 1) return x[:, :, pad:-pad, pad:-pad]
Revised: x = torch.cat(x, 1) return x[:, :, pad:-pad, pad:-pad]

Original: def as_mlp_block(x):shortcut=x x=norm(x)x=
Revised: def as_mlp_block(x):shortcut=x x=norm(x)x=

Original: actn(norm(proj(x)))x_lr=
Revised: actn(norm(proj(x)))x_lr=

Original: actn(proj(shift(x, 3)))
Revised: actn(proj(shift(x, 3)))

Original: x_td =
Revised: x_td =

Original: actn(proj(shift(x, 2)))
Revised: actn(proj(shift(x, 2)))

Original: x = x_lr + x_td x = proj(norm(x)) return x + shortcut
Revised: x = x_lr + x_td x = proj(norm(x)) return x + shortcut

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] Convolution. For convolution operation, a sliding kernel with the shape of k × k (receptive ﬁeld region R ) is performed on X to obtain the output Y conv i,j :

Original: [Figure]
Revised: [Equation]

Revised: where W conv ∈ R k × k × C is the learnable weight.

Revised: h and w are the height and width of X , respectively.

Revised: As shown in Figure 3, the convolution operation has a local receptive ﬁeld, thus it is more suitable at extracting features with the local dependencies.

Original: Swin Transformer.
Revised: Swin Transformer.

Original: Swin Transformer introduces a window into the transformer-based architecture to cover the local attention.
Revised: Swin Transformer introduces a window into the transformer-based architecture to cover the local attention.

Original: The input X from a window is embedded to obtain Q, K, V matrix, and the output Y swin is the attention combination of features within the window:
Revised: The input X from a window is embedded to obtain Q, K, V matrix, and the output Y swin is the attention combination of features within the window:

Original: [Equation]
Revised: [Equation]

Original: The introduction of locality further improves the performance of the transformer-based architecture and reduce the computational complexity.
Revised: The introduction of locality further improves the performance of the transformer-based architecture and reduces the computational complexity.

Original: MLP-Mixer.
Revised: MLP-Mixer.

Original: MLP-Mixer abandons the attention operation.
Revised: MLP-Mixer abandons the attention operation.

Original: It ﬁrstly transposes the input X , and then a token-mixing MLP is appended to obtain the output Y mixer :
Revised: It ﬁrst transposes the input X , and then a token-mixing MLP is appended to obtain the output Y mixer i,j

Original: [Equation]
Revised: [Equation]

Original: where W t ∈ R hw × hw is the learnable weight in token-mixing MLP.
Revised: where W mixer ∈ R hw × hw is the learnable weight in token-mixing MLP.

Original: MLP-Mixer perceives the global information only with matrix transposition and MLP.
Revised: MLP-Mixer perceives the global information only with matrix transposition and MLP.

Original: AS-MLP.
Revised: AS-MLP.

Original: AS-MLP axially shifts the feature map as shown in Figure 2b.
Revised: AS-MLP axially shifts the feature map as shown in Figure 2b.

Original: Given the input X and the shift size s  , X is ﬁrstly divided into s splits in the horizontal and vertical direction, which is represented to Concat ( X h 1 , · · · , X hs ) and Concat ( X v 1 , · · · , X vs ) .
Revised: Given the input X , shift size s and dilation rate d , X is ﬁrst divided into s splits in the horizontal and vertical direction.

Original: After the axial shift in Figure 2b, the output Y as  is:
Revised: After the axial shift in Figure 2b, the output Y as i,j is:

Original: [Equation]
Revised: [Equation]

Original: where W hc and W vc are the learnable weights of channel projection in the horizontal and vertical directions, X h =
Revised: where W as-h , W as-v ∈ R C are the learnable weights of channel projection in the horizontal and vertical directions (here we omit activation function and bias).

Original: Concat ( −→ X h 1 , · · · , −→ X hs ) and X v = Concat ( −→ X v 1 , · · · , −→ X vs ) , which shows the axial shift.

Original: Unlike MLP-Mixer, we pay more attention to the local dependencies through axial shift of features and channel projection.
Revised: Unlike MLP-Mixer, we pay more attention to the local dependencies through axial shift of features and channel projection.

Original: Such an operation is closely related to Shift (Wu et al., 2018) and TSM (Lin et al., 2019).
Revised: Such an operation is closely related to Shift (Wu et al., 2018) and TSM (Lin et al., 2019).

Original: However, our method has the following characteristics:
Revised: However, our method has the following characteristics:

Original: i) we use axial shift in the horizontal and vertical directions, which focuses more on the information exchange in two directions; ii) the proposed network is built upon Swin transformer and pure MLP-based architecture, where only linear layer is used and BN is replaced by LN; iii) as will be shown in Sec. 
Revised: i) we use axial shift in the horizontal and vertical directions, which focuses more on the information exchange in two directions; ii) the proposed network is built upon Swin transformer and pure MLP-based architecture, where only linear layer is used and BN is replaced by LN; iii) as will be shown in Sec. 4, our network achieves superior performance, which shows the effectiveness of our method.

Original: 4, our network achieves superior performance, which shows the effectiveness of our method.
Revised: Although such an operation can be implemented in the original shift (Wu et al., 2018), the feature channel needs to split into k 2 groups to achieve a receptive ﬁeld of size k × k . However, the axial operation makes the feature channel only need to be divided into k groups instead of k 2 groups, which reduces the complexity. [Table]

Original: 3. V ARIANTS OF AS-MLP ARCHITECTURE
Revised: 3. V ARIANTS OF AS-MLP ARCHITECTURE

Original: Figure 1 only shows the tiny version of our AS-MLP architecture.
Revised: Figure 1 only shows the tiny version of our AS-MLP architecture.

Original: Following DeiT (Touvron et al., 2020) and Swin Transformer (Liu et al., 2021b), we also stack different number of AS-MLP blocks (the number of blocks in four stages) and expand the channel dimension ( C in Figure 1) to obtain variants of the AS-MLP architecture of different model sizes, which are AS-MLP-Tiny (ASMLP-T), AS-MLP-Small (AS-MLP-S) and AS-MLP-Base (AS-MLP-B), respectively.
Revised: Following DeiT (Touvron et al., 2021b) and Swin Transformer (Liu et al., 2021b), we also stack different number of AS-MLP blocks (the number of blocks in four stages) and expand the channel dimension ( C in Figure 1) to obtain variants of the AS-MLP architecture of different model sizes, which are AS-MLP-Tiny (AS-MLPT), AS-MLP-Small (AS-MLP-S) and AS-MLP-Base (AS-MLP-B), respectively.

Original: The speciﬁc conﬁguration is as follows:
Revised: The speciﬁc conﬁguration is as follows:

Original: • AS-MLP-T: C = 96, the number of blocks in four stages = { 2, 2, 6, 2 } ;[Table] • AS-MLP-S: C = 96, the number of blocks in four stages = { 2, 2, 18, 2 } ; • AS-MLP-B: C = 128, the number of blocks in four stages = { 2, 2, 18, 2 } .
Revised: • AS-MLP-T: C = 96, the number of blocks in four stages = { 2, 2, 6, 2 } ; • AS-MLP-S: C = 96, the number of blocks in four stages = { 2, 2, 18, 2 } ; • AS-MLP-B: C = 128, the number of blocks in four stages = { 2, 2, 18, 2 } .

Original: The detailed conﬁgurations can be found in Appendix A.1.
Revised: The detailed conﬁgurations can be found in Appendix A.1.

Original: Table 1 in Sec.
Revised: Table 1 in Sec.

Original: 4 shows Top-1 accuracy, model size (Params), computation complexity (FLOPs) and throughput of different variants.
Revised: 4 shows Top-1 accuracy, model size (Params), computation complexity (FLOPs) and throughput of different variants.

Original:  E XPERIMENTS 4. I MAGE C LASSIFICATION ON THE I MAGE N ET -1K D ATASET
Revised:  E XPERIMENTS 4. I MAGE C LASSIFICATION ON THE I MAGE N ET -1K D ATASET

Original: Settings.
Revised: Settings.

Original: To evaluate the effectiveness of our AS-MLP, we conduct experiments of the image classiﬁcation on the ImageNet-1K benchmark, which is collected in (Deng et al., 2009).
Revised: To evaluate the effectiveness of our AS-MLP, we conduct experiments of the image classiﬁcation on the ImageNet-1K benchmark, which is collected in (Deng et al., 2009).

Original: It contains 1.28M training images and 20K validation images from a total of 1000 classes.
Revised: It contains 1.28M training images and 20K validation images from a total of 1000 classes.

Original: We report the experimental results with single-crop Top-1 accuracy.
Revised: We report the experimental results with single-crop Top-1 accuracy.

Original: Such a dataset is also utilized to conduct the ablation study experiments.

Original: Following the training strategy of Swin Transformer (Liu et al., 2021b), we use an initial learning rate of 0.001 with cosine decay and 20 epochs of linear warm-up.
Revised: We use an initial learning rate of 0.001 with cosine decay and 20 epochs of linear warm-up.

Original: The AdamW (Loshchilov & Hutter, 2017) optimizer is employed to train the whole model for 300 epochs with a batch size of 1024.
Revised: The AdamW (Loshchilov & Hutter, 2019) optimizer is employed to train the whole model for 300 epochs with a batch size of 1024.

Original: We also use label smoothing (Szegedy et al., 2016) with a smooth ratio of 0. and DropPath (Huang et al., 2016) strategy.
Revised: Following the training strategy of Swin Transformer (Liu et al., 2021b), we also use label smoothing (Szegedy et al., 2016) with a smooth ratio of 0.1 and DropPath (Huang et al., 2016) strategy.

Original: Results.
Revised: Results.

Original: All image classiﬁcation results are shown in Table 1.
Revised: All image classiﬁcation results are shown in Table 1.

Original: We divide all network architectures into CNN-based, Transformer-based and MLP-based architectures.
Revised: We divide all network architectures into CNN-based, Transformer-based and MLP-based architectures.

Original: The input resolution is 224 × 224 .
Revised: The input resolution is 224 × 224 .

Original: Our proposed AS-MLP outperforms other MLP-based architectures when keeping similar parameters and FLOPs.
Revised: Our proposed AS-MLP outperforms other MLP-based architectures when keeping similar parameters and FLOPs.

Original: e.g. , AS-MLP-S obtains higher top-1 accuracy (83.1%) with less parameters than Mixer-B/16 (Tolstikhin et al., 2021) (76.4%) and ViP-Medium/7 (Hou et al., 2021)
Revised: e.g. , AS-MLP-S obtains higher top-1 accuracy (83.1%) with less parameters than Mixer-B/16 (Tolstikhin et al., 2021) (76.4%) and ViP-Medium/7 (Hou et al., 2021)

Original: Furthermore, it achieves competitive performance compared with transformer-based architectures, e.g. , AS-MLP-B (83.3%) vs .
Revised: Furthermore, it achieves competitive performance compared with transformer-based architectures, e.g. , AS-MLP-B (83.3%) vs .

Original: Swin-B (Liu et al., 2021b) (83.3%), which shows the effectiveness of our AS-MLP architecture.
Revised: Swin-B (Liu et al., 2021b) (83.3%), which shows the effectiveness of our AS-MLP architecture.

Original: [Table]
Revised: [Table]

Original: Results of Mobile Setting.
Revised: Results of Mobile Setting.

Original: In addition to standard experiments, we also compare the results of AS-MLP in the mobile setting, which is shown in Table 2.
Revised: In addition to standard experiments, we also compare the results of AS-MLP in the mobile setting, which is shown in Table 2.

Original: We build the
Revised: We build the

Original: Swin (mobile) model and AS-MLP (mobile) model with similar parameters (about 10M).
Revised: Swin (mobile) model and AS-MLP (mobile) model with similar parameters (about 10M).

Original: The speciﬁc network details can be found in Appendix A.3.
Revised: The speciﬁc network details can be found in Appendix A.3.

Original: The experimental results show that our model signiﬁcantly exceeds Swin Transformer (Liu et al., 2021b) in the mobile setting (76.05% vs .
Revised: The experimental results show that our model signiﬁcantly exceeds Swin Transformer (Liu et al., 2021b) in the mobile setting (76.05% vs .

Original: 4. A BLATION S TUDY
Revised: 4. T HE C HOICE AND I MPACT OF AS-MLP B LOCK

Original: The core component in the AS-MLP block is the axial shift.
Revised: The core component in the AS-MLP block is the axial shift.

Original: We perform  the ablation studies for the axial shift operation, including different conﬁgurations 3 of the AS-MLP block and its connection types.
Revised: We perform experiments to analyze the choices of different conﬁgurations 3 of the AS-MLP block and its connection types and the impact of AS-MLP block.

Original: All ablations are conducted based on the AS-MLP-T, as shown in the setting of Sec. 3.4.
Revised: All ablations are conducted based on the AS-MLP-T, as shown in the setting of Sec. 3.4.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Different Conﬁgurations of AS-MLP Block.
Revised: Different Conﬁgurations of AS-MLP Block.

Original: In order to encourage the information ﬂow from different channels in the spatial dimension, the features from the horizontal shift and the vertical shift are aggregated together in Figure 2.
Revised: In order to encourage the information ﬂow from different channels in the spatial dimension, the features from the horizontal shift and the vertical shift are aggregated together in Figure 2.

Original: We evaluate the inﬂuence of different conﬁgurations of AS-MLP block, including shift size, padding method, and dilation rate, which are similar to the conﬁguration of a convolution kernel.
Revised: We evaluate the inﬂuence of different conﬁgurations of AS-MLP block, including shift size, padding method, and dilation rate, which are similar to the conﬁguration of a convolution kernel.

Original: All experiments of different conﬁgurations are shown in Table 3a.
Revised: All experiments of different conﬁgurations are shown in Table 3a.

Original: We have three ﬁndings as follows: i) ‘Zero padding’ is more suitable for the design of AS-MLP block than other padding methods 4 ; ii) increasing the dilation rate slightly reduces the performance of AS-MLP, which is consistent with CNN-based architecture.
Revised: We have three ﬁndings as follows: i) ‘Zero padding’ is more suitable for the design of AS-MLP block than other padding methods 4 ; ii) increasing the dilation rate slightly reduces the performance of AS-MLP, which is consistent with CNN-based architecture.

Original: Dilation is usually used for semantic segmentation rather than image classiﬁcation; iii) when expanding the shift size, the accuracy will increase ﬁrst and then decrease.
Revised: Dilation is usually used for semantic segmentation rather than image classiﬁcation; iii) when expanding the shift size, the accuracy will increase ﬁrst and then decrease.

Original: A possible reason is that the receptive ﬁeld is enlarged (shift size = 5 or 7) such that AS-MLP pays attention to the global dependencies, but when shift size is 9, the network pays too much attention to the global dependencies, thus neglecting the extraction of local features, which leads to lower accuracy.
Revised: A possible reason is that the receptive ﬁeld is enlarged (shift size = 5 or 7) such that AS-MLP pays attention to the global dependencies, but when shift size is 9, the network pays too much attention to the global dependencies, thus neglecting the extraction of local features, which leads to lower accuracy.

Original: Therefore, we use the conﬁguration (shift size = 5, zero padding, dilation rate = 1) in all experiments, including object detection and semantic segmentation.
Revised: Therefore, we use the conﬁguration (shift size = 5, zero padding, dilation rate = 1) in all experiments, including object detection and semantic segmentation.

Original: Connection Type.
Revised: Connection Type.

Original: We also compare the different connection types of AS-MLP block, such as serial connection and parallel connection, and the results are shown in Table 3b. 
Revised: We also compare the different connection types of AS-MLP block, such as serial connection and parallel connection, and the results are shown in Table 3b. Parallel connection [Table]

Original: Parallel connection consistently outperforms serial connection in terms of different shift sizes, which shows the effectiveness of the parallel connection.
Revised: consistently outperforms serial connection in terms of different shift sizes, which shows the effectiveness of the parallel connection.

Original: When the shift size is 1, the serial connection is better but it is not representative because only channel-mixing MLP is used.
Revised: When the shift size is 1, the serial connection is better but it is not representative because only channel-mixing MLP is used

Original: 4. O BJECT D ETECTION ON COCO

Original: Settings.

Original: For the object detection and instance segmentation, we employ mmdetection (Chen et al., 2019) as the framework and COCO (Lin et al., 2014) as the evaluation dataset, which consists of 118K training data and 5K validation data.

Original: We compare the performance of our AS-MLP with other

Original: [Table]
Revised: [Table]

Original: backbones on COCO.
Revised: MLP block.

Original: Following Swin Transformer (Liu et al., 2021b), we consider two typical object detection frameworks: Mask R-CNN (He et al., 2017) and Cascade R-CNN (Cai & Vasconcelos, 2018). The training strategies are as follows: optimizer (AdamW), learning rate (0.0001), weight decay (0.05), and batch size ( 2 imgs/per GPU × 8 GPUs).
Revised: The Impact of AS-MLP Block.

Original: We utilize the typical multi-scale training strategy (Carion et al., 2020;
Revised: We also evaluate the impact of AS-MLP block in Table 4. Here we design ﬁve baselines: i)

Original: Sun et al., 2021)
Revised: Global-MLP; ii) AxialMLP; iii) Window-MLP; iv) shift size (5, 1); v) shift size (1, 5).

Original: (the shorter side is between 480 and 800 and the longer side is at most 1333). All backbones are initialized with weights pre-trained on ImageNet-1K and all models are trained with 3x schedule (36 epochs).
Revised: The ﬁrst three baselines are designed from the perspective of how to use MLP for feature fusion at different positions, and the latter two are designed from the perspective of the axial shift in a single direction.

Original: Results.
Revised: The speciﬁc settings are listed in Appendix A.5. The results in Table 4 show that our AS-MLP block with shift size (5, 5) outperforms other baselines. 4. O BJECT D ETECTION ON COCO

Original: The  results are shown in Table 4.
Revised: The experimental setting is listed in Appendix A.4, and the results are shown in Table 5.

Original: It is worth noting that we do not compare our method with MLP-Mixer (Tolstikhin et al., 2021) because it uses a ﬁxed spatial dimension for token-mixing MLP, which cannot be transferred to the object detection.
Revised: It is worth noting that we do not compare our method with MLP-Mixer (Tolstikhin et al., 2021) because it uses a ﬁxed spatial dimension for token-mixing MLP, which cannot be transferred to the object detection.

Original: As far as we know, we are the ﬁrst work to apply MLP-based architecture to object detection.
Revised: As far as we know, we are the ﬁrst work to apply MLP-based architecture to object detection.

Original: Our AS-MLP achieves comparable performance with Swin Transformer in the similar resource limitation.
Revised: Our AS-MLP achieves comparable performance with Swin Transformer in the similar resource limitation.

Original: To be speciﬁc, Cascade Mask R-CNN + Swin-B achieves 51.9 AP b with 145M parameters and 982 GFLOPs, and Cascade Mask R-CNN + AS-MLP-B obtains 51.5 AP b with 145M parameters and 961 GFLOPs.
Revised: To be speciﬁc, Cascade Mask R-CNN + Swin-B achieves 51.9 AP b with 145M parameters and 982 GFLOPs, and Cascade Mask R-CNN + AS-MLP-B obtains 51.5 AP b with 145M parameters and 961 GFLOPs.

Original: The visualizations of object detection are shown in Appendix C.
Revised: The visualizations of object detection are shown in Appendix C.

Original: 4. S EMANTIC S EGMENTATION ON ADE20K
Revised: S EMANTIC S EGMENTATION ON ADE20K

Original: Settings.

Original: Following Swin Transformer (Liu et al., 2021b), we conduct experiments of AS-MLP on the challenging semantic segmentation dataset, ADE20K, which contains 20,210 training images and 2,000 validation images.

Original: We utilize UperNet (Xiao et al., 2018) and AS-MLP backbone as our main experimental results.

Original: The framework is based on mmsegmentation (Contributors, 2020).

Original: The training strategies are as follows: optimizer (AdamW), learning rate ( 6 × 10 − 5 ), weight decay (0.01), and batch size ( 2 imgs/per GPU × 8 GPUs).

Original: We utilize random horizontal ﬂipping, random re-scaling within ratio range [0.5, 2.0] and random photometric distortion as data augmentation.

Original: The input image resolution is 512 × 512 , the stochastic depth ratio is set as 0.3 and all models are initialized with weights pre-trained on ImageNet-1K and are trained 160K iterations.

Original: Results. Table 5 shows the performance of our AS-MLP on the ADE20K dataset.
Revised: The experimental setting is listed in Appendix A.4 and Table 6 shows the performance of our ASMLP on the ADE20K dataset.

Original: Note that we are also the ﬁrst to apply the MLP-based architecture to semantic segmentation.
Revised: Note that we are also the ﬁrst to apply the MLP-based architecture to semantic segmentation.

Original: The results are impressive.
Revised: The results are impressive.

Original: With slightly fewer FLOPs, AS-MLP-T achieves better result than Swin-T (46.5 vs. 45.8 MS mIoU).
Revised: With slightly fewer FLOPs, AS-MLP-T achieves better result than Swin-T (46.5 vs. 45.8 MS mIoU).

Original: For the large model, UperNet + Swin-B has 49.7 MS mIoU with 121M parameters and 1188 GFLOPs, and UperNet + AS-MLP-B has 49.5 MS mIoU with 121M parameters and 1166 GFLOPs, which also shows the effectiveness of our AS-MLP architecture in processing the downstream task.
Revised: For the large model, UperNet + SwinB has 49.7 MS mIoU with 121M parameters and 1188 GFLOPs, and UperNet + AS-MLP-B has 49. MS mIoU with 121M parameters and 1166 GFLOPs, which also shows the effectiveness of our ASMLP architecture in processing the downstream task.

Original: The visualizations of semantic segmentation are shown in Appendix C.
Revised: The visualizations of semantic segmentation are shown in Appendix C.

Original: [Table]
Revised: [Table]

Original: 4. V ISUALIZATION
Revised: 4. V ISUALIZATION

Original: [Figure]
Revised: [Figure]

Original: of features from Swin
Revised: of features from Swin

Original: Transformer and our AS-MLP.
Revised: Transformer and our AS-MLP.

Original: We visualize the heatmap of learned features from Swin Transformer and AS-MLP in Figure 4, where the ﬁrst column shows the image from ImageNet, and the second column shows the activation heatmap of the last layer of Swin transformer (Swin-B).
Revised: We visualize the heatmap of learned features from Swin Transformer and AS-MLP in Figure 4, where the ﬁrst column shows the image from ImageNet, and the second column shows the activation heatmap of the last layer of Swin transformer (Swin-B).

Original: The third, fourth, and ﬁfth columns respectively indicate the response after the horizontal shift (AS-MLP (h)), the vertical shift (AS-MLP (v)) and the combination of both in the last layer of AS-MLP (AS-MLPB).
Revised: The third, fourth, and ﬁfth columns respectively indicate the response after the horizontal shift (AS-MLP (h)), the vertical shift (AS-MLP (v)) and the combination of both in the last layer of AS-MLP (AS-MLPB).

Original: From Figure 4, one can see that i) AS-MLP can better focus on object regions compared to
Revised: From Figure 4, one can see that i) AS-MLP can better focus on object regions compared to

Original: Swin transformer; ii) AS-MLP (h) can better focus on the vertical part of objects (as shown in the second row) because it shifts feature in the horizontal direction.
Revised: Swin transformer; ii) AS-MLP (h) can better focus on the vertical part of objects (as shown in the second row) because it shifts feature in the horizontal direction.

Original: It is more reasonable because the shift in the horizontal direction can cover the edge of the vertical part, which is more helpful for recognizing the object.
Revised: It is more reasonable because the shift in the horizontal direction can cover the edge of the vertical part, which is more helpful for recognizing the object.

Original: Similarly, AS-MLP (v) can better focus on the horizontal part of objects (as shown in the fourth row).
Revised: Similarly, AS-MLP (v) can better focus on the horizontal part of objects (as shown in the fourth row).

Original: These phenomena further show the superiority of AS-MLP.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we propose an axial shifted MLP architecture, named AS-MLP, for vision.
Revised: In this paper, we propose an axial shifted MLP architecture, named AS-MLP, for vision.

Original: Compared with MLP-Mixer, we pay more attention to the local features extraction and make full use of the channel interaction between different spatial positions through a simple feature axial shift.
Revised: Compared with MLP-Mixer, we pay more attention to the local features extraction and make full use of the channel interaction between different spatial positions through a simple feature axial shift.

Original: With the proposed AS-MLP, we further improve the performance of MLP-based architecture and the experimental results are impressive.
Revised: With the proposed AS-MLP, we further improve the performance of MLP-based architecture and the experimental results are impressive.

Original: Our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset.
Revised: Our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset.

Original: Such a simple yet effective method outperforms all MLP-based architectures and achieves competitive performance compared to the transformerbased architectures even with slightly lower FLOPs.
Revised: Such a simple yet effective method outperforms all MLP-based architectures and achieves competitive performance compared to the transformerbased architectures even with slightly lower FLOPs.

Original: We are also the ﬁrst work to apply AS-MLP to the downstream tasks ( e.g. , object detection and semantic segmentation).
Revised: We are also the ﬁrst work to apply AS-MLP to the downstream tasks ( e.g. , object detection and semantic segmentation).

Original: The results are also competitive or even better compared to transformer-based architectures, which shows the ability of
Revised: The results are also competitive or even better compared to transformer-based architectures, which shows the ability of

Original: MLP-based architectures in handling downstream tasks.
Revised: MLP-based architectures in handling downstream tasks.

Original: For future work, we will investigate the effectiveness of AS-MLP in natural language processing, and further explore the performance of AS-MLP on downstream tasks.
Revised: For future work, we will investigate the effectiveness of AS-MLP in natural language processing, and further explore the performance of AS-MLP on downstream tasks.

Original: A T HE A RCHITECTURE D ETAILS A. T HE D ETAILED C ONFIGURATIONS OF D IFFERENT A RCHITECTURES
Revised: A T HE A RCHITECTURE D ETAILS A. T HE D ETAILED C ONFIGURATIONS OF D IFFERENT A RCHITECTURES

Original: We show the detailed conﬁgurations of different architectures in Table 6, where we assume the size of the input image is 224 × 224 .
Revised: We show the detailed conﬁgurations of different architectures in Table 7, where we assume the size of the input image is 224 × 224 .

Original: The second column shows the output size of the image after each stage.
Revised: The second column shows the output size of the image after each stage.

Original: Following Swin Transformer (Liu et al., 2021b), we use “Concat n × n ” to indicate a concatenation of n × n neighboring features in a patch.
Revised: Following Swin Transformer (Liu et al., 2021b), we use “Concat n × n ” to indicate a concatenation of n × n neighboring features in a patch.

Original: “shift size (5, 5)” means that the shift size in the horizontal and vertical directions is
Revised: “shift size (5, 5)” means that the shift size in the horizontal and vertical directions is

Original: [Table]
Revised: [Table]

Original: A. T HE C OMPUTATIONAL C OMPLEXITY OF AS-MLP A RCHITECTURE
Revised: A. T HE C OMPUTATIONAL C OMPLEXITY OF AS-MLP A RCHITECTURE

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we show the speciﬁc computational complexity in each layer of AS-MLP architecture.
Revised: we show the speciﬁc computational complexity in each layer of AS-MLP architecture.

Original: The symbol deﬁnition is ﬁrstly given as follows.
Revised: The symbol deﬁnition is ﬁrst given as follows.

Original: An input image: I ∈ R 3 × H × W ; patch size ( p, p ) ; the number of blocks in four stages: { n 1 , n 2 , n 3 , n 4 } ; Channel dimension C ; MLP ratio: r .
Revised: An input image: I ∈ R 3 × H × W ; patch size ( p, p ) ; the number of blocks in four stages: { n 1 , n 2 , n 3 , n 4 } ; Channel dimension C ; MLP ratio: r .

Original: The speciﬁc computational complexity is shown in Table
Revised: The speciﬁc computational complexity is shown in Table

Original: [Table]
Revised: [Table]

Original: where only convolution operation is computed.
Revised: where only convolution operation is computed.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: A. T HE N ETWORK D ETAILS IN THE M OBILE S ETTING
Revised: A. T HE N ETWORK D ETAILS IN THE M OBILE S ETTING

Original: In addition to AS-MLP-T, AS-MLP-S, and AS-MLP-B, we also design AS-MLP in the mobile setting.
Revised: In addition to AS-MLP-T, AS-MLP-S, and AS-MLP-B, we also design AS-MLP in the mobile setting.

Original: For a fair comparison, we modify the Swin Transformer correspondingly to adopt to the mobile setting.
Revised: For a fair comparison, we modify the Swin Transformer correspondingly to adopt to the mobile setting.

Original: The conﬁgurations are as follow:
Revised: The conﬁgurations are as follow:

Original: • Swin (mobile): C = 64, the number of blocks in four stages = { 2, 2, 2, 2 } , the number of heads = { 2, 4, 8, 16 } ; • AS-MLP (mobile): C = 64, the number of blocks in four stages = { 2, 2, 2, 2 } ;
Revised: • Swin (mobile): C = 64, the number of blocks in four stages = { 2, 2, 2, 2 } , the number of heads = { 2, 4, 8, 16 } ; • AS-MLP (mobile): C = 64, the number of blocks in four stages = { 2, 2, 2, 2 } ;

Original: B M ORE E XPERIMENTS B. M ORE E XPERIMENTAL R ESULTS ON COCO
Revised: A. T HE S ETTINGS OF O BJECT D ETECTION AND S EMANTIC S EGMENTATION

Revised: Object Detection on COCO.

Revised: For the object detection and instance segmentation, we employ mmdetection (Chen et al., 2019) as the framework and COCO (Lin et al., 2014) as the evaluation dataset, which consists of 118K training data and 5K validation data.

Revised: We compare the performance of our

Revised: AS-MLP with other backbones on COCO.

Revised: Following Swin Transformer (Liu et al., 2021b), we consider two typical object detection frameworks: Mask R-CNN (He et al., 2017) and Cascade R-CNN (Cai & Vasconcelos, 2018).

Revised: The training strategies are as follows: optimizer (AdamW), learning rate (0.0001), weight decay (0.05), and batch size ( 2 imgs/per GPU × 8 GPUs).

Revised: We utilize the typical multi-scale training strategy (Carion et al., 2020;

Revised: Sun et al., 2021)

Revised: (the shorter side is betweenand 800 and the longer side is at most 1333).

Revised: All backbones are initialized with weights pre-trained on ImageNet-1K and all models are trained with 3x schedule (36 epochs).

Revised: Semantic Segmentation on ADE20K. Following Swin Transformer (Liu et al., 2021b), we conduct experiments of AS-MLP on the challenging semantic segmentation dataset, ADE20K, which contains 20,210 training images and 2,000 validation images.

Revised: We utilize UperNet (Xiao et al., 2018) and AS-MLP backbone as our main experimental results.

Revised: The framework is based on mmsegmentation (Contributors, 2020).

Revised: The training strategies are as follows: optimizer (AdamW), learning rate ( 6 × 10 − 5 ), weight decay (0.01), and batch size ( 2 imgs/per GPU × 8 GPUs).

Revised: We utilize random horizontal ﬂipping, random re-scaling within ratio range [0.5, 2.0] and random photometric distortion as data augmentation.

Revised: The input image resolution is 512 × 512 , the stochastic depth ratio is set as 0.3 and all models are initialized with weights pre-trained on ImageNet-1K and are trained 160K iterations.

Revised: A. B ASELINES

Revised: We list the speciﬁc conﬁgurations of baselines in Sec.

Revised: 4.2 as follows.

Revised: • Global-MLP: following MLP-Mixer (Tolstikhin et al., 2021), we use global MLP (tokenmixing MLP) along with full spatial size instead of AS-MLP block in our architecture conﬁgurations.

Revised: For Global-MLP, the model weights trained with ﬁxed image size cannot be adapted to downstream tasks with various input sizes.

Revised: • Axial-MLP: built upon Global-MLP, Axial-MLP employs two axial MLPs along with horizontal and vertical directions instead of global MLP.

Revised: Similar to Global-MLP, the model weights trained with ﬁxed image size cannot be adapted to downstream tasks with various input sizes.

Revised: • Window-MLP: as stated in Sec 1, we set ﬁxed window ( 7 × 7 ) in our architecture conﬁgurations and perform MLP operations within the window.

Revised: • Shift size (5, 1): horizontal shift is 5 and vertical shift is 1 in AS-MLP block.

Revised: • Shift size (1, 5): horizontal shift is 1 and vertical shift is 5 in AS-MLP block.

Revised: B

Revised: M

Original:  Table 4 lists the object detection and instance segmentation results of different backbones with 3x schedule (36 epochs).
Revised: [Equation] E XPERIMENTS B. M ORE E XPERIMENTAL R ESULTS ON COCO Table 5 lists the object detection and instance segmentation results of different backbones with 3x schedule (36 epochs).

Original: For a complete comparison, we also conductthe experiments with 1x schedule  (12 epochs).
Revised: For a complete comparison, we also conduct experiments with 1x schedule [Table] (12 epochs).

Original: The results are shown in Table 8.
Revised: The results are shown in Table 9.

Original: Our AS-MLP-T outperforms Swin-T (Liu et al., 2021b) under the Mask R-CNN (44.0 vs. 43.7 AP b ) and Cascade Mask R-CNN (48.4 vs. 48.1 AP b ) frameworks.
Revised: Our AS-MLP-T outperforms Swin-T (Liu et al., 2021b) under the Mask R-CNN (44.0 vs. 43.7 AP b ) and Cascade Mask R-CNN (48.4 vs. 48.1 AP b ) frameworks.

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: B. T HE C OMPLETE C LASSIFICATION A CCURACY AND T HROUGHPUT C OMPARISON
Revised: B. T HE C OMPLETE C LASSIFICATION A CCURACY AND T HROUGHPUT C OMPARISON

Original: Table 9 shows the complete accuracy comparison with other state-of-the-art architectures on the
Revised: Table 10 shows the complete accuracy comparison with other state-of-the-art architectures on the

Original: ImageNet-1K dataset.
Revised: ImageNet-1K dataset.

Original: In addition, Table 1 shows the throughput results of the AS-MLP architecture measured with the batch size 64 on a single V100 GPU (32GB).
Revised: In addition, Table 1 shows the throughput results of the AS-MLP architecture measured with the batch size 64 on a single V100 GPU (32GB).

Original: In order to make a fair comparison with other papers, we also conduct a thorough evaluation of throughput.
Revised: In order to make a fair comparison with other papers, we also conduct a thorough evaluation of throughput.

Original: The results are shown in
Revised: The results are shown in

Original: Figure 5, where we list the throughputs when the batch size is 1, 4, 8, 16, 32, 64, 128, respectively.
Revised: Figure 5, where we list the throughputs when the batch size is 1, 4, 8, 16, 32, 64, 128, respectively.

Original: B. E VALUATION A CCURACY
Revised: B. E VALUATION A CCURACY

Original: In Figure 6, we visualize the evaluation accuracy of AS-MLP and Swin transformer on the ImageNet,
Revised: In Figure 6, we visualize the evaluation accuracy of AS-MLP and Swin transformer on the ImageNet,

Original: COCO and ADE20K datasets during training.
Revised: COCO and ADE20K datasets during training.

Original: For image classiﬁcation on ImageNet, AS-MLP-T keeps pace with Swin-T in each epoch and they ﬁnally converge to the similar accuracy (81.3 vs.
Revised: For image classiﬁcation on ImageNet, AS-MLP-T keeps pace with Swin-T in each epoch and they ﬁnally converge to the similar accuracy (81.3 vs.

Original: For object detection and semantic segmentation on COCO and ADE20K, we can see that 
Revised: For object detection and semantic segmentation on COCO and ADE20K, we can see that [Table] [Table]

Original: AS-MLP-T achieves better performance than Swin-T in the early stage, and keeps winning during the training process. 
Revised: AS-MLP-T achieves better performance than Swin-T in the early stage, and keeps winning during the training process. B. C OMPARISONS TO S HIFT R ES N ET In this section

Original: [Figure]
Revised: [Table]

Revised: we compare the performance with ShiftResNet (Wu et al., 2018) in Table 11.

Revised: The results of ShiftResNet50 with different conﬁgurations are from Table 3 of ShiftResNet paper (Wu et al., 2018).

Revised: Our AS-MLP (mobile) achieves better accuracy with less parameters

Revised: [Table]

Revised: [Figure]

Original: C T HE V ISUALIZATION OF R ESULTS ON COCO AND ADE20K
Revised: C T HE V ISUALIZATION OF R ESULTS ON COCO AND ADE20K

Original: We visualize the object detection and instance segmentation results on the COCO dataset in Figure
Revised: We visualize the object detection and instance segmentation results on the COCO dataset in Figure

Original: [Figure]
Revised: [Figure]

Original: where the Cascade Mask R-CNN model with the AS-MLP-T backbone is used.
Revised: where the Cascade Mask R-CNN model with the AS-MLP-T backbone is used.

Original: We also visualize the semantic segmentation results on the ADE20K dataset in Figure
Revised: We also visualize the semantic segmentation results on the ADE20K dataset in Figure

Original: [Figure]
Revised: [Figure]

Original: where we utilize the UperNet model with the AS-MLP-T backbone.
Revised: where we utilize the UperNet model with the AS-MLP-T backbone.

Original: The object can be detected and segmented correctly.
Revised: The object can be detected and segmented correctly.