Original: C YCLE MLP: A MLP- LIKE A RCHITECTURE FOR D ENSE P REDICTION
Revised: C YCLE MLP: A MLP- LIKE A RCHITECTURE FOR D ENSE P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper presents a simple MLP-like architecture, CycleMLP, which is a versatile backbone for visual recognition and dense predictions.
Revised: This paper presents a simple MLP-like architecture, CycleMLP, which is a versatile backbone for visual recognition and dense predictions.

Original: As compared to modern MLP architectures, e.g. , MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Touvron et al., 2021a), and gMLP (Liu et al., 2021a), whose architectures are correlated to image size and thus are infeasible in object detection and segmentation,
Revised: As compared to modern MLP architectures, e.g. , MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Touvron et al., 2021a), and gMLP (Liu et al., 2021a), whose architectures are correlated to image size and thus are infeasible in object detection and segmentation,

Original: CycleMLP has two advantages compared to modern approaches.
Revised: CycleMLP has two advantages compared to modern approaches.

Original: (1) It can cope with various image sizes.
Revised: (1) It can cope with various image sizes.

Original: (2) It achieves linear computational complexity to image size by using local windows.
Revised: (2) It achieves linear computational complexity to image size by using local windows.

Original: In contrast, previous MLPs have O ( N 2 ) computations due to fully spatial connections.
Revised: In contrast, previous MLPs have O ( N 2 ) computations due to fully spatial connections.

Original: We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e.g.
Revised: We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e.g.

Original: Swin Transformer (Liu et al., 2021b), while using fewer parameters and FLOPs.
Revised: Swin Transformer (Liu et al., 2021b), while using fewer parameters and FLOPs.

Original: We expand the MLP-like models’ applicability, making them a versatile backbone for dense prediction tasks.
Revised: We expand the MLP-like models’ applicability, making them a versatile backbone for dense prediction tasks.

Original: CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation.
Revised: CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation.

Original: In particular,
Revised: In particular,

Original: CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K dataset with fewer FLOPs.
Revised: CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K dataset with fewer FLOPs.

Original: Moreover, CycleMLP also shows excellent zero-shot robustness on
Revised: Moreover, CycleMLP also shows excellent zero-shot robustness on

Original: ImageNet-C dataset.
Revised: ImageNet-C dataset.

Revised: Code is available in the supplementary material.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Vision models in computer vision have been long dominated by convolutional neural networks (CNNs) (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
Revised: Vision models in computer vision have been long dominated by convolutional neural networks (CNNs) (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Recently, inspired by the successes in Natural Language Processing (NLP) ﬁeld, Transformers (Vaswani et al., 2017) are adopted into the computer vision community.
Revised: Recently, inspired by the successes in Natural Language Processing (NLP) ﬁeld, Transformers (Vaswani et al., 2017) are adopted into the computer vision community.

Original: Built with multi-head self-attention layers, multi-layer perceptrons (MLPs), and skip connections (He et al., 2016), Transformers make numerous breakthroughs on visual tasks (Dosovitskiy et al., 2020; Touvron et al., 2020; Liu et al., 2021b).
Revised: Built with multi-head self-attention layers, multi-layer perceptrons (MLPs), and skip connections (He et al., 2016), Transformers make numerous breakthroughs on visual tasks (Dosovitskiy et al., 2020; Touvron et al., 2020; Liu et al., 2021b).

Original: More recently, (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) have validated that building models solely on MLPs and skip connections without the self-attention layers can achieve surprisingly promising results on ImageNet (Deng et al., 2009) classiﬁcation.
Revised: More recently, (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) have validated that building models solely on MLPs and skip connections without the self-attention layers can achieve surprisingly promising results on ImageNet (Deng et al., 2009) classiﬁcation.

Original: Compared with CNNs, these MLP-based models even achieve comparable recognition performance when trained on large scale datasets ( e.g., ImageNet-21K and JFT-300M).
Revised: Compared with CNNs, these MLP-based models even achieve comparable recognition performance when trained on large scale datasets ( e.g., ImageNet-21K and JFT-300M).

Original: [Table]
Revised: [Table]

Original: Despite promising results on visual recognition tasks, these MLP-like models can not be used in dense prediction tasks ( e.g., object detection and semantic segmentation) due to the three challenges: (1) Current models are composed of blocks with non-hierarchical architectures, which make the model infeasible to provide pyramid and high-resolution feature representations.
Revised: Despite promising results on visual recognition tasks, these MLP-like models can not be used in dense prediction tasks ( e.g., object detection and semantic segmentation) due to the three challenges: (1) Current models are composed of blocks with non-hierarchical architectures, which make the model infeasible to provide pyramid and high-resolution feature representations.

Original: (2) Current models cannot deal with ﬂexible input scales due to the Spatial FC as shown in Figure 1b.
Revised: (2) Current models cannot deal with ﬂexible input scales due to the Spatial FC as shown in Figure 1b.

Original: The spatial FC is conﬁgured by an image-size related weight 1 , which requires input image to have a ﬁxed scale.
Revised: The spatial FC is conﬁgured by an image-size related weight 1 , which requires input image to have a ﬁxed scale.

Original: Thus, this structure typically requires the input image with
Revised: Thus, this structure typically requires the input image with

Original: [Figure]

Original:  𝐶 
Revised: 𝐻×𝑊 𝐶 (

Original: [Figure]
Revised: [Figure]

Original:  𝐻×𝑊 𝐶 (
Revised: FC 𝐻×𝑊 𝐶 (b)

Original: [Figure] [Figure] FC 𝐶 𝑊 𝐻 Stepsize 𝛿 ! 

Revised: Spatial FC 𝐻×𝑊 𝐶 (c) Cycle FC 𝐶 𝑊 𝐻 Stepsize 𝛿 !

Original: =- 𝛿 ! 1 =0 𝛿 ! 2 = 𝛿 !
Revised: =- 𝛿 ! 1 =0 𝛿 ! 2 = 𝛿 !

Original: 7 =-1 𝛿 ! 

Revised: 7 =-1 𝛿 !

Original: = 𝑆 !=
Revised: = 𝑆 !

Revised: = 3,𝑆 "

Original: 𝑆 ! =
Revised: =𝑆 ! =

Original: 𝐻 𝑆 !
Revised: 𝐻,𝑆 " =𝑆 !

Original: =
Revised: =

Original: 𝛿 ! 9 =
Revised: 1,𝑆 " =𝛿 ! 9 =

Original: (d) (e) (f)  a ﬁxed scale during both the training and inference procedure.
Revised: (d) (e) (f) For simplicity, omit batch dimension and more can be found in a ﬁxed scale during both the training and inference procedure.

Original: It contradicts the requirements of dense prediction tasks, which usually adopt a multi-scale training strategy (Carion et al., 2020) and different input resolutions in training and inference stages (Lin et al., 2014; Cordts et al., 2016).
Revised: It contradicts the requirements of dense prediction tasks, which usually adopt a multi-scale training strategy (Carion et al., 2020) and different input resolutions in training and inference stages (Lin et al., 2014; Cordts et al., 2016).

Original: (3) The computational and memory costs of the current MLP models are quadratic to input image sizes for dense prediction tasks (e.g., COCO benchmark (Lin et al., 2014)).
Revised: (3) The computational and memory costs of the current MLP models are quadratic to input image sizes for dense prediction tasks (e.g., COCO benchmark (Lin et al., 2014)).

Original: To address the ﬁrst challenge, we construct a hierarchical architecture to generate pyramid features.
Revised: To address the ﬁrst challenge, we construct a hierarchical architecture to generate pyramid features.

Original: For the second and third issues, we propose a novel variant of fully connected layer, named as Cycle
Revised: For the second and third issues, we propose a novel variant of fully connected layer, named as Cycle

Original: Fully-Connected Layer (Cycle FC) , as illustrated in Figure 1c.
Revised: Fully-Connected Layer (Cycle FC) , as illustrated in Figure 1c.

Original: The Cycle FC is capable of dealing with various image scales and has linear computational complexity to image size.
Revised: The Cycle FC is capable of dealing with various image scales and has linear computational complexity to image size.

Original: Our Cycle FC is inspired by Channel FC layer illustrated in Figure 1a, which is designed for channel information communication (Lin et al., 2013; Szegedy et al., 2015;
Revised: Our Cycle FC is inspired by Channel FC layer illustrated in Figure 1a, which is designed for channel information communication (Lin et al., 2013; Szegedy et al., 2015;

Original: He et al., 2016; Howard et al., 2017).
Revised: He et al., 2016; Howard et al., 2017).

Original: The main spirit of Channel FC lies in that it can deal with ﬂexible image sizes since it is conﬁgured by image-size agnostic weight of C in and C out .
Revised: The main spirit of Channel FC lies in that it can deal with ﬂexible image sizes since it is conﬁgured by image-size agnostic weight of C in and C out .

Original: However, the Channel FC is infeasible to aggregate spatial context information due to its limited receptive ﬁeld.
Revised: However, the Channel FC is infeasible to aggregate spatial context information due to its limited receptive ﬁeld.

Original: Our Cycle FC is designed to enjoy Channel FC’s merit of taking input with arbitrary resolution and linear computational complexity while enlarging its receptive ﬁeld for context aggregation.
Revised: Our Cycle FC is designed to enjoy Channel FC’s merit of taking input with arbitrary resolution and linear computational complexity while enlarging its receptive ﬁeld for context aggregation.

Original: Specifically, Cycle FC samples points in a cyclical style along the channel dimension (Figure 1c).
Revised: Specifically, Cycle FC samples points in a cyclical style along the channel dimension (Figure 1c).

Original: In this way, Cycle FC has the same complexity (both the number of parameters and FLOPs) as channel
Revised: In this way, Cycle FC has the same complexity (both the number of parameters and FLOPs) as channel

Original: FC while increasing the receptive ﬁeld simultaneously.
Revised: FC while increasing the receptive ﬁeld simultaneously.

Original: To this end, we adopt Cycle FC to replace the Spatial FC for spatial context aggregation ( i.e., token mixing) and build a family of MLP-like models for both recognition and dense prediction tasks.
Revised: To this end, we adopt Cycle FC to replace the Spatial FC for spatial context aggregation ( i.e., token mixing) and build a family of MLP-like models for both recognition and dense prediction tasks.

Original: The contributions of this paper are as follows: (1) We propose a new MLP-like operator, Cycle FC, which is computational friendly to cope with ﬂexible input resolutions.
Revised: The contributions of this paper are as follows: (1) We propose a new MLP-like operator, Cycle FC, which is computational friendly to cope with ﬂexible input resolutions.

Original: (2) We take the ﬁrst attempt to build a family of hierarchical MLP-like architectures (CycleMLP) based on Cycle FC operator for dense prediction tasks.
Revised: (2) We take the ﬁrst attempt to build a family of hierarchical MLP-like architectures (CycleMLP) based on Cycle FC operator for dense prediction tasks.

Original: (3) Extensive experiments on various tasks (e.g., ImageNet classiﬁcation,
Revised: (3) Extensive experiments on various tasks (e.g., ImageNet classiﬁcation,

Original: COCO object instance detection, and segmentation, and ADE20K semantic segmentation) demonstrate that CycleMLP outperforms existing MLP-like models and is comparable to and sometimes better than CNNs and Transformers on dense predictions.
Revised: COCO object instance detection, and segmentation, and ADE20K semantic segmentation) demonstrate that CycleMLP outperforms existing MLP-like models and is comparable to and sometimes better than CNNs and Transformers on dense predictions.

Original: Related Work.
Revised: Related Work.

Original: Convolution Neural Networks (CNNs) has dominated the visual backbones for several years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
Revised: Convolution Neural Networks (CNNs) has dominated the visual backbones for several years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: (Dosovitskiy
Revised: (Dosovitskiy

Original: [Figure]
Revised: [Figure]

Original: All models are trained on ImageNet-1K (Deng et al.
Revised: All models are trained on ImageNet-1K (Deng et al.

Original: 2009) without extra data.
Revised: 2009) without extra data.

Original: CycleMLP surpasses existing MLP-like models such as MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Touvron et al., 2021a), gMLP (Liu et al., 2021a), S 2 -MLP (Yu et al., 2021) and ViP (Hou et al., 2021).
Revised: CycleMLP surpasses existing MLP-like models such as MLP-Mixer (Tolstikhin et al., 2021), ResMLP (Touvron et al., 2021a), gMLP (Liu et al., 2021a), S 2 -MLP (Yu et al., 2021) and ViP (Hou et al., 2021).

Original: et al., 2020) introduced the ﬁrst pure Transformer-based (Vaswani et al., 2017) model into computer vision and achieved promising performance, especially pre-trained on the large scale JFT dataset.
Revised: et al., 2020) introduced the ﬁrst pure Transformer-based (Vaswani et al., 2017) model into computer vision and achieved promising performance, especially pre-trained on the large scale JFT dataset.

Original: Recently, some works (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) removed the attention in Transformer and proposed pure MLP-based models.
Revised: Recently, some works (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) removed the attention in Transformer and proposed pure MLP-based models.

Original: Please see Appendix A for a comprehensive review of the literature on the visual backbones.
Revised: Please see Appendix A for a comprehensive review of the literature on the visual backbones.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce CycleMLP models for vision tasks including recognition and dense predictions.
Revised: In this section, we introduce CycleMLP models for vision tasks including recognition and dense predictions.

Original: To begin with, in Sec.
Revised: To begin with, in Sec.

Original: 2.1 we formulate our proposed novel operator, Cycle FC, which serves as a basic component for building CycleMLP models.
Revised: 2.1 we formulate our proposed novel operator, Cycle FC, which serves as a basic component for building CycleMLP models.

Original: Then we compare Cycle FC with
Revised: Then we compare Cycle FC with

Original: Channel FC and multi-head attention adopted in recent Transformer-based models (Dosovitskiy et al., 2020; Touvron et al., 2020; Liu et al., 2021b) in Sec. 2.2.
Revised: Channel FC and multi-head attention adopted in recent Transformer-based models (Dosovitskiy et al., 2020; Touvron et al., 2020; Liu et al., 2021b) in Sec. 2.2.

Original: Finally, we present the detailed conﬁgurations of CycleMLP models in Sec. 2.3.
Revised: Finally, we present the detailed conﬁgurations of CycleMLP models in Sec. 2.3.

Original: 2. C YCLE F ULLY -C ONNECTED L AYER
Revised: 2. C YCLE F ULLY -C ONNECTED L AYER

Original: Notation.
Revised: Notation.

Original: We denote an input feature map as X ∈ R H × W × C in , where H, W denote the height and width of the image and C in is the number of feature channels.
Revised: We denote an input feature map as X ∈ R H × W × C in , where H, W denote the height and width of the image and C in is the number of feature channels.

Original: We use subscripts to index the feature map.
Revised: We use subscripts to index the feature map.

Original: For example, X i,j,c is the value of c th channel at the spatial position ( i, j ) and X i,j, : are values of all channels at the spatial ( i, j ) .
Revised: For example, X i,j,c is the value of c th channel at the spatial position ( i, j ) and X i,j, : are values of all channels at the spatial ( i, j ) .

Original: The motivation behind Cycle FC is to enlarge receptive ﬁeld of MLP-like models to cope with downstream dense prediction tasks while maintaining the computational efﬁciency.
Revised: The motivation behind Cycle FC is to enlarge receptive ﬁeld of MLP-like models to cope with downstream dense prediction tasks while maintaining the computational efﬁciency.

Original: As illustrated in Figure 1a, Channel FC applies weighting matrix on X along the channel dimension on ﬁxed position ( i, j ) .
Revised: As illustrated in Figure 1a, Channel FC applies weighting matrix on X along the channel dimension on ﬁxed position ( i, j ) .

Original: However, Cycle FC introduces a receptive ﬁeld of ( S H , S W ) , where S H and S W are stepsize along with the height and width dimension respectively (illustrated in Figure 1 (d)).
Revised: However, Cycle FC introduces a receptive ﬁeld of ( S H , S W ) , where S H and S W are stepsize along with the height and width dimension respectively (illustrated in Figure 1 (d)).

Original: The basic Cycle FC operator can be formulated as below:
Revised: The basic Cycle FC operator can be formulated as below:

Original: [Equation]
Revised: [Equation]

Original: where W mlp ∈ R C in × C out and b ∈ R C out are parameters of Cycle FC.
Revised: where W mlp ∈ R C in × C out and b ∈ R C out are parameters of Cycle FC.

Original: δ i ( c ) and δ j ( c ) are the spatial offset of the two axis on the c th channel, which are deﬁned as below:
Revised: δ i ( c ) and δ j ( c ) are the spatial offset of the two axis on the c th channel, which are deﬁned as below:

Original: [Equation]
Revised: [Equation]

Original: Examples.
Revised: Examples.

Original: We provide several examples (Figure 1 (d)-(f)) to illustrate the stepsize.
Revised: We provide several examples (Figure 1 (d)-(f)) to illustrate the stepsize.

Original: For the sake of visualization convenience, we set the tensor’s W = 1 .
Revised: For the sake of visualization convenience, we set the tensor’s W = 1 .

Original: Thus, these three examples naturally all have
Revised: Thus, these three examples naturally all have

Original: S W = 1 .
Revised: S W = 1 .

Original: Figure 1 (d) illustrates the offsets along two axis when S H = 3 , that is δ j ( c ) ≡ 

Revised: Figure 1 (d) illustrates the offsets along two axis when S H = 3 , that is δ j ( c ) ≡

Original: and δ i ( c ) = {− 1 , 0 , 1 , − 1 , 0 , 1 , · · · } when c = 0 , 1 , 2 , · · · , 8 .
Revised: and δ i ( c ) = {− 1 , 0 , 1 , − 1 , 0 , 1 , · · · } when c = 0 , 1 , 2 , · · · , 8 .

Original: Figure 1a (e) shows that when S H = H ,
Revised: Figure 1a (e) shows that when S H = H ,

Original: Cycle FC has a global receptive ﬁeld.
Revised: Cycle FC has a global receptive ﬁeld.

Original: Figure 1a (f) shows that when S H = 1 , there will be no offset along either axis and thus Cycle FC degrades to Channel FC (Figure 1a).
Revised: Figure 1a (f) shows that when S H = 1 , there will be no offset along either axis and thus Cycle FC degrades to Channel FC (Figure 1a).

Original: We also provide a more general case where W (cid:54) = 1 and S H = 3 , S W = 3 in Figure 6 (Appendix).
Revised: We also provide a more general case where W (cid:54) = 1 and S H = 3 , S W = 3 in Figure 7 (Appendix).

Original: The offsets δ i ( c ) and δ j ( c ) enlarge the receptive ﬁeld of Cycle FC as compared to Channel FC (Figure 1a), which applies weights solely on the same spatial position for all channels.
Revised: The offsets δ i ( c ) and δ j ( c ) enlarge the receptive ﬁeld of Cycle FC as compared to Channel FC (Figure 1a), which applies weights solely on the same spatial position for all channels.

Original: The larger receptive ﬁeld in return brings improvements on dense prediction tasks like semantic segmentation and object detection as shown in Table 1.
Revised: The larger receptive ﬁeld in return brings improvements on dense prediction tasks like semantic segmentation and object detection as shown in Table 1.

Original: Meanwhile, Cycle FC still maintains computational efﬁciency and ﬂexibility on input resolution.
Revised: Meanwhile, Cycle FC still maintains computational efﬁciency and ﬂexibility on input resolution.

Original: Both the FLOPs and the number of parameters are linear to the spatial scale which are exactly the same as those of Channel FC.
Revised: Both the FLOPs and the number of parameters are linear to the spatial scale which are exactly the same as those of Channel FC.

Original: In contrast, although Spatial
Revised: In contrast, although Spatial

Original: FC has a global receptive ﬁeld over the whole spatial space, its computational cost is quadratic to the image scale.
Revised: FC has a global receptive ﬁeld over the whole spatial space, its computational cost is quadratic to the image scale.

Original: Besides, it fails to handle inputs with different resolutions.
Revised: Besides, it fails to handle inputs with different resolutions.

Original: 2. C OMPARISON BETWEEN MULTI - HEAD SELF - ATTENTION (MHSA) AND C YCLE FC
Revised: 2. C OMPARISON BETWEEN MULTI - HEAD SELF - ATTENTION (MHSA) AND C YCLE FC

Original: Inspired by Cordonnier et al.
Revised: Inspired by Cordonnier et al.

Original: (2020), when re-parametried properly, a multi-head self-attention layer with N h heads can be formulated as below, which is similar to a convolution with kernel size √ N h × √
Revised: (2020), when re-parametried properly, a multi-head self-attention layer with N h heads can be formulated as below, which is similar to a convolution with kernel size √ N h × √

Original: N h .
Revised: N h .

Original: (Please refer to Appendix C for detailed derivation)
Revised: (Please refer to Appendix C for detailed derivation)

Original: [Equation]
Revised: [Equation]

Original: where W mhsa ,h ∈ R C in × C out is the parameter matrix for h th head in MSHA.
Revised: where W mhsa ,h ∈ R C in × C out is the parameter matrix for h th head in MHSA.

Original: b ∈ R C out is the bias vector.
Revised: b ∈ R C out is the bias vector.

Original: { ∆ i ( h ) , ∆ j ( h ) } = { (0 , 0) , (1 , 0) , ( − 1 , 0) , · · · } contains all possible positional shift in convolution with kernel size √ N h × √ N h .
Revised: { ∆ i ( h ) , ∆ j ( h ) } = { (0 , 0) , (1 , 0) , ( − 1 , 0) , · · · } contains all possible positional shift in convolution with kernel size √ N h × √ N h .

Original: Further, we stack all W mhsa ,h together and reshape it into W mhsa ∈ R K × K × C in × C out .
Revised: Further, we stack all W mhsa ,h together and reshape it into W mhsa ∈ R K × K × C in × C out .

Original: Then a relationship between W mlp and W mhsa can be formulated as follow.
Revised: Then a relationship between W mlp and W mhsa can be formulated as follow.

Original: [Equation]
Revised: [Equation]

Original: equation 4 shows that only the weights of W mhsa on spatial shift ( δ i ( c )+1 , δ j ( c )+1) are taken into account in W mlp .
Revised: equation 4 shows that only the weights of W mhsa on spatial shift ( δ i ( c )+1 , δ j ( c )+1) are taken into account in W mlp .

Original: This indicates that Cycle FC introduce an inductive bias that the weighting matrix in MSHA should be sparse.
Revised: This indicates that Cycle FC introduce an inductive bias that the weighting matrix in MHSA should be sparse.

Original: Thus Cycle FC inherits the large receptive ﬁeld introduced in MSHA .
Revised: Thus Cycle FC inherits the large receptive ﬁeld introduced in MHSA .

Original: The receptive ﬁeld in Cycle FC is enlarged to ( S H , S W ) , which enables Cycle FC to tackle with downstream dense prediction tasks better.
Revised: The receptive ﬁeld in Cycle FC is enlarged to ( S H , S W ) , which enables Cycle FC to tackle with downstream dense prediction tasks better.

Original: Meanwhile, with the sparsity inductive bias, Cycle FC maintains computational efﬁciency in MLP-based methods as compared to convolution and multihead self-attention.
Revised: Meanwhile, with the sparsity inductive bias, Cycle FC maintains computational efﬁciency in MLP-based methods as compared to convolution and multihead self-attention.

Original: The parameter size in Cycle FC is C in × C out
Revised: The parameter size in Cycle FC is C in × C out

Original: while W mhsa ∈ R K × K × C in × C out .
Revised: while W mhsa ∈ R K × K × C in × C out .

Original: 2. O VERALL A RCHITECTURE
Revised: 2. O VERALL A RCHITECTURE

Original: Patch Embedding.
Revised: Patch Embedding.

Original: Given the raw input image with the size of H × W × 3 , our model ﬁrst splits it into patches by a patch embedding module (Dosovitskiy et al., 2020).
Revised: Given the raw input image with the size of H × W × 3 , our model ﬁrst splits it into patches by a patch embedding module (Dosovitskiy et al., 2020).

Original: Each patch is then treated as a “token”.
Revised: Each patch is then treated as a “token”.

Original: Speciﬁcally, we follow (Fan et al., 2021;
Revised: Speciﬁcally, we follow (Fan et al., 2021;

Original: Wang et al., 2021a) to adopt an overlapping patch embedding module with the window size 7 and stride 4.
Revised: Wang et al., 2021a) to adopt an overlapping patch embedding module with the window size 7 and stride 4.

Original: These raw patches are further projected to a higher dimension (denoted as C ) by a linear embedding layer.
Revised: These raw patches are further projected to a higher dimension (denoted as C ) by a linear embedding layer.

Original: Therefore, the overall patch embedding module generates the features with the shape of H 4 × W 4 × C .
Revised: Therefore, the overall patch embedding module generates the features with the shape of H 4 × W 4 × C .

Original: CycleMLP Block.
Revised: CycleMLP Block.

Original: Then, we sequentially apply several Cycle FC Bloc blocks.
Revised: Then, we sequentially apply several Cycle FC Bloc blocks.

Original: Comparing with the previous MLP blocks (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) visualized in Figure 5 (Appendix), the key difference of Cycle FC block is that it utilizes our proposed Cycle
Revised: Comparing with the previous MLP blocks (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a) visualized in Figure 5 (Appendix), the key difference of Cycle FC block is that it utilizes our proposed Cycle

Original: Fully-Connected Layer (Cycle FC) for spatial projection and advances the models in context aggregation and information communication.
Revised: Fully-Connected Layer (Cycle FC) for spatial projection and advances the models in context aggregation and information communication.

Original: Speciﬁcally, the Cycle FC block consists of three parallel
Revised: Speciﬁcally, the Cycle FC block consists of three parallel

Original: Cycle FCs, which have stepsizes S H × S W of 1 × 7 , 7 × 1 , and 1 × 1 .
Revised: Cycle FCs, which have stepsizes S H × S W of 1 × 7 , 7 × 1 , and 1 × 1 .

Original: This design is inspired by the factorization of convolution (Szegedy et al., 2016) and criss-cross attention (Huang et al., 2019).
Revised: This design is inspired by the factorization of convolution (Szegedy et al., 2016) and criss-cross attention (Huang et al., 2019).

Original: Then, there is a channel-MLP with two linear layers and a GELU (Hendrycks & Gimpel, 2016) non-linearity in between.
Revised: Then, there is a channel-MLP with two linear layers and a GELU (Hendrycks & Gimpel, 2016) non-linearity in between.

Original: A LayerNorm (LN) (Ba et al., 2016) layer is applied before both parallel
Revised: A LayerNorm (LN) (Ba et al., 2016) layer is applied before both parallel

Original: Cycle FC layers and channel-MLP modules.
Revised: Cycle FC layers and channel-MLP modules.

Original: A residual connection (He et al., 2016) is applied after each module.
Revised: A residual connection (He et al., 2016) is applied after each module.

Original: Stage.
Revised: Stage.

Original: The blocks with the same architecture are stacked to form one Stage (He et al., 2016).
Revised: The blocks with the same architecture are stacked to form one Stage (He et al., 2016).

Original: The number of tokens (feature scale) is maintained within each stage.
Revised: The number of tokens (feature scale) is maintained within each stage.

Original: At each stage transition , the channel capacity of the processed tokens is expanded while the number of tokens is reduced.
Revised: At each stage transition , the channel capacity of the processed tokens is expanded while the number of tokens is reduced.

Original: This
Revised: This

Original: [Table]
Revised: [Table]

Original: MLP-like models.
Revised: MLP-like models.

Original: [Table]
Revised: [Table]

Original: strategy effectively reduces the spatial resolution complexity.
Revised: strategy effectively reduces the spatial resolution complexity.

Original: Overall, each of our model variants has four stages, and the output feature at the last stage has a shape of H 32 × W 32 × C 4 .
Revised: Overall, each of our model variants has four stages, and the output feature at the last stage has a shape of H 32 × W 32 × C 4 .

Original: These stage settings are widely utilized in both CNN (Simonyan & Zisserman, 2014;
Revised: These stage settings are widely utilized in both CNN (Simonyan & Zisserman, 2014;

Original: He et al., 2016) and Transformer (Wang et al., 2021b; Liu et al., 2021b) models.
Revised: He et al., 2016) and Transformer (Wang et al., 2021b; Liu et al., 2021b) models.

Original: Therefore, CycleMLP can conveniently serve as a general-purpose visual backbone and a generic replacement for existing backbones.
Revised: Therefore, CycleMLP can conveniently serve as a general-purpose visual backbone and a generic replacement for existing backbones.

Original: Model Variants.
Revised: Model Variants.

Original: The design principle of the model’s macro structure is mainly inspired by the philosophy of hierarchical Transformer (Wang et al., 2021b;
Revised: The design principle of the model’s macro structure is mainly inspired by the philosophy of hierarchical Transformer (Wang et al., 2021b;

Original: Liu et al., 2021b) models, which reduce the number of tokens at the transition layers as the network going deeper and increase the channel dimension.
Revised: Liu et al., 2021b) models, which reduce the number of tokens at the transition layers as the network going deeper and increase the channel dimension.

Original: In this way, we can build a hierarchical architecture that is critical for dense prediction tasks (Lin et al., 2014; Zhou et al., 2017).
Revised: In this way, we can build a hierarchical architecture that is critical for dense prediction tasks (Lin et al., 2014; Zhou et al., 2017).

Original: Speciﬁcally, we build two model zoos following two widely used Transformer architectures, PVT (Wang et al., 2021b) and Swin (Liu et al., 2021b).
Revised: Speciﬁcally, we build two model zoos following two widely used Transformer architectures, PVT (Wang et al., 2021b) and Swin (Liu et al., 2021b).

Original: Models in PVT-style are named from CycleMLP-B1 to CycleMLP-B5 and in Swin-Style are named as CycleMLP-T, -S, and -B, which represent models in tiny, small and base sizes.
Revised: Models in PVT-style are named from CycleMLP-B1 to CycleMLP-B5 and in Swin-Style are named as CycleMLP-T, -S, and -B, which represent models in tiny, small and base sizes.

Original: These models are built by adapting several architecture-related hyper-parameters, including S i , C i , E i and L i which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i .
Revised: These models are built by adapting several architecture-related hyper-parameters, including S i , C i , E i and L i which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i .

Original: Detailed conﬁgurations of these models are in Table 11 (Appendix).
Revised: Detailed conﬁgurations of these models are in Table 11 (Appendix).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst examine CycleMLP by conducting experiments on ImageNet-1K (Deng et al., 2009) image classiﬁcation.
Revised: In this section, we ﬁrst examine CycleMLP by conducting experiments on ImageNet-1K (Deng et al., 2009) image classiﬁcation.

Original: Then, we present a bunch of baseline models achieved by CycleMLP in dense prediction tasks, i.e., COCO (Lin et al., 2014) object detection, instance segmentation, and ADE20K
Revised: Then, we present a bunch of baseline models achieved by CycleMLP in dense prediction tasks, i.e., COCO (Lin et al., 2014) object detection, instance segmentation, and ADE20K

Original: (Zhou et al., 2017) semantic segmentation.
Revised: (Zhou et al., 2017) semantic segmentation.

Original: [Table]
Revised: [Table]

Original: We adopt CycleMLP-B2 variant for this ablation study.
Revised: We adopt CycleMLP-B2 variant for this ablation study.

Original: Double check marks ( (cid:51)(cid:51) ) denote two same branches.
Revised: Double check marks ( (cid:51)(cid:51) ) denote two same branches.

Original: [Table]
Revised: [Table]

Original: when stepsize is 7.
Revised: when stepsize is 7.

Original: However, the stepsize has negligible inﬂuence on the ImageNet classiﬁcation.
Revised: However, the stepsize has negligible inﬂuence on the ImageNet classiﬁcation.

Original: 3. I MAGE N ET -1K C LASSIFICATION
Revised: 3. I MAGE N ET -1K C LASSIFICATION

Original: The experimental settings for ImageNet classiﬁcation are mostly from DeiT (Touvron et al., 2020),
Revised: The experimental settings for ImageNet classiﬁcation are mostly from DeiT (Touvron et al., 2020),

Original: Swin (Liu et al., 2021b).
Revised: Swin (Liu et al., 2021b).

Original: The detailed experimental settings for ImageNet classiﬁcation can be found in Appendix E.1.
Revised: The detailed experimental settings for ImageNet classiﬁcation can be found in Appendix E.1.

Original: Comparison with MLP-like Models.
Revised: Comparison with MLP-like Models.

Original: We ﬁrst compare CycleMLP with existing MLP-like models and the results are summarized in Table 2 and Figure 2.
Revised: We ﬁrst compare CycleMLP with existing MLP-like models and the results are summarized in Table 2 and Figure 2.

Original: The accuracy-FLOPs tradeoff of CycleMLP consistently outperforms existing MLP-like models (Tolstikhin et al., 2021; Touvron et al., 2021a;
Revised: The accuracy-FLOPs tradeoff of CycleMLP consistently outperforms existing MLP-like models (Tolstikhin et al., 2021; Touvron et al., 2021a;

Original: Liu et al., 2021a; Guo et al., 2021; Yu et al., 2021;
Revised: Liu et al., 2021a; Guo et al., 2021; Yu et al., 2021;

Original: Hou et al., 2021) under a wide range of FLOPs, which we attribute to the effectiveness of our Cycle FC.
Revised: Hou et al., 2021) under a wide range of FLOPs, which we attribute to the effectiveness of our Cycle FC.

Original: Speciﬁcally, compared with one of the pioneering MLP work, i.e., gMLP (Liu et al., 2021a), CycleMLP-B2 achieves the same top-1 accuracy (81.6%) as gMLP-B while reducing more than 3 × FLOPs (3.9G for CycleMLP-B2 and 15.8G for gMLP-B).
Revised: Speciﬁcally, compared with one of the pioneering MLP work, i.e., gMLP (Liu et al., 2021a), CycleMLP-B2 achieves the same top-1 accuracy (81.6%) as gMLP-B while reducing more than 3 × FLOPs (3.9G for CycleMLP-B2 and 15.8G for gMLP-B).

Original: Furthermore, compared with existing SOTA MLP-like model, i.e., ViP (Hou et al., 2021), our model CycleMLP-B utilizes less FLOPs (15.2G) than ViP-Large/7 (24.4G, the largest one of ViP family) while achiving higher top-1 accuracy.
Revised: Furthermore, compared with existing SOTA MLP-like model, i.e., ViP (Hou et al., 2021), our model CycleMLP-B utilizes less FLOPs (15.2G) than ViP-Large/7 (24.4G, the largest one of ViP family) while achiving higher top-1 accuracy.

Original: It is noted that all previous MLP-like models listed in Table 2 do not conduct experiments on dense prediction tasks due to the incapability of dealing with variable input scales, which is discussed in
Revised: It is noted that all previous MLP-like models listed in Table 2 do not conduct experiments on dense prediction tasks due to the incapability of dealing with variable input scales, which is discussed in

Original: Sec. 1.
Revised: Sec. 1.

Original: However, CycleMLP solved this issue by adopting Cycle FC.
Revised: However, CycleMLP solved this issue by adopting Cycle FC.

Original: The experimental results on dense prediction tasks are presented in Sec.
Revised: The experimental results on dense prediction tasks are presented in Sec.

Original: 3.3 and Sec. 3.4.
Revised: 3.3 and Sec. 3.4.

Original: Comparison with SOTA Models.
Revised: Comparison with SOTA Models.

Original: Table 3 further compares CycleMLP with previous state-ofthe-art CNN, Transformer and Hybrid architectures.
Revised: Table 3 further compares CycleMLP with previous state-ofthe-art CNN, Transformer and Hybrid architectures.

Original: It is interesting to see that CycleMLP models achieve comparable performance to Swin Transformer (Liu et al., 2021b), which is the state-ofthe-art Transformer-based model.
Revised: It is interesting to see that CycleMLP models achieve comparable performance to Swin Transformer (Liu et al., 2021b), which is the state-ofthe-art Transformer-based model.

Original: Speciﬁcally, CycleMLP-B achieves slightly better top-1 accuracy (83.4%) than Swin-B (83.3%) with similar parameters and FLOPs.
Revised: Speciﬁcally, CycleMLP-B achieves slightly better top-1 accuracy (83.4%) than Swin-B (83.3%) with similar parameters and FLOPs.

Original: GFNet
Revised: GFNet

Original: (Rao et al., 2021) utilizes the fast Fourier transform (FFT) (Cooley & Tukey, 1965) to learn spatial information and achieves similar performance as CycleMLP on ImageNet-1K classiﬁcation.
Revised: (Rao et al., 2021) utilizes the fast Fourier transform (FFT) (Cooley & Tukey, 1965) to learn spatial information and achieves similar performance as CycleMLP on ImageNet-1K classiﬁcation.

Original: However, the architecture of GFNet is correlated with the input resolution, and extra operation (parameter interpolation) is required when input scale changes, which may hurt the performance of dense predictions.
Revised: However, the architecture of GFNet is correlated with the input resolution, and extra operation (parameter interpolation) is required when input scale changes, which may hurt the performance of dense predictions.

Original: We will thoroughly compare CycleMLP with GFNet in Sec.
Revised: We will thoroughly compare CycleMLP with GFNet in Sec.

Original: 3.4 on ADE20K.
Revised: 3.4 on ADE20K.

Original: 3. A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: In this subsection, we conduct extensive ablation studies to analyze each component of our design.
Revised: In this subsection, we conduct extensive ablation studies to analyze each component of our design.

Original: Unless otherwise stated, We adopt CycleMLP-B2 instantiation in this subsection.
Revised: Unless otherwise stated, We adopt CycleMLP-B2 instantiation in this subsection.

Original: Cycle Fully-Connected Layer.
Revised: Cycle Fully-Connected Layer.

Original: To demonstrate the advantage of the Cycle FC, we compare
Revised: To demonstrate the advantage of the Cycle FC, we compare

Original: CycleMLP-B2 with two other baseline models equipped with channel FC and Spatial FC as spatial context aggregation operators, respectively.
Revised: CycleMLP-B2 with two other baseline models equipped with channel FC and Spatial FC as spatial context aggregation operators, respectively.

Original: The differences of these operators are visualized in Figure 1, and the comparison results are shown in Table 1.
Revised: The differences of these operators are visualized in Figure 1, and the comparison results are shown in Table 1.

Original: CycleMLP-B2 outperforms the counterparts built on both Spatial and Channel FC for ImageNet classiﬁcation, COCO object detection, instance segmentation, and ADE20K semantic segmentation.
Revised: CycleMLP-B2 outperforms the counterparts built on both Spatial and Channel FC for ImageNet classiﬁcation, COCO object detection, instance segmentation, and ADE20K semantic segmentation.

Original: The results validate that Cycle FC is capable of serving as a general-purpose, plug-and-play operator for spatial information communication and context aggregation.
Revised: The results validate that Cycle FC is capable of serving as a general-purpose, plug-and-play operator for spatial information communication and context aggregation.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: e N e t
Revised: e N e t

Original: T o
Revised: T o

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: cc
Revised: cc

Original: [Figure]
Revised: [Figure]

Original: e t
Revised: e t

Original: [Figure]
Revised: [Figure]

Original: Table 4 further details the ablation study on the structure of CycleMLP block.
Revised: Table 4 further details the ablation study on the structure of CycleMLP block.

Original: It is observed that the top-1 accuracy drops signiﬁcantly after removing one of the three parallel branches, especially when discarding the 1 × 7 or 7 × 1 branch.
Revised: It is observed that the top-1 accuracy drops signiﬁcantly after removing one of the three parallel branches, especially when discarding the 1 × 7 or 7 × 1 branch.

Original: To eliminate the probability that the fewer parameters and FLOPs cause the performance drop, we further use two same branches (denoted as “ (cid:51)(cid:51) ” in
Revised: To eliminate the probability that the fewer parameters and FLOPs cause the performance drop, we further use two same branches (denoted as “ (cid:51)(cid:51) ” in

Original: Table 4) and one 1 × 1 branch to align the parameters and FLOPs.
Revised: Table 4) and one 1 × 1 branch to align the parameters and FLOPs.

Original: The accuracy still drops relative to CycleMLP, which further demonstrates the necessity of these three unique branches.
Revised: The accuracy still drops relative to CycleMLP, which further demonstrates the necessity of these three unique branches.

Original: Resolution adaptability.
Revised: Resolution adaptability.

Original: One remarkable advantage of CycleMLP is that it can take arbitraryresolution images as input without any modiﬁcation.
Revised: One remarkable advantage of CycleMLP is that it can take arbitraryresolution images as input without any modiﬁcation.

Original: On the contrary, GFNet (Rao et al., 2021) needs to interpolate the learnable parameters on the ﬂy when the input scale is different from the one for training.
Revised: On the contrary, GFNet (Rao et al., 2021) needs to interpolate the learnable parameters on the ﬂy when the input scale is different from the one for training.

Original: We compare the resolution adaptability by directly evaluating models at a broad spectrum of resolutions using the weight pre-trained on 224 × 224, without ﬁne-tuning.
Revised: We compare the resolution adaptability by directly evaluating models at a broad spectrum of resolutions using the weight pre-trained on 224 × 224, without ﬁne-tuning.

Original: Figure 3 (left) shows that the absolute Top-1 accuracy on ImagNet and Figure 3 (right) shows the accuracy differences between one speciﬁc resolution and the resolution of 224 × 224.
Revised: Figure 3 (left) shows that the absolute Top-1 accuracy on ImagNet and Figure 3 (right) shows the accuracy differences between one speciﬁc resolution and the resolution of 224 × 224.

Original: Compared with DeiT and
Revised: Compared with DeiT and

Original: GFNet, CycleMLP is more robust when resolution varies.
Revised: GFNet, CycleMLP is more robust when resolution varies.

Original: In particular, at the 128 × 128, CycleMLP saves more than 2 points drop compared to GFNet.
Revised: In particular, at the 128 × 128, CycleMLP saves more than 2 points drop compared to GFNet.

Original: Furthermore, at higher resolution, the performance drop of CycleMLP is less than GFNet.
Revised: Furthermore, at higher resolution, the performance drop of CycleMLP is less than GFNet.

Original: Note that the superiority of CycleMLP becomes more signiﬁcant when the resolution changes to a greater extent.
Revised: Note that the superiority of CycleMLP becomes more signiﬁcant when the resolution changes to a greater extent.

Original: 3. O BJECT D ETECTION AND I NSTANCE S EGMENTATION
Revised: 3. O BJECT D ETECTION AND I NSTANCE S EGMENTATION

Original: Settings.
Revised: Settings.

Original: We conduct object detection and instance segmentation experiments on COCO (Lin et al., 2014) dataset.
Revised: We conduct object detection and instance segmentation experiments on COCO (Lin et al., 2014) dataset.

Original: We ﬁrst follow the experimental settings of PVT (Wang et al., 2021b), which are introduced in Appendix.
Revised: We ﬁrst follow the experimental settings of PVT (Wang et al., 2021b), which are introduced in Appendix.

Original: E.2.
Revised: E.2.

Original: The corresponding results are presented in Table 6.
Revised: The corresponding results are presented in Table 6.

Original: Then, in order to compare fairly with Swin Transformer
Revised: Then, in order to compare fairly with Swin Transformer

Original: [Table]
Revised: [Table]

Original: which adopts a different experimental recipe with PVT, we further follow the experimental settings of Swin with our CycleMLP-S model and the results are presented in Table 7.
Revised: which adopts a different experimental recipe with PVT, we further follow the experimental settings of Swin with our CycleMLP-S model and the results are presented in Table 7.

Original: [Table]
Revised: [Table]

Original: We compare CycleMLP with various backbones including ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and PVT (Wang et al., 2021b).
Revised: We compare CycleMLP with various backbones including ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and PVT (Wang et al., 2021b).

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: Firstly, as shown in Table
Revised: Firstly, as shown in Table

Original: [Table]
Revised: [Table]

Original: CycleMLP-based RetinaNet consistently surpasses the CNNbased ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and Transformer-based PVT (Wang et al., 2021b) under similar parameter constraints, indicating that CycleMLP can server as an excellent general-purpose backbone.
Revised: CycleMLP-based RetinaNet consistently surpasses the CNNbased ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and Transformer-based PVT (Wang et al., 2021b) under similar parameter constraints, indicating that CycleMLP can server as an excellent general-purpose backbone.

Original: Furthermore, using Mask R-CNN (He et al., 2017) for instance segmentation also demonstrates similar comparison results.
Revised: Furthermore, using Mask R-CNN (He et al., 2017) for instance segmentation also demonstrates similar comparison results.

Original: Furthermore, from Table
Revised: Furthermore, from Table

Original: [Table]
Revised: [Table]

Original: the CycleMLP can achieve a slightly better performance than Swin Transformer.
Revised: the CycleMLP can achieve a slightly better performance than Swin Transformer.

Original: 3. S EMANTIC S EGMENTATION
Revised: 3. S EMANTIC S EGMENTATION

Original: Settings.
Revised: Settings.

Original: We conduct semantic segmentation experiments on ADE20K
Revised: We conduct semantic segmentation experiments on ADE20K

Original: (Zhou et al., 2017) dataset and present the detailed settings in Appendix.
Revised: (Zhou et al., 2017) dataset and present the detailed settings in Appendix.

Original: E.3.
Revised: E.3.

Original: Table 8 and Table 9 show the experimental results using training recipes from PVT and Swin respectively.
Revised: Table 8 and Table 9 show the experimental results using training recipes from PVT and Swin respectively.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: As shown in Table 8, CycleMLP outperforms ResNet (He et al., 2016) and
Revised: As shown in Table 8, CycleMLP outperforms ResNet (He et al., 2016) and

Original: PVT (Wang et al., 2021b) signiﬁcantly with similar parameters.
Revised: PVT (Wang et al., 2021b) signiﬁcantly with similar parameters.

Original: Moreover, compared to the state-of-the-art
Revised: Moreover, compared to the state-of-the-art

Original: Transformer-based backbone, Swin Transformer (Liu et al., 2021b), CycleMLP can obtain comparable or even better performance.
Revised: Transformer-based backbone, Swin Transformer (Liu et al., 2021b), CycleMLP can obtain comparable or even better performance.

Original: Speciﬁcally, CycleMLP-B2 surpasses Swin-T by 0.9 mIoU with slightly less parameters (30.6M v.s. 31.9M).
Revised: Speciﬁcally, CycleMLP-B2 surpasses Swin-T by 0.9 mIoU with slightly less parameters (30.6M v.s. 31.9M).

Original: Although GFNet (Rao et al., 2021) achieves similar performance as CycleMLP on ImageNet classiﬁcation, CycleMLP notably outperforms GFNet on ADE20K semantic segmentation where input scale varies.
Revised: Although GFNet (Rao et al., 2021) achieves similar performance as CycleMLP on ImageNet classiﬁcation, CycleMLP notably outperforms GFNet on ADE20K semantic segmentation where input scale varies.

Original: We attribute the superiority of CycleMLP under a scale-variable scenario to the capability of dealing with arbitrary scales.
Revised: We attribute the superiority of CycleMLP under a scale-variable scenario to the capability of dealing with arbitrary scales.

Original: On the contrary, GFNet (Rao et al., 2021) requires additional
Revised: On the contrary, GFNet (Rao et al., 2021) requires additional

Original: [Table]
Revised: [Table]

Original: heuristic operation (weight interpolation) when the input scale varies, which may hurt the performance.
Revised: heuristic operation (weight interpolation) when the input scale varies, which may hurt the performance.

Original: Moreover
Revised: Moreover

Original: [Table]
Revised: [Table]

Original: we also visualized the receptive ﬁeld following (Xie et al., 2021)
Revised: we also visualized the receptive ﬁeld following (Xie et al., 2021)

Original: [Table]
Revised: [Table]

Original: and the results are visualized in Figure 4, which demonstrate that our CycleMLP has a larger effective receptive ﬁeld than Swin.
Revised: and the results are visualized in Figure 4, which demonstrate that our CycleMLP has a larger effective receptive ﬁeld than Swin.

Original: [Table]
Revised: [Table]

Original: We further conduct experiments on ImageNet-C (Hendrycks & Gimpel, 2016) to analyze the robustness ability of the CycleMLP, following (Mao et al., 2021) and results are presented in Table 10.
Revised: We further conduct experiments on ImageNet-C (Hendrycks & Gimpel, 2016) to analyze the robustness ability of the CycleMLP, following (Mao et al., 2021) and results are presented in Table 10.

Original: Compared with both Transformers ( e
Revised: Compared with both Transformers ( e

Original: [Table]
Revised: [Table]

Original: g. DeiT and Swin) and existing MLP models ( e.g. MLP-Mixer,
Revised: g. DeiT and Swin) and existing MLP models ( e.g. MLP-Mixer,

Original: ResMLP, gMLP), CycleMLP achieves a stronger robustness ability.
Revised: ResMLP, gMLP), CycleMLP achieves a stronger robustness ability.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a versatile MLP-like architecture, CycleMLP, in this work.
Revised: We present a versatile MLP-like architecture, CycleMLP, in this work.

Original: CycleMLP is built upon the
Revised: CycleMLP is built upon the

Original: Cycle Fully-Connected Layer (Cycle FC)
Revised: Cycle Fully-Connected Layer (Cycle FC)

Original: , which is capable of dealing with variable input scales and can serve as a generic, plug-and-play replacement of vanilla FC layers.
Revised: , which is capable of dealing with variable input scales and can serve as a generic, plug-and-play replacement of vanilla FC layers.

Original: Experimental results demonstrate that CycleMLP outperforms existing MLP-like models on ImageNet classiﬁcation and achieves promising performance on dense prediction tasks, i e., object detection, instance segmentation and semantic segmentation.
Revised: Experimental results demonstrate that CycleMLP outperforms existing MLP-like models on ImageNet classiﬁcation and achieves promising performance on dense prediction tasks, i e., object detection, instance segmentation and semantic segmentation.

Original: This work indicates that an attention-free architecture can also serve as a general vision backbone
Revised: This work indicates that an attention-free architecture can also serve as a general vision backbone

Original: A L ITERATURE ON V ISION M ODEL
Revised: A L ITERATURE ON V ISION M ODEL

Original: CNN-based Models.
Revised: CNN-based Models.

Original: Originally introduced over twenty years ago (LeCun et al., 1989), convolutional neural networks (CNN) have been widely adopted since the success of the
Revised: Originally introduced over twenty years ago (LeCun et al., 1989), convolutional neural networks (CNN) have been widely adopted since the success of the

Original: AlexNet (Krizhevsky et al., 2012) which outperformed prevailing approaches based on hand-crafted image features.
Revised: AlexNet (Krizhevsky et al., 2012) which outperformed prevailing approaches based on hand-crafted image features.

Original: There have been several attempts made to improve the design of CNN-based models.
Revised: There have been several attempts made to improve the design of CNN-based models.

Original: VGG (Simonyan & Zisserman, 2014) demonstrated a state-of-the-art performance on ImageNet via deploying small ( 3 × 3 ) convolution kernels to all layers.
Revised: VGG (Simonyan & Zisserman, 2014) demonstrated a state-of-the-art performance on ImageNet via deploying small ( 3 × 3 ) convolution kernels to all layers.

Original: He
Revised: He

Original: et al .introduced skip-connections in
Revised: et al .introduced skip-connections in

Original: ResNets (He et al., 2016), enabling a model variant with more than 1000 layers.
Revised: ResNets (He et al., 2016), enabling a model variant with more than 1000 layers.

Original: DenseNet (Huang et al., 2017) connected each layer to every other layer in a feed-forward fashion, strengthening feature propagation and reducing the number of parameters.
Revised: DenseNet (Huang et al., 2017) connected each layer to every other layer in a feed-forward fashion, strengthening feature propagation and reducing the number of parameters.

Original: In parallel with these architecture design works, some other works also made signiﬁcant contributions to the popularity of CNNs, including normalization (Ioffe & Szegedy, 2015; Ba et al., 2016), data augmentation (Cubuk et al., 2020; Yun et al., 2019; Zhang et al., 2017), etc.
Revised: In parallel with these architecture design works, some other works also made signiﬁcant contributions to the popularity of CNNs, including normalization (Ioffe & Szegedy, 2015; Ba et al., 2016), data augmentation (Cubuk et al., 2020; Yun et al., 2019; Zhang et al., 2017), etc.

Original: Transformer-based Models.
Revised: Transformer-based Models.

Original: Transformers were ﬁrst proposed by Vaswani et al .for machine translation and have since become the dominant choice in many NLP tasks (Devlin et al., 2018;
Revised: Transformers were ﬁrst proposed by Vaswani et al .for machine translation and have since become the dominant choice in many NLP tasks (Devlin et al., 2018;

Original: Wang et al., 2018; Yang et al., 2019; Brown et al., 2020).
Revised: Wang et al., 2018; Yang et al., 2019; Brown et al., 2020).

Original: Recently, transformer have also led to a series of breakthroughs in computer vision community since the invention of ViT (Dosovitskiy et al., 2020), and have been working as a de facto standard for various tasks, e.g., image classiﬁcation (Dosovitskiy et al., 2020; Touvron et al., 2020;
Revised: Recently, transformer have also led to a series of breakthroughs in computer vision community since the invention of ViT (Dosovitskiy et al., 2020), and have been working as a de facto standard for various tasks, e.g., image classiﬁcation (Dosovitskiy et al., 2020; Touvron et al., 2020;

Original: Yuan et al., 2021), detection and segmentation (Wang et al., 2021b;
Revised: Yuan et al., 2021), detection and segmentation (Wang et al., 2021b;

Original: Liu et al., 2021b; Zheng et al., 2021; Xie et al., 2021), video recognition (Wang et al., 2021c;
Revised: Liu et al., 2021b; Zheng et al., 2021; Xie et al., 2021), video recognition (Wang et al., 2021c;

Original: Bertasius et al., 2021; Arnab et al., 2021; Fan et al., 2021) and so on.
Revised: Bertasius et al., 2021; Arnab et al., 2021; Fan et al., 2021) and so on.

Original: Moreover, there has also been lots of interest in adopting transformer to cross aggregate multiple modality information (Radford et al., 2021; Gabeur et al., 2020; Dzabraev et al., 2021).
Revised: Moreover, there has also been lots of interest in adopting transformer to cross aggregate multiple modality information (Radford et al., 2021; Gabeur et al., 2020; Dzabraev et al., 2021).

Original: Furthermore, combining CNNs and transformers is also explored in (Srinivas et al., 2021; Li et al., 2021; Wu et al., 2021; Touvron et al., 2021b).
Revised: Furthermore, combining CNNs and transformers is also explored in (Srinivas et al., 2021; Li et al., 2021; Wu et al., 2021; Touvron et al., 2021b).

Original: MLP-based Models.
Revised: MLP-based Models.

Original: MLP-based models (Tolstikhin et al., 2021; Touvron et al., 2021a;
Revised: MLP-based models (Tolstikhin et al., 2021; Touvron et al., 2021a;

Original: Liu et al., 2021a) differ from the above discussed CNN- and Transformer-based models because they resort to neither convolution nor self-attention layers.
Revised: Liu et al., 2021a) differ from the above discussed CNN- and Transformer-based models because they resort to neither convolution nor self-attention layers.

Original: Instead, they use MLP layers over feature patches on spatial dimensions to aggregate the spatial context.
Revised: Instead, they use MLP layers over feature patches on spatial dimensions to aggregate the spatial context.

Original: These MLP-based models share similar macro structures but differ from each other in the detailed design of the micro block.
Revised: These MLP-based models share similar macro structures but differ from each other in the detailed design of the micro block.

Original: In addition,
Revised: In addition,

Original: MLP-based models provide more efﬁcient computation than transformer-based models since they do not need to calculate afﬁnity matrix using key-query multiplication.
Revised: MLP-based models provide more efﬁcient computation than transformer-based models since they do not need to calculate afﬁnity matrix using key-query multiplication.

Original: Concurrent to our work,
Revised: Concurrent to our work,

Original: S 2 -MLP (Yu et al., 2021) utilizes a spatial-shift operation for spatial information communication.
Revised: S 2 -MLP (Yu et al., 2021) utilizes a spatial-shift operation for spatial information communication.

Original: The similar aspect between our work and S 2 -MLP lies in that we all conduct MLP operations along the channel dimension.
Revised: The similar aspect between our work and S 2 -MLP lies in that we all conduct MLP operations along the channel dimension.

Original: However, our Cycle FC is different from S 2 -MLP in: (1) S 2 -MLP achieves communications between patches by splitting feature maps along channel dimension into several groups and shifting different groups in different directions.
Revised: However, our Cycle FC is different from S 2 -MLP in: (1) S 2 -MLP achieves communications between patches by splitting feature maps along channel dimension into several groups and shifting different groups in different directions.

Original: It introduces extra splitting and shifting operations on the feature map.
Revised: It introduces extra splitting and shifting operations on the feature map.

Original: On the contrary, we propose a novel operator-Cycle Fully-Connected
Revised: On the contrary, we propose a novel operator-Cycle Fully-Connected

Original: Layer-for spatial context aggregation.
Revised: Layer-for spatial context aggregation.

Original: It does not modify the feature map and is formulated as a generic, plug-and-play MLP unit that can be used as a direct replacement of vanilla without any adjustments.
Revised: It does not modify the feature map and is formulated as a generic, plug-and-play MLP unit that can be used as a direct replacement of vanilla without any adjustments.

Original: (2) We design a pyramid structure for and conduct extensive experiments on classiﬁcation, object detection, instance segmentation, and semantic segmentation.
Revised: (2) We design a pyramid structure for and conduct extensive experiments on classiﬁcation, object detection, instance segmentation, and semantic segmentation.

Original: However, the output feature map of S 2 -MLP has only one single scale in low resolution, which is unsuitable for dense prediction tasks.
Revised: However, the output feature map of S 2 -MLP has only one single scale in low resolution, which is unsuitable for dense prediction tasks.

Original: Only ImageNet classiﬁcation is evaluated on S 2 -MLP.
Revised: Only ImageNet classiﬁcation is evaluated on S 2 -MLP.

Original: We compared Cycle FC with S 2 -MLP in details in the Section 3.
Revised: We compared Cycle FC with S 2 -MLP in details in the Section 3.

Original: B C OMPARISON OF MLP B LOCKS
Revised: B C OMPARISON OF MLP B LOCKS

Original: We summary MLP blocks proposed by recent MLP-related works in Figure 5.
Revised: We summary MLP blocks proposed by recent MLP-related works in Figure 5.

Original: We notice that existing MLP blocks, i.e., MLP-Mixer, ResMLP and gMLP share similar method of Spatial Proj :
Revised: We notice that existing MLP blocks, i.e., MLP-Mixer, ResMLP and gMLP share similar method of Spatial Proj :

Original: Transpose → Fully-Connected over spatial dimension → Transpose back.
Revised: Transpose → Fully-Connected over spatial dimension → Transpose back.

Original: These models can not cope with variable image scales as the FC layers in Spatial Proj are conﬁgured by the seq len .
Revised: These models can not cope with variable image scales as the FC layers in Spatial Proj are conﬁgured by the seq len .

Original: The blocks used for building CycleMLP consist of our proposed novel Cycle FC, whose conﬁguration has nothing to do with image scales and can naturally deal with dynamic image scales. 
Revised: The blocks used for building CycleMLP consist of our proposed novel Cycle FC, whose conﬁguration has nothing to do with image scales and can naturally deal with dynamic image scales. [Figure] [Figure] Transpose(

Original: [Figure]
Revised: [Figure]

Revised: FC(

Revised: [Figure]

Revised: [Figure]

Revised: dim)

Revised: FC(dim

Revised: [Figure]

Original: seq
Revised: *dim) [Figure] FC( seq_len, 0.5*seq_len ) GeLU

Revised: FC( 0.

Original: [Figure]
Revised: [Figure]

Revised: *seq_len

Original: [Figure] 
Revised: [Figure] seq_ [Figure] [Figure]

Revised: a

Original: [Figure]
Revised: [Figure]

Original:  (Tolstikhin et al., 2021) 
Revised: MLP-Mixer (Tolstikhin et al., 2021) [Figure] Affine (Bias) Spatial

Original: [Figure]
Revised: [Figure]

Revised: Affine

Revised: Affine (Bias)

Revised: Channel Proj

Revised: Affine

Revised: FC(seq_len, seq_len)

Revised: Transpose(

Revised: [Figure]

Revised: Transpose(1, 2)

Revised: FC(4*dim, dim)

Revised: FC(dim, 4*dim)

Revised: GeLU (

Revised: [Figure]

Original:  ResMLP (Touvron et al., 2021a)
Revised: ) ResMLP (Touvron et al., 2021a)

Original: [Figure]
Revised: [Figure]

Original: ×
Revised: [Figure] seq_ [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 𝑆 𝐹 𝐶 :×
Revised: (Liu et al., 2021a) [Figure] [Figure]

Original: S 𝐹 𝐶 :×
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: C F ROM MULTI - HEAD SELF - ATTENTION TO CONVOLUTION
Revised: C F ROM MULTI - HEAD SELF - ATTENTION TO CONVOLUTION

Original: In this section, we provide details in how MHSA can be transferred into a convolution-like operator in equation 3.
Revised: In this section, we provide details in how MHSA can be transferred into a convolution-like operator in equation 3.

Original: To start with, the a MHSA layer can be formulated as below:
Revised: To start with, the a MHSA layer can be formulated as below:

Original: [Equation]
Revised: [Equation]

Original:  where W out ∈ R ( N h C out ) × C (cid:48) out and b ∈ R C (cid:48) out are parameters for the ﬁnal linear projection.
Revised: (5) where W out ∈ R ( N h C out ) × C (cid:48) out and b ∈ R C (cid:48) out are parameters for the ﬁnal linear projection.

Original: SA h is the h th self-attention module.
Revised: SA h is the h th self-attention module.

Original: Then we reshape X into X ∈ R HW × C in and let
Revised: Then we reshape X into X ∈ R HW × C in and let

Original: T = H × W , which indicates that there are T tokens in X .
Revised: T = H × W , which indicates that there are T tokens in X .

Original: SA h can be deﬁned as follow:
Revised: SA h can be deﬁned as follow:

Original: [Equation]
Revised: [Equation]

Original: where V = XW val , Q = XW qry , K = XW key are respectively the value, query and key matrix with learnable matrices W v ∈ R C in × C out , W q ∈ R C in × C k , W k ∈ R C in × C k .
Revised: where V = XW val , Q = XW qry , K = XW key are respectively the value, query and key matrix with learnable matrices W v ∈ R C in × C out , W q ∈ R C in × C k , W k ∈ R C in × C k .

Original: P ∈ R T × C in is the positional embedding matrix containing positional information for every input token, which can be replaced by the output of any function f P that encodes the position of tokens.
Revised: P ∈ R T × C in is the positional embedding matrix containing positional information for every input token, which can be replaced by the output of any function f P that encodes the position of tokens.

Original: And A ∈ R T × T is the attention matrix where each element A i,j is the attention score between the i th and j th token in X .
Revised: And A ∈ R T × T is the attention matrix where each element A i,j is the attention score between the i th and j th token in X .

Original: With absolute positional encoding, the second line in equation 6 can be expanded as (Cordonnier et al., 2020):
Revised: With absolute positional encoding, the second line in equation 6 can be expanded as (Cordonnier et al., 2020):

Original: [Equation]
Revised: [Equation]

Original: When we apply relative positional encoding scheme in (Dai et al., 2019), A is re-parametried into:
Revised: When we apply relative positional encoding scheme in (Dai et al., 2019), A is re-parametried into:

Original: [Equation]
Revised: [Equation]

Original: ) where r δ q,k is a positional encoding for relative distance δ q,k = ( δ 1 , δ 2 ) between token q and k in X .
Revised: ) where r δ q,k is a positional encoding for relative distance δ q,k = ( δ 1 , δ 2 ) between token q and k in X .

Original: ˆ W key is introduced to only pertain to the positional encoding r q,k .
Revised: ˆ W key is introduced to only pertain to the positional encoding r q,k .

Original: u and v are learnable parameter vectors that replace the original P q, : W qry term, which implies that the attention bias remains the same regardless of the absolution positions of the query.
Revised: u and v are learnable parameter vectors that replace the original P q, : W qry term, which implies that the attention bias remains the same regardless of the absolution positions of the query.

Original: If we set W qry =
Revised: If we set W qry =

Original: W key = 

Revised: W key =

Original: and ˆ W key
Revised: and ˆ W key

Original: =
Revised: =

Original: I , the ﬁrst three terms in equation 8 vanish and A q,k =
Revised: I , the ﬁrst three terms in equation 8 vanish and A q,k =

Original: vr (cid:124) q,k .
Revised: vr (cid:124) q,k .

Original: We set { ∆ i ( h ) , ∆ j ( h ) } = { (0 , 0) , (1 , 0) , ( − 1 , 0) , · · · } contains all possible positional shift in convolution with kernel size √
Revised: We set { ∆ i ( h ) , ∆ j ( h ) } = { (0 , 0) , (1 , 0) , ( − 1 , 0) , · · · } contains all possible positional shift in convolution with kernel size √

Original: N h ×√ N h .
Revised: N h ×√ N h .

Original: For each head h , let r q,k
Revised: For each head h , let r q,k

Original: =
Revised: =

Original: ( (cid:107) δ q,k (cid:107) 2 , δ 1 , δ 2 ) and v h = − α h
Revised: ( (cid:107) δ q,k (cid:107) 2 , δ 1 , δ 2 ) and v h = − α h

Original: (1 , − 2∆ i ( h ) , − 2∆ j ( h )) , each softmax attention matrix becomes:
Revised: (1 , − 2∆ i ( h ) , − 2∆ j ( h )) , each softmax attention matrix becomes:

Original: [Equation]
Revised: [Equation]

Original: Substitute softmax( A h ) into equation 

Revised: Substitute softmax( A h ) into equation

Original: and we get
Revised: and we get

Original: [Equation]
Revised: [Equation]

Original: D A RCHITECTURE V ARIANTS
Revised: D A RCHITECTURE V ARIANTS

Original: In order to conduct fair and convenient comparison, we build two model zoos: the one is in PVTStyle (named as CycleMLP-B1 to -B5) and the other in Swin-Style (named as CycleMLP-T, -S andB).
Revised: In order to conduct fair and convenient comparison, we build two model zoos: the one is in PVTStyle (named as CycleMLP-B1 to -B5) and the other in Swin-Style (named as CycleMLP-T, -S andB).

Original: These models are scaled up by adapting several architecture-related hyper-parameters, including
Revised: These models are scaled up by adapting several architecture-related hyper-parameters, including

Original: S i , C i , E i and L i which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i .
Revised: S i , C i , E i and L i which represent the stride of the transition, the token channel dimension, the number of blocks and the expansion ratio respectively at Stage i .

Original: Detailed conﬁgurations of these models are in Table 11.
Revised: Detailed conﬁgurations of these models are in Table 11.

Original: E E XPERIMENTAL SETUPS
Revised: E E XPERIMENTAL SETUPS

Original: E.
Revised: E.

Original: I MAGE N ET C LASSIFICATION
Revised: I MAGE N ET C LASSIFICATION

Original: Settings.
Revised: Settings.

Original: We train our models on the ImageNet-1K dataset (Deng et al., 2009), which contains 1.2M training images and 50K validation images evenly spreading 1,000 categories.
Revised: We train our models on the ImageNet-1K dataset (Deng et al., 2009), which contains 1.2M training images and 50K validation images evenly spreading 1,000 categories.

Original: We follow the standard practice in the community by reporting the top-1 accuracy on the validation set.
Revised: We follow the standard practice in the community by reporting the top-1 accuracy on the validation set.

Original: Our code is implemented based on PyTorch (Paszke et al., 2019) framework and heavily relies on the timm (Wightman, 2019) repository.
Revised: Our code is implemented based on PyTorch (Paszke et al., 2019) framework and heavily relies on the timm (Wightman, 2019) repository.

Original: For apple-to-apple comparison, our training strategy is mostly adopted from DeiT (Touvron et al., 2020), which includes RandAugment (Cubuk et al., 2020),
Revised: For apple-to-apple comparison, our training strategy is mostly adopted from DeiT (Touvron et al., 2020), which includes RandAugment (Cubuk et al., 2020),

Original: Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019)
Revised: Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019)

Original: random erasing (Zhong et al., 2020) and stochastic depth (Huang et al., 2016).
Revised: random erasing (Zhong et al., 2020) and stochastic depth (Huang et al., 2016).

Original: The optimizer is AdamW (Loshchilov & Hutter, 2017) with the momentum of 0.9 and weight decay of 5 × 10 − 2 by default.
Revised: The optimizer is AdamW (Loshchilov & Hutter, 2017) with the momentum of 0.9 and weight decay of 5 × 10 − 2 by default.

Original: The cosine learning rate schedule is adopted with the initial value of 1 × 10 − 3 .
Revised: The cosine learning rate schedule is adopted with the initial value of 1 × 10 − 3 .

Original: All models are trained for 300 epochs on 8 Tesla V
Revised: All models are trained for 300 epochs on 8 Tesla V

Original: GPUs with a total batch size of 1024.
Revised: GPUs with a total batch size of 1024.

Original: [Table]
Revised: [Table]

Original: Further kernel optimization for Cycle FC may bring a faster speed but is beyond the scope of this work.
Revised: Further kernel optimization for Cycle FC may bring a faster speed but is beyond the scope of this work.

Original: E. COCO I NSTANCE S EGMENTATION
Revised: E. COCO I NSTANCE S EGMENTATION

Original: We conduct object detection and instance segmentation experiments on COCO (Lin et al., 2014) dataset, which contains 118K and 5K images for train and validation splits.
Revised: We conduct object detection and instance segmentation experiments on COCO (Lin et al., 2014) dataset, which contains 118K and 5K images for train and validation splits.

Original: We adopt the mmdetection (Chen et al., 2019) toolbox for all experiments in this subsection.
Revised: We adopt the mmdetection (Chen et al., 2019) toolbox for all experiments in this subsection.

Original: To evaluate the our CycleMLP backbones, we adopt two widely used detectors, i.e., RetinaNet (Lin et al., 2017) and Mask R-CNN
Revised: To evaluate the our CycleMLP backbones, we adopt two widely used detectors, i.e., RetinaNet (Lin et al., 2017) and Mask R-CNN

Original: (He et al., 2017).
Revised: (He et al., 2017).

Original: All backbones are initialized with ImageNet pre-trained weights and other newly added layers are initialized via Xavier (Glorot & Bengio, 2010).
Revised: All backbones are initialized with ImageNet pre-trained weights and other newly added layers are initialized via Xavier (Glorot & Bengio, 2010).

Original: We use the AdamW (Loshchilov & Hutter, 2017) optimizer with the initial learning rate of 1 × 10 − 4 .
Revised: We use the AdamW (Loshchilov & Hutter, 2017) optimizer with the initial learning rate of 1 × 10 − 4 .

Original: All models are trained on 8 Tesla V100 GPUs with a total batch size of 16 for 12 epochs ( i.e., 1 × training scheduler).
Revised: All models are trained on 8 Tesla V100 GPUs with a total batch size of 16 for 12 epochs ( i.e., 1 × training scheduler).

Original: The input images are resized to the shorted side of 800 pixels and the longer side does not exceed 1333 pixels during training.
Revised: The input images are resized to the shorted side of 800 pixels and the longer side does not exceed 1333 pixels during training.

Original: We do not use the multi-scale (Carion et al., 2020; Zhu et al., 2020; Sun et al., 2021) training strategy.
Revised: We do not use the multi-scale (Carion et al., 2020; Zhu et al., 2020; Sun et al., 2021) training strategy.

Original: In the testing stage, the shorter side of input images is resized to 800 pixels while no constraint on the longer side.
Revised: In the testing stage, the shorter side of input images is resized to 800 pixels while no constraint on the longer side.

Original: E. ADE20K S EMANTIC S EGMENTATION
Revised: E. ADE20K S EMANTIC S EGMENTATION

Original: We conduct semantic segmentation experiments on ADE20K
Revised: We conduct semantic segmentation experiments on ADE20K

Original: (Zhou et al., 2017) dataset, which covers a broad range of 150 semantic categories.
Revised: (Zhou et al., 2017) dataset, which covers a broad range of 150 semantic categories.

Original: ADE20K contains 20K training, 2K validation and 3K testing images.
Revised: ADE20K contains 20K training, 2K validation and 3K testing images.

Original: We adopt the mmsegmenation (Contributors, 2020) toolbox as our codebase in this subsection.
Revised: We adopt the mmsegmenation (Contributors, 2020) toolbox as our codebase in this subsection.

Original: The experimental settings mostly follow PVT (Wang et al., 2021b), which trains models for 40K iterations on 8 Tesla V100 GPUs with 4 samples per GPU.
Revised: The experimental settings mostly follow PVT (Wang et al., 2021b), which trains models for 40K iterations on 8 Tesla V100 GPUs with 4 samples per GPU.

Original: The backbone is initialized with the pre-trained weights on ImageNet.
Revised: The backbone is initialized with the pre-trained weights on ImageNet.

Original: All models are optimized by AdamW (Loshchilov &
Revised: All models are optimized by AdamW (Loshchilov &

Original: Hutter, 2017).
Revised: Hutter, 2017).

Original: The initial learning rate is conﬁgured as 2 × 10 − 4 with the polynomial decay parameter of 0.9.
Revised: The initial learning rate is conﬁgured as 2 × 10 − 4 with the polynomial decay parameter of 0.9.

Original: Input images are randomly resized and cropped to 512 × 512 at the training phase.
Revised: Input images are randomly resized and cropped to 512 × 512 at the training phase.

Original: During testing, we scale the images to the shorted side of 512.
Revised: During testing, we scale the images to the shorted side of 512.

Original: We adopt the simple approach Semantic
Revised: We adopt the simple approach Semantic

Original: FPN (Kirillov et al., 2019) as the semantic segmentation method following (Wang et al., 2021b) for fair comparison.
Revised: FPN (Kirillov et al., 2019) as the semantic segmentation method following (Wang et al., 2021b) for fair comparison.

Original: [Table]
Revised: [Table]

Revised: F S AMPLING S TRATEGIES

Revised: We explore more sampling strategies in this subsection, including random sampling and dilated sampling inspired by dilated convolution (Yu & Koltun, 2016;

Revised: Chen et al., 2018) (as shown in Figure 6).

Revised: We also compare the dense sampling method with ours.

Revised: Random sampling.

Revised: As shown in Table 13, we conduct experiments with random sampling for three independent trials and observe that the averaged Top-1 accuracy on ImageNet-1K drops by 1 . 3% .

Revised: We hypothesize that the decreased performance is caused by the fact that random sampling will totally disturb the semantic information of objects, which is essential to image recognition.

Revised: Compared with the random sampling strategy, our cyclical sampling is able to aggregate the adjacent pixels, which beneﬁts in capturing the semantic information.

Revised: Dilated Stepsize (Figure 6).

Revised: As shown in Table 13, we observe the result of dilated sampling is better than the random one ( +1 . 0% acc) but lower than ours ( − 0 .

Revised: 5% acc).

Revised: In fact, compared with the random sampling, dilated solutions take their advantages in local information aggregation.

Revised: However, compared with the cyclical sampling strategy, dilated solutions lose the ﬁne-grained information for recognition.

Revised: It may hurt the accuracy performance to some extent.

Revised: Dense sampling.

Revised: we conduct ablation studies by using dense sampling strategies (i.e., vanilla convolution with kernel size 1 × 3 and 3 × 1).

Revised: Since dense sampling strategies incredibly increase the models’ parameters and FLOPs, we do not have enough time to thoroughly optimize the model for 300 epochs.

Revised: Therefore, for fair comparisons, we conducted extra ablation studies on training models for 100 epochs with the strictly same learning conﬁgurations.

Revised: The results shown in Tabledemonstrate that the sparse sampling strategy (ours) outperforms the dense one.

Revised: The comparison indicates that the dense sampling strategies introduce redundant parameters, which makes the model hard to optimize.

Revised: Our sparse sampling strategy with fewer parameters is proven to be efﬁcient and optimization friendly.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: G V ISUALIZATION E XAMPLES

Revised: For easier understanding of our proposed CycleMLP, we visualize several instances of CycleMLP in Figure 7, including general case with stepsize 3 × 3 (7a), even stepsize (7b), and examples where stepsize along height or width equals to 1 (7c, 7d).

Revised: We note that given speciﬁc number of input and output channels, no matter how the stepsize changes, the number of parameters of the CycleMLP does not change.

Revised: Therefore, there is a trade-off of representation abilities between spatial and channel dimensions, which will be discussed in details in following experimental analysis.

Revised: Experiments: We further conduct experiments on CycleMLPs with stepsize of 2 × 7, 7 × 2, 7 × 3, 3 × 7, and 4 × 4, respectively.

Revised: The results are summarized Table 15.

Revised: For fair comparisons, all the models in the above table have the same parameters and FLOPs.

Revised: We observe that the model with stepsize of 1 × 7 and 7 × 1 achieves the best performance, especially for semantic segmentation on

Revised: ADE20K.

Revised: To analyze the impact of stepsize on the performance, we take Figure 7 for better illustration.

Revised: One can see that enlarging the stepsize can expand the spatial receptive ﬁeld.

Revised: However, at a cost, it will reduce the number of periods (groups) running along the channel dimension, which may hurt the channel-wise representation abilities.

Revised: Taking a feature map with C = 18 for example, the CycleMLP with stepsize 3 × 3 (Figure 7(a)) runs through only 2 channel groups (curly brackets in the ﬁgure).

Revised: However, the CycleMLP with stepsize 3 × 1 (Figure 7(c)) will run through 6 groups in total, making better use of the representation in the channel dimension.

Revised: That’s to say, there is a trade-off between spatial and channel representation.

Revised: We empirically found that CyCleMLP with stepsize of 1 × 7 and 7 × 1 achieves the best performance.

Original: [Table]
Revised: [Table]

Revised: [1, 0, 16] [2, 0, 17] [H, W, C] (

Original: [Figure]
Revised: [Figure]

Revised: S W : 3 ×𝐶 𝑊 𝐻 𝐻 $ 𝐻 % 𝐻 & 𝐻 ’ 𝑊

Revised: $ 𝑊 % 𝑊 & [0, 0, 0] [0, 1, 1] [0, 2, 2] [0, 0, 3] [0, 1, 4] [0, 2, 5] [0, 0, 6] [0, 1, 7] [0, 2, 8] [0, 0, 9] [0, 1, 10] [0, 2, 11] [0, 0, 12] [0, 1, 13] [0, 2, 14] [0, 0, 15] [0, 1, 16] [0, 2, 17]

Revised: [H, W, C]

Revised: [Figure]