Paper: MogaNet: Multi-order Gated Aggregation Network (ICLR-2024)

Reviewer comment (Writing):
The core idea of this work is hard to follow. Why the proposed network design is "an appropriate composition of convolutional locality priors and global context aggregations" is not entirely evident.

Author response:
As stated in the first point above, our work is inspired by an ICLR 2022 work [1], which is the theoretical foundation of this work. Due to page limitations, we cannot put the entire theoretical derivation of multi-order interaction in the paper. The issues and theoretical foundations referred to in this work are not unstandardized concepts but rather issues derived through solid theoretical studies. We agree that we did not fully explore this aspect in our original submission. Thus, we have endeavored to condense and integrate the necessary understanding of it in the revision. Please refer to our general response for the clear illustration and concepts of MogaNet. For definitions and derivations regarding the interaction, please refer to [1].

Reviewer comment (Evaluation):
In addition to the quantitative depiction in Fig. 3, qualitative analysis of the reasons why earlier network designs failed at this point would be extremely helpful in explaining this central argument.

Author response:
To let you better grasp the theoretical starting point of our paper, we put aside the lengthy theoretical derivation, and based on the existing related research, I would like to make some top-down elaboration on the understanding of the interaction in MogaNet. The study in [1] empirically found that current deep neural networks commonly suffer from a middle-order deficient representation bottleneck. 

To the best of our knowledge, we suggest that the interaction in practice is mainly influenced by three factors: **the data distribution, the loss function, and the network architecture design.** [2] found that in computer vision tasks, the ideal interaction curve tends to be the strongest in the low-order, and the middle-order is comparable to the high-order one. This is related to the data distribution property of computer vision tasks. [1] not only provided the theoretical foundation but also presented a loss function to solve this issue. [3] affected the interaction curve by dealing with the data distribution. 

As for ViT,  we believe that the main reason for the lack of middle-order, despite the use of adaptive data-dependent feature extraction, is the absence of a targeted module design. Global self-attention fails to promote middle-order interaction, which is easy to recognize by its definition, properties, and the findings of prior works.

There is no research on affecting the interactions by network architecture design. Therefore, combining the conclusions of [1] and our empirical analysis in this work, we argue that there is an absence of middle-order interaction in the current deep models. MogaNet is the first attempt to address this issue from the novel perspective of network architecture design.

Reviewer comment (Theory):
The perspective on multi-order interactions that serves as the foundation and impetus for this work is presented in [r1]. Analysis of the factors why existing networks result in an inappropriate composition of local and global context is still lacking.

Author response:
To let you better grasp the theoretical starting point of our paper, we put aside the lengthy theoretical derivation, and based on the existing related research, I would like to make some top-down elaboration on the understanding of the interaction in MogaNet. The study in [1] empirically found that current deep neural networks commonly suffer from a middle-order deficient representation bottleneck. 

To the best of our knowledge, we suggest that the interaction in practice is mainly influenced by three factors: **the data distribution, the loss function, and the network architecture design.** [2] found that in computer vision tasks, the ideal interaction curve tends to be the strongest in the low-order, and the middle-order is comparable to the high-order one. This is related to the data distribution property of computer vision tasks. [1] not only provided the theoretical foundation but also presented a loss function to solve this issue. [3] affected the interaction curve by dealing with the data distribution. 

As for ViT,  we believe that the main reason for the lack of middle-order, despite the use of adaptive data-dependent feature extraction, is the absence of a targeted module design. Global self-attention fails to promote middle-order interaction, which is easy to recognize by its definition, properties, and the findings of prior works.

There is no research on affecting the interactions by network architecture design. Therefore, combining the conclusions of [1] and our empirical analysis in this work, we argue that there is an absence of middle-order interaction in the current deep models. MogaNet is the first attempt to address this issue from the novel perspective of network architecture design.

Reviewer comment (Theory):
It is thought that the self-attention mechanism is more adaptable than typical gate attention. Why does ViT fail to learn middle-order interactions when self-attention is present?

Author response:
As for ViT,  we believe that the main reason for the lack of middle-order, despite the use of adaptive data-dependent feature extraction, is the absence of a targeted module design. Global self-attention fails to promote middle-order interaction, which is easy to recognize by its definition, properties, and the findings of prior works.

Reviewer comment (Evaluation):
For the purpose of capturing low-, middle-, and high-order interactions, Moga Block uses three parallel branches. To confirm that the model concentrates more on the middle-order as anticipated, it would be preferable to give the gate values (or attention map).

Author response:
Thank you for your constructive feedback. We have provided a visualization comparison and visualization ablation study in Fig.5, Fig.8, and Fig. A4 in our manuscript. You can recognize the effect of MogaNet through these visualization results. Intuitively, the encoded extremely low- or high-order interactions focus on local textures or extremely discriminative parts, not localizing precise semantic parts. The gating operation effectively eliminates the disturbing contextual noise and makes the model effectively locate the semantic patterns.

Reviewer comment (Experiments):
The subtraction \(Y - GAP(Y)\), which "forces the network against its implicitly preferred extreme-order interactions," is the main function of FD. Why does the CA block not include this operation?

Author response:
We are delighted that you have noticed our FD(.) and CA(.) designs. In fact, the FD(.) and CA(.) are, as you said, built on the same concept and purpose. You may have recognized that they go hand in hand by going through the general response and other responses on novelty. CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation to force the network against its implicitly preferred interactions.

Reviewer comment (Presentation):
Wrong markup in Fig. 5, i.e., two \(1:0:0\).

Author response:
We appreciate you pointing out this issue. We have modified this in the revision. Many thanks again.

Reviewer comment (Evaluation):
Will MogaNet be less robust because low-order interaction "represents common and widely shared local patterns with great robustness"?

Author response:
MogaNet would not lose robustness. the texture portrayed by low-order interaction lacks high-level semantic features. Concretely, taking computer vision as an example, the model needs to face real-world complex scenes. Too much low-order interaction would make the model lose its discriminative capacity, reducing its robustness in real-world applications. Thus, MogaNet would not become less robust. Extensive experiments on various downstream tasks have empirically verified this point.

Reviewer comment (Experiments):
It would be interesting to know how the losses (presented in [r1]) and MogaNet pitted against (or collaborate with) each other, because both supervision signal and structure design matter for deep neural networks.

Author response:
Special thanks for your extremely valuable suggestions! We conducted the experiments you designated on CIFAR-100 with 224$\times$224 resolutions and 200 epochs training. The results show that simply combining MogaNet with the loss function in [1] (forcing the network to learn the specific interactions) does not yield performance gains, while Swin-T or ConvNeXt-T have little gains using the low-order or the high-order interaction loss. After analysis, we conclude that the loss function constraint of [1] forcing up a certain interaction might exert a negative effect on MogaNet. There might be two reasons. On the one hand, MogaNet expects to complement the interaction components that the network is originally prone to neglect rather than deliberately elevating a certain part. On the other hand, MogaNet has achieved high performance on CIFAR-100 compared to Swin-T and ConvNeXt, which might be too saturated to optimize by the loss functions. Based on these interesting findings, we plan to conduct more in-depth research based on this interesting result in our future work. We really appreciate your constructive suggestions.

| Loss Function      |   Swin-T  | ConvNeXt-T | MogaNet-S |
|--------------------|:---------:|:----------:|:---------:|
| Normal Training    |   78.25   |    82.74   | **85.31** |
| Low Interaction    | **78.33** |    82.75   |   84.60   |
| Middle Interaction |   77.97   |    82.46   |   85.28   |
| High Interaction   |   78.12   |  **82.80** |   84.98   |

Reviewer comment (Writing):
The two expressions before and after Equation 1 contradict each other, i.e., "in the same shape" and "downsampled features".

Author response:
We sincerely appreciate you pointing out this issue. We have modified this in the revision.

Reviewer comment (Novelty):
From the architecture perspective, the proposed MogaNet exploit multiple depth-wise convolutions for token mixing and channel attention-based FFN for channel mixing. The idea on multiple depth-wise convolutions for token mixing was used in RepLKNet, while channel attention-based FFN was explored in LocalViT and DHVT. So I have a bit doubt on technological novelty of core ideas on designing overall architectures of MogaNet. I suggest the authors can give more detailed analysis on novelty of MogaNet from the architecture perspective.

Author response:
### **2. Technological Novelty.**

Specifically, MogaNet's network architecture can be categorized into FD(.), Multi-DW(.) and CA(.). The first two together form the Spatial Moga(.) block. You have mentioned the following counterparts: VAN [1], FocalNet [2], LocalViT [3], Conv2Former [4], and PVT-V2 [5]. First, please forgive me for reiterating that MogaNet is from a completely unique starting point compared to these works, and we are not solving the same problem. Second, just at the level of network architecture design, we are also quite apart, and MogaNet is still novel.

**(a) For the Spatial Operation:** the VAN [1] uses large DW convolution kernels in series to perform convolution operations. FocalNet [2] utilizes DW convolution kernels to perform convolution and gating operations in series for hierarchical feature maps. Conv2Former [4] applies a so-called “convolutional modulation” operation for feature extraction. Different from all of them, MogaNet first uses a DWConv, then performs channel-wise splitting operations, and performs different operations (different sizes of DWConvs and identity connections) for different groups of channels **in parallel**, followed by a double-branch activated gating operation. This is a completely different design from the aforementioned works. More importantly, we also have different starting points. We have also performed extensive ablation studies and empirical analysis, and our results show that our design achieves exceptional performance gains while being closely aligned with our aforementioned main purpose.

**(b) For the Channel Operation:** Firstly, starting from LocalViT [3] and PVT-V2 [5], most of the ViT or modern ConvNet models utilize “CFFN” (Convolutional FFN, i.e., 1$\times$1 Conv → 3$\times$3 DWConv → 1$\times$1 Conv) as the CMixer. In other words, this is a consensus now for the community and should not be taken as a reason for the “limited novelty” of MogaNet. Secondly, MogaNet's CA(.) is designed to address the aforementioned macro-objectives, and its design concept is the same as that of FD(.). Meanwhile, to the best of our knowledge, CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation to **force the network to learn the inherently overlooked features**. Combined with the outstanding performance of CA(.) and the provided empirical analysis, we believe that it is certainly evident to verify the novelty of CA(.).

### Reference

[1] Visual Attention Network, CVMJ 2023.

[2] Focal Modulation Networks, NeurIPS 2022.

[3] LocalViT: Bringing Locality to Vision Transformers, ArXiv 2021.

[4] A Simple Transformer-Style ConvNet for Visual Recognition, ArXiv 2022.

[5] PVT v2: Improved Baselines with Pyramid Vision Transformer, CVMJ 2022.

Reviewer comment (Evaluation):
The compared results show that performance gains of the proposed MogaNet over existing popular architectures is not significant. So could the authors show more advantages of the MogaNet?

Author response:
### **3.  About Performance Gains.**

We fully respect your assessment that the performance gains are not significant. However, we would like to emphasize two points in this regard:

**(a) Consistent gains across a broad range of tasks.**  

We have conducted extensive experiments in mainstream image benchmarks and various downstream tasks. The results show that MogaNet achieves the consistent best results. To be clear, many related prior works have not done that many experiments, and they are also accepted by top conferences. In contrast, MogaNet is rather the more empirically testable work with sound experiments.

**(b) Comparison with prior SOTA models.** 

Taking the small CNN models as an example, we rank the SOTA models by **chronological order** and count their performance gains on ImageNet-1K. Note that we reproduce RepLKNet-T$^\dagger$ according to the layer setting in SLaK. We observe that, with the exception of the earlier "low-hanging fruit" work, all the SOTAs provide a gain of **0.2~0.7% top-1 accuracy** over the previous work (ConvNeXt as the baseline), and some of them even suffer from performance **degradation**. In contrast, MogaNet-S shows a performance gain of **1.3% accuracy** over ConvNeXt-T, which exceeds the results of prior works accepted by top conferences. Meanwhile, we also add the self-supervised pre-training result of MogaNet (using A2MIM [5]) in comparison to ConvNeXt.V2 [6] (using CMAE [6]), indicating that MogaNet can further benefit from self-supervised learning.

| Architecture           | Date      | \# Param. | Pre-train |   Top-1 (Gain)  |
|------------------------|:---------:|:---------:|:---------:|:---------------:|
| ConvNeXt-T             | CVPR'2022 |    29M    |     -     |   82.1 (+0.0)   |
| RepLKNet-T$^\dagger$      | CVPR'2022 |    29M    |     -     |   82.4 (+0.3)   |
| HorNet-T$_{7\times 7}$ | NIPS'2022 |    22M    |     -     |   82.8 (+0.7)   |
| FocalNet-T (LRF)       | NIPS'2022 |    29M    |     -     |   82.3 (+0.2)   |
| VAN-B2                 | CVMJ'2023 |    27M    |     -     |   82.8 (+0.7)   |
| SLaK-T                 | ICLR'2023 |    30M    |     -     |   82.5 (+0.4)   |
| ConvNeXt.V2-T          | CVPR'2023 |    28M    |    CMAE   |   83.0 (+0.9)   |
| **MogaNet-S**          |    Ours   |    25M    |     -     | **83.4 (+1.3)** |
| **MogaNet-S**          |    Ours   |    25M    |   A2MIM   | **83.5 (+1.4)** |

Thus, we believe that the gains brought by MogaNet are perfectly adequate and not trivial.

Reviewer comment (Evaluation):
Besides parameters and FLOPs, latency is more important for practice applications. So could the authors show some results on the latency (e.g., training time and inference time) of the MogaNet?

Author response:
### **4. Latency Results.**

Due to the limited space, we have not shown the latency results in the main test. However, we provide results for the throughput in Appendix C.3, and MogaNet demonstrates a competitive throughput performance. It proves that MogaNet can take into account the throughput efficiency, in addition to the parameter efficiency and FLOPs. Please view Table A3 in Appendix C.3 for details.

Reviewer comment (Experiments):
There exist much more efficient and effective channel attention, besides SE. Therefore, the authors would better compare more efficient channel attention methods to verify the effectiveness of CA module.

Author response:
### **5. CA Counterpart Comparison.**

In the CMixer, compared to popular channel aggregation modules (SE, GLU, SwiGLU [1]) and specially-designed FFN variants (CMT [2] and RepFFN [3]) on ImageNet-1K based on MogaNet-S. CFFN (1$\times$1 Conv → 3$\times$3 DWConv → 1$\times$1 Conv) served as the baseline in PVTv2 [4]. Note that RepFFN utilizes 4 group convolution kernels (gConv) to reduce the parameters and apply a structural re-parameterization to merge the kernels into the FC layer for inference.

| Method           |    Data    | \# Param. | Additional Modules      |   Top-1  |
|------------------|:----------:|:---------:|---------------------------|:--------:|
| CFFN (PVTv2)     |  CVMJ'2022 |   24.5M   | -                          |   82.9   |
| +SE              |  ICCV'2019 |  +0.71M   | GAP, 2xFC, ReLU      |   83.1   |
| +GLU             | arXiv'2020 |   +0.0M   | Split, Sigmoid             |   82.9   |
| +SwiGLU (DINOv2) | arXiv'2023 |   +0.0M   | Split, SiLU         |   83.0   |
| +IRFFN (CMT)     |  CVPR'2022 |  +0.05M   | BN, GELU          |   83.0   |
| +RepMLP          |  CVPR'2022 |  +1.75M   | 4xgConv (k=1, 3, 5, 7), 4xBN |   83.2   |
| **+CA**          |    Ours    |  +0.52M   | FC, Scale, GELU           | **83.4** |

Overall, we sincerely appreciate your effort and valuable feedback. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We are more than pleased and are looking forward to hearing back from you!

Reviewer comment (Novelty):
Despite good experiments and visualizations, I think the novelty is limited. As described in the introduction, the low-order interactions are modeling the local features, such as edge and texture. The high-order on the other hand models high-level semantic features. So multi-order feature aggreation indicates the multiscale aggregation with low and high level features. This paper implements it via depth-wise convolution with different kernel size and further adds gated operation, introducing multi-order gated aggregation. However, FocalNet exhibits similar behavior, proposing hierarchical gated aggregation with locality perception and gated aggregation. So I think the proposed MogaNet has similar motivation and mechanism with FocalNet [1].

Author response:
We express our gratitude for your valuable review and constructive feedback. We have adjusted our revision and provided a general response to some common issues. We invite you to go through the general response first, as they may have answered some of your confusion, and you will have a new understanding of the novelty and contributions of MogaNet.

Upon reflection, we agree that there are opportunities to strengthen the novelty and clarify the contributions of our proposed method.  Specifically, MogaNet's network architecture can be categorized into FD(.), Multi-DW(.) and CA(.). The first two together form the Spatial Moga(.) block. Here, we will address your concern about the novelty from the following three aspects: 

### **1. On Novelty**

You raise important points regarding the novelty and differentiation of our method from related work. We agree that aggregating multi-scale features is a common theme in modern ConvNets, or the so-called “convolutional modulations”. While related work explores multi-scale aggregation, we believe our MogaNet is quite novel from both its purpose and the design itself.

Our MogaNet work was inspired by an ICLR22 work [1], which revealed for the first time the problem that existing deep neural networks suffer from a representation bottleneck under a hierarchical theoretical analysis (Multi-order Game-theoretic Interaction) and thereby proposed a loss function to solve this problem. This work was recognized by ICLR2022, but no further attempts have been made to solve this problem. Thus, we are inspired to address this issue from the novel perspective of network architecture design (instead of loss function).

Specifically, deep neural networks show a natural proclivity towards multi-order interactions - the most expressive middle-order interactions are not well encoded in the current learning process, and we attempt to address this issue with network architectures that can **force the network to learn to encode more expressive interactions that would have otherwise been inherently overlooked.** To facilitate understanding, from a computer vision standpoint, you can roughly, but not quite accurately, interpret this as forcing the network to capture features that were discriminative but inherently ignored.

To the best of our knowledge, **we are the first to address this issue [1] with a specific network architecture design and are also the first to approach network architecture design from the multi-order interaction [1] perspective.** Therefore, it is clear that we have a completely different starting point compared to other related works. Again, briefly speaking, we are making a new attempt to solve the problem that was first raised by the ICLR 2022 work [1] rather than a merely "borrowed" network design without a top-down guideline. As such, we believe that MogaNet is obviously interesting and novel from an academic research point of view. Moreover, we summarize the performance gains of MogaNet and related ConvNets on ImageNet-1K as follows, indicating the effectiveness and importance of our work in the line of improving ConvNets.

### Reference

[1] Discovering and Explaining the Representation Bottleneck of DNNs, ICLR 2022.

Reviewer comment (Novelty):
The proposed method is a variant of convolutional modulation, but lacks an in-depth discussion on differences with recent CNNs based on convolutional modulations, such as VAN [2], FocalNet [1], and Conv2Former [3]. Besides, VAN [2] and FocalNet [1] should be added in Figure 3 for a comprehensive analysis on the interaction strength.

Author response:
### **2. On Discussion of Other Convolutional Modulations.**

You have mentioned the following similar works: VAN [1], FocalNet [2], and Conv2Former [3] (an occurrent work as MogaNet in late 2022). While related literature exists, the **intention and starting points** behind our methods are actually **far apart**. Firstly, MogaNet is from a completely unique starting point compared to these works, and we are not solving the same problem. We aim to solve this obstacle within DNNs by architecture design. We actually own completely different motivations and targets, so we don’t discuss them in this work as the available pages are limited. Second, just at the level of network architecture design, we are also quite apart, and MogaNet is still novel and important:

**(a) For the Spatial Operation:** the VAN [1] uses large DW convolution kernels in series to perform convolution operations. FocalNet [2] utilizes DW convolution kernels to perform convolution and gating operations in series for hierarchical feature maps. Conv2Former [3] applies a so-called “convolutional modulation” operation for feature extraction. Different from all of them, MogaNet first uses a DWConv, then performs channel-wise splitting operations, and performs different operations (different sizes of DWConvs and identity connections) for different groups of channels **in parallel**, followed by a double-branch activated gating operation. This is a completely different design from the aforementioned works. More importantly, we also have different starting points. We have also performed extensive ablation studies and empirical analysis, and our results show that our design achieves exceptional performance gains while being closely aligned with our aforementioned main purpose. In addition, MogaNet's performance is far above them.

**(b) For the Channel Operation:** First, starting from LocalViT [4] and PVT-V2 [5], most of the ViT or modern ConvNet models utilize CFFN (Convolutional FFN, i.e., 1$\times$1 Conv → 3$\times$3 DWConv → 1$\times$1 Conv) as the channel feature extractor. In other words, this is a consensus now for the community and should not be taken as a reason for the “limited novelty” of MogaNet. Second, MogaNet's CA(.) is designed to address the aforementioned macro-objectives, and its design concept is the same as that of FD(.), which is **forcing the network to learn to encode more expressive interactions that would have been inherently (originally) overlooked.** Meanwhile, to the best of our knowledge, CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation. Combined with the outstanding performance of CA(.) and the provided empirical analysis, we believe that it is certainly evident to demonstrate the novelty of CA(.).

We appreciate your helpful feedback and will add a related description in our revision. You rightly note that we could have better situated our work within the broader context of convolutional modulations. We have revised our Sec.1, Sec.2, and Sec.3 in our revision. As shown in the following table, we conduct interaction analysis experiments, which show that MogaNet not only surpasses the popular modern ConvNets you mentioned but exhibits better interaction performance, where the expressive middle-order interactions (0.3n~0.5n) are effectively enhanced. View Appendix B.1 and Fig. A2 for details.

| Modules                |    Date    |   Top-1  |  0.0n | 0.05n |  0.1n |    0.3n   |    0.5n   |    0.7n   |  0.9n | 0.95n |  1.0n |
|------------------------|:----------:|:--------:|:-----:|:-----:|:-----:|:---------:|:---------:|:---------:|:-----:|:-----:|:-----:|
| ConvNeXt-T             |  CVPR'2022 |   82.1   |  2.98 |  2.11 |  1.19 |    0.60   |    0.44   |    0.42   |  0.60 |  1.02 |  2.03 |
| HorNet-T$_{7\times 7}$ |  NIPS'2022 |   82.8   | +0.62 | -0.16 | +0.02 |   +0.01   |   +0.02   |   +0.02   | -0.02 | -0.06 | -0.55 |
| FocalNet-T (LRF)       |  NIPS'2022 |   82.3   | +0.73 | -0.28 | +0.06 |   +0.07   |   +0.04   |   +0.07   | +0.06 | -0.07 | -0.47 |
| VAN-B2                 |  CVMJ'2023 |   82.8   | -0.08 | -0.13 | +0.18 |   +0.14   |   +0.11   |   +0.13   | +0.05 | -0.18 | -0.77 |
| SLaK                   |  ICLR'2023 |   82.5   | +0.13 | -0.21 | -0.02 |   +0.01   |   +0.04   |   +0.08   | +0.07 | +0.02 | -0.16 |
| Conv2Former-T          | arXiv'2022 |   83.2   | -0.32 | -0.29 | +0.20 |   +0.16   |   +0.12   |   +0.15   | +0.08 | -0.10 | -0.75 |
| **MogaNet**            |    Ours    | **83.4** | -0.70 | -0.26 | +0.23 | **+0.19** | **+0.22** | **+0.20** | +0.13 | -0.09 | -0.35 |

### Reference

[1] Visual Attention Network, CVMJ 2023.

[2] Focal Modulation Networks, NeurIPS 2022.

[3] A Simple Transformer-Style ConvNet for Visual Recognition, ArXiv 2022.

[4] LocalViT: Bringing Locality to Vision Transformers, ArXiv 2021.

[5] PVT v2: Improved Baselines with Pyramid Vision Transformer, CVMJ 2022.

Reviewer comment (Experiments):
Regarding Figure 7 of the ablation study, I am confused that the main improvement is not from multi-order convolution or gate, which are claimed as major contributions of this work. Instead, the main improvement is from the CA, which is embedded in the feed-forward network. Note that other networks do not have CA in their feed-forward network, introducing somewhat unfair comparison. Therefore, I think the authors should better clarify the mechanism of the CA and claim it as the major contribution, not only emphasizing the proposed multi-order gated aggregation.

Author response:
### **3. On the Main Improvement in MogaNet.**

The reviewer makes a fair point. Our ablation study could have better disambiguated the contributions of multi-order aggregation (Moga) vs. CA. To clarify, the spatial Moga(.) module and channel CA(.) module collaborate **as an integration** to achieve our goal, where **the expressive but originally overlooked interactions encoded in DNNs can be forced to enhance.** As shown in the following table, we ablate the proposed modules by adding them one by one to the baseline (similar to Table 1) and calculate the distributions of interaction strengths. Note that 0.3n~0.5n reflects the favorite middle-order interactions. In particular, CA(.) is embedded to allow aggregation to dynamically attend to useful features. CA(.) is designed to address the aforementioned macro-objectives, and its design concept is the same as that of FD(.). As provided in the new ablation study of our revision, the CA(.) can only perform its best coupled with Moga(.). It empirically indicates that the Moga(.) and CA(.) collaborate as a unified design for the same goal but are designed from different feature spaces (spatial-wise and channel-wise, respectively).

| Modules            |  Parts |   Top-1  |  0.0n | 0.05n |  0.1n |  0.3n |  0.5n |  0.7n |  0.9n | 0.95n |  1.0n |
|--------------------|:------:|:--------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
| Baseline           |    -   |   82.2   |  2.98 |  2.11 |  1.19 |  0.60 |  0.44 |  0.42 |  0.60 |  1.02 |  2.03 |
| +Gating branch     | SMixer |   +0.2   | -0.06 | -0.03 | +0.01 | +0.03 | +0.01 | +0.02 | -0.01 | +0.02 | +0.01 |
| +Multi-order DW(.) | SMixer |   +0.5   | -0.35 | -0.22 | +0.15 | +0.10 | +0.12 | +0.11 | +0.09 | -0.13 | -0.20 |
| +FD(.)             | SMixer |   +0.7   | -0.47 | -0.24 | +0.15 | +0.13 | +0.17 | +0.15 | +0.10 | -0.10 | -0.28 |
| +CA(.)             | CMixer | **+1.2** | -0.70 | -0.26 | +0.23 | +0.19 | +0.22 | +0.20 | +0.13 | -0.09 | -0.35 |

In conclusion, we thank you for pushing us to better position our work with respect to the rising landscape of modern ConvNets. We sincerely appreciate all your valuable feedback to help us improve our work.  We believe that the issues raised by the reviewers have been addressed in the revision. We hope that you will reconsider our submission with this response and the revision. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We are more than pleased and looking forward to hearing back from you!