Paper: Efficient Modulation for Vision Networks (ICLR-2024)

Reviewer comment (Evaluation):
One of the most curious things about the paper is the ablation results in Table 5. From there it seems that replacing the modulation with just a regular residual connection (sum) has quite modest performance drop (abs perf drop 1%). Without the multiplication, unless I am mistaken, the architecture reduces to a ResNet with specific hyperparameters and two-path construction. Could the authors discuss this. I think both contributions are valuable, but I am wondering whether it is correct to attribute the performance of the architecture to the modulation since other aspects seem to play even larger role than abs. perf. of 1%.

Author response:
**We apologize for any confusion caused by Table 5 and appreciate your curiosity regarding its results**. To clarify, the performance variations observed in the table are attributed to different factors. The reduction in performance in top rows is due to a decrease in parameters and FLOPs. However, the change from multiplication to summation in our model, which does not alter the parameters or FLOPs, results in a significant accuracy drop of 1%.

To provide a clearer understanding, we have revised Table 5 to include FLOPs and parameters, and have explicitly indicated which components were removed or altered:




| f(.) | Conv | g(.) | Params. | FLOPs | Acc | Acc drop | Comment |
|---|---|---|------|----|----|-------|-------|
| ✓ | | | 11.3M | 1.2G | 72.7 | -7.8 | Remove second FC layer, no sptial interction |
| | ✓ | | 10.3M | 1.1G | 78.6 | -0.9 | Remove two FC layers |
| | | ✓ | 11.3M | 1.2G | 72.3 | -8.2 | Remove first FC layer, no sptail interaction |
| ✓ | ✓ | | 11.6M | 1.3G | 79.8 | -0.7 | Remove second FC layer |
| | ✓ | ✓ | 11.6M | 1.3G | 79.6 | -0.9 | Remove first FC layer |
| ✓ | ✓ | ✓ | 12.9M | 1.4G | 80.5 | - | - |
| mul. | to | sum | 12.9M | 1.4G | 79.5 | -1.0 | Change multiplication to summation |


This updated table elucidates the critical role of the multiplication operation in our method. Changing from multiplication to summation, while keeping the parameters and FLOPs constant, leads to a 1.0% decrease in accuracy, a notable gap for efficient models. In fact, this performance gap is larger than that caused by reducing parameters and FLOPs (see row 2, 4 and row 5).  Row 1 and Row 2 means no convlutional layer in the network, hence no spatial interactions, resulting to significant performance drop. The primary objective of this ablation study was to assess the impact of each component in our design individually. Replacing multiplication with summation does transform our network into a ResNet-like architecture, ignoring detailed designs.

Reviewer comment (Experiments):
Continuation of the above (without mult). Would the architecture without mult be second best of the architectures compared in Table 2. If so, please add it there.

Author response:
**Response:** Yes, when we replace multiplication with summation, our model achieves an accuracy of 79.5%. We appreciate your suggestion and will include this variant in Table 2 of our paper for a more comprehensive comparison.

Reviewer comment (Novelty):
Since this work is most closely related to VAN and FocalNets those works should be at least briefly mentioned in the related work section as well.

Author response:
**Response:** Thank you for highlighting the importance of discussing VAN and FocalNet. We have detailed our comparison with these models in Section 3.1 and Table 4, as shown here:


| Model            | Top-1 (%)↑ | GPU (ms)↓ | CPU (ms)↓ | Param. | FLOPs |
|------------------|------------|-----------|-----------|--------|-------|
| VAN-B0           | 75.4       | 4.5       | 16.3      | 4.1M   | 0.9G  |
| FocalNet@4M      | 74.5       | 4.2       | 16.8      | 4.6M   | 0.7G  |
| EfficientMod-xxs | **76.0**       | **3.0**       | **10.2**      | 4.7M   | 0.6G  |


We recognize the value of your suggestion and will further expand our discussion on VAN and FocalNet in the related work section to provide a more thorough context.

Reviewer comment (Presentation):
In Fig 1 and corresponding text, would it make sense to mention where nonlinearities are applied?

Author response:
**Response:** The activation function is applied after convolutional layers. We will show that in the figure.

Reviewer comment (Presentation):
Is Figure 1a) missing the softmax part of the transformer architecture? I think this is one of the key differences to the modulation designs.

Author response:
**Response:** You are correct in observing the omission of the softmax layer in Figure 1a. This was done for simplicity, and as noted in the caption, activation and normalization layers were also excluded. We will revise the figure to include the softmax layer, ensuring a more accurate representation of the transformer architecture.

Reviewer comment (Writing):
In section 3.1, the FC layer seems the same as a 1x1 pointwise convolutional layer. If this is correct, it might be beneficial to mention this.

Author response:
**Response:** Yes, FC layer is mathmatically same as 1x1 point-wise convolutional layer. Thanks for your detailed question and we will will make a note of it in the updated manuscript.

Reviewer comment (Theory):
Examining Figure 1, and comparing (b) and (c), the proposed EfficientMod block fuses the MLP on the top and the modulation block as one unified block to improve efficiency. It is conceivable that this might limit performance when using the same number of parameters. The authors should elucidate the principles behind this design, not only from the perspective of efficiency but also in terms of representational ability.

Author response:
**Response:** We appreciate your thoughtful recommendations. We examined the effectiveness of slimming the modulation architecture by integrating an extra MLP layer with our EfficientMod block and modifying expansion ratios to retain comparable parameters and FLOPs. That is, the EfficientMod is divided into an EfficientMod block (having an expansion ratio of 1) and an MLP block (with an expansion ratio of (r-1)). Here are the results based on EfficientMod-s-Conv:

| Method | Parameters | FLOPs | Accuracy | GPU Latency | CPU Latency |
|----|-----|-----|------|-----|----|
| EfficientMod-s-Conv (sperate MLP) | 12.9M | 1.5G | 80.6 | 6.2 ms | 26.2 ms |
| EfficientMod-s-Conv | 12.9M | 1.5G | 80.5 (-0.1) | 5.8 ms (-0.4) | 25.0 ms (-1.2) |


As shown in the table, the sperating MLP block just have a marginally better representational ability (80.6 vs. 80.5), which is negligible. 
By fusing MLP with the EfficientMod block, we are able to increase the inference speed on the CPU and GPU a lot.

Reviewer comment (Experiments):
Could the authors discuss whether the transformer block can also benefit from the proposed method for efficiency (at least it is feasible to fuse the MLP into one unified block)?

Author response:
**Response**: It is feasiable to Transformer block to fuse the MLP into one unified block. We thank the reviewer for the insightful questions, which has a great impact to Transformer-based architectures, not only in efficient networks, but also in large models. 

Indeed, there are some endeavors that tried to fuse MLP block to a unified block for faster inference. For example, ViT-22B [1] applies the Attention and MLP blocks in
parallel for faster inference, instead of sequentially as in the standard Transformer. In detail, the Q, K, V, and first layer in the original MLP block are fused into one layer, the attention-out layer and the second layer in the original MLP block are fused. Also, PaLM [2] also considers this strategy to speed up inference, rewriting the standard formulation of Transformer from $y=x + MLP(LN(x+Attention(LN(x))))$ to $y=x + MLP(LN(x) + Attention(LN(x)))$. As indicated, the unified block results in roughly 15% faster training speed at large scales. 

Hence, it is feasible to fuse the MLP block to a unified block for faster inference. Our EfficientMod represents an early exploration of this approach in the realm of efficient networks.



[1] Dehghani, Mostafa, et al. "Scaling vision transformers to 22 billion parameters." ICML, 2023.

[2] Brown, Tom, et al. "Language models are few-shot learners." NeurIPS, 2020.

Reviewer comment (Presentation):
In Figure 8 and Figure 9, one of the notations in the legend should be 'EfficientMod'.

Author response:
**Response**: Thank you for pointing out this. We will amend the legend to correctly represent 'EfficientMod' (not 'MobileMod') in our revised manuscript.

Reviewer comment (Evaluation):
In Table 2, there are no latency reported for state-of-the-art efficient models.

Author response:
Latency (for both CPU and GPU) is primarily reported in Table 1, comparing state-of-the-art efficient models. Table 2 focuses on demonstrating the empirical improvements of introducing attention to our method. 

However, to address your concern, we have benchmarked all models in Table 2 (incorporating results from Table 1) and present the updated results here:


| Arch.|Model | Params | FLOPs | Acc. | Epoch | Latency GPU | Latency CPU |
|:--------:|:----:|:------:|:-----:|:----:|:-----:|-----|--------|
| Conv |RSB-ResNet-18 |12.0M |1.8G | 70.6 |300| 2.0 | 11.5|
| Conv |RepVGG-A1 |12.8M |2.4G | 74.5 |120| 2.0 | 15.0|
| Conv | PoolFormer-s12 |11.9M |1.8G | 77.2 |300| 5.0 | 22.3|
| Conv | GhostNetv2x1.6 |12.3M |0.4G | 77.8 |450| 5.7 | 19.1|
| Conv |RegNetX-3.2GF |15.3M |3.2G | 78.3 |100| 6.1 | 22.7|
| Conv |FasterNet-T2|15.0M |1.9G | 78.9 |300| 4.4 | 18.4|
| Conv |ConvMLP-M |17.4M |3.9G | 79.0 |300| 4.1 | 32.1|
| Conv | GhostNet-A |11.9M |0.6G | 79.4 |450| - | - |
| Conv |MobileOne-S4|14.8M |3.0G | 79.4 |300| 4.8 | 26.6|
| Conv | EfficientMod-s |12.9M |1.5G | **80.5** |300| 5.8 | 25.0|
| + Atten. | EfficientMod-s |12.9M |1.4G | **81.0** |300| 5.5 | 23.5|


As illustrated, our method demonstrates both high speed and promising performance. While a single scale of models may not fully represent the performance-latency trade-off across different models, we invite readers to refer to Figure 3 in our submission for a comprehensive efficiency analysis of our method.

Reviewer comment (Theory):
The proposed method seems simple and more analysis and motivations for the design are needed to understand the principal of the design choice.

Author response:
We appreciate your perspective on the simplicity of our method. We believe that its simplicity, coupled with effectiveness, could be an advantage
. The primary motivation for our design is the suitability of the modulation mechanism for efficient networks, given its computational efficiency and dynamic nature, blending the benefits of convolution and attention. This motivation of modulation mechanism enables EfficientMod to be straightforward yet impactful.

In detail, we further elaborate on our motivations and the analysis behind our design choices to clarify our approach:

**Principles and Contributions:** At the core of our work, we have summarized the modulation mechanism and further developed the EfficientMod block. This design effectively harnesses the strengths of both convolution (i.e., local representation) and attention mechanisms (dynamics).

**Customization of the EfficientMod Block:** We have tailored the EfficientMod block to enhance efficiency and effectiveness. This customization involved streamlining the modulation design and simplifying the context modeling process. These refinements are also important in achieving best balance between performance and computational efficiency.

**Integration with Attention Blocks:** In integrating attention blocks, we strategically placed attention blocks only in the last two stages of the network for computational efficiency.

Reviewer comment (Novelty):
There are limited technical contributions in the work. This paper focuses on improving the latency of previous works. The improvements/changes from previous works are mainly engineering designs, for example, fuse multiple FC layers together, fuse multiple DWConv into a larger one, replace reshape operation with repeat. The guidance is mainly from previous works such as ShuffleNet v2, which is to reduce fragmented operations for improved latency. There are limited new insights.

Author response:
**Q1: Limited technical contributions in the work.**

**Response**: We understand your concerns regarding the perceived technical contributions. While certain implementation aspects like layer fusion and reshaping operations are not our main focus, we emphasize that our primary contributions lie in:

**Summarization of Modulation Mechanism**: We provide a comprehensive overview of the modulation mechanism in Section 3.1, highlighting its unique position in the landscape of efficient network designs.

**Efficient Modulation Design**: As detailed in Section 3.2, we introduce the EfficientMod block, a novel building block for efficient networks, leveraging both convolutional and attention mechanisms.


Compared with other efficient network designs, we hold the idea that modulation is unique and distinguished. It can take advantage of both convolution (local representation, efficiency) and attention mechanism (dynamics). Here we list a taxonomy of the representative efficient networks in Table 1 based on their main technical contributions:

| Technical Contributions | Networks |
|-------|----------|
| DW-Convolution | MobileNet, MobileNetv2 |
| Channel Shuffle | ShuffleNet, ShuffleNetv2 |
| Feature Re-use | FasterNet |
| Re-parameterization | MobileOne |
| Neural Architecure Searching | EfficientNet, EfficientFormerV2 |
| Hybrid architecture | MobileViT, EfficientFormerV2, MobileFormer |
| Modulation Mechanism | EfficientMod (ours) |

This table underscores the distinctiveness of our contribution, positioning the modulation mechanism as a key innovation in EfficientMod. Our work focuses on advancing this aspect, offering a significant step forward in the field of efficient network design.

Reviewer comment (Presentation):
Fig 1 is good to illustrate the difference between the proposed method and previous works. But it could be better to expand Fig 1 (c) in details when explaining the method. This makes it easier to understand the proposed structure and details.

Author response:
**Q3: It could be better to expand Fig 1 (c) in details when explaining the method.**

**Response**: Thank you for your kind suggestions,  we would revise the Fig 1 (c) and add more details in it, ensuring that our method is communicated more effectively and comprehensibly.

Reviewer comment (Experiments):
In Table 1, why VAN and FocalNet results are not included? They seem to be the most relevant works.

Author response:
**Q4: In Table 1, why VAN and FocalNet results are not included? They seem to be the most relevant works.**

**Response:**  The comparison (EfficientMod, VAN, and FocalNet) is detailed in our Ablation Study Table 4, rather than Table 1. This is because VAN and FocalNet models do not fall under the category of 'efficient models' and lack small variants. 

For the study, we developed a FocalNet variant with 4M parameters (FocalNet@4M), selecting the best-performing variant among several tested. Our analysis in Table 4 demonstrates that EfficientMod not only outperforms FocalNet and VAN in terms of efficiency but also exhibits significantly lower latency on both CPU and GPU.

For your convenience，we included Table 4 in our submission here:

| Model | Top-1(%)↑| GPU (ms)↓ | CPU (ms)↓   | Param. | FLOPs |
|:------:|---------|------|-------|--------|-------|
|VAN-B0 | 75.4  | 4.5  | 16.3 | 4.1M   | 0.9G  |
|FocalNet@4M  | 74.5 | 4.2 | 16.8 | 4.6M   | 0.7G  |
| EfficientMod-xxs  | **76.0**     | **3.0** | **10.2**  | 4.7M   | 0.6G  |

Reviewer comment (Evaluation):
In Table 2, why adding Attention even reduced the FLOPs?

Author response:
**Q5: In Table 2, why adding Attention even reduced the FLOPs?**

**Response:** The reason is that we replaced some EfficientMod blocks with Attention blocks to keep almost the same number of parameters, for a fair comparison. 

This involved adjusting the expansion ratio in Attention blocks and other hyper-parameters. We believe that the number of parameters plays a crucial role in performance, and maintaining a consistent parameter number is essential for a fair comparison.

Reviewer comment (Evaluation):
In Table 1, are the GPU and CPU latency of different models measured on the same device?

Author response:
**Q6: In Table 1, are the GPU and CPU latency of different models measured on the same device?**

**Response:** Yes, all models were benchmarked on the same device to ensure consistency in our measurements. 

Additionally, we conducted benchmarks across various GPU types, as detailed in Supplementary Table 10. This approach was taken to maintain uniformity in the testing environment and minimize latency variances across different GPU models.