Paper: Simple CNN for Vision (ICLR-2024)

Reviewer comment (Experiments):
Experiments: The paper should delve deeper into the experimental setup, data augmentation techniques, training specifics, and more. For example, it is unclear to compare with other baselines that have different parameter sizes. The author should clarify this section.

Author response:
Thanks for your comment. All of our settings are the same as Swin Transformer and ConvNeXt, we have not made any further adjustments.

Reviewer comment (Presentation):
Lack of Visualizations: Additional figures visualizing feature maps or demonstrating how the receptive field increases would offer more insights into the workings of SCNN.

Author response:
We use the script in RepLKNet(https://github.com/DingXiaoH/RepLKNet-pytorch/tree/main/erf) to visualize the receptive field, and put these images in the **appendix (page 12)**. For models trained in imagenet-1k, all of them could capture a large receptive field. After adding more convolutions and GSiLU, the features become more discriminative and the final model focuses more on center and edge areas.

Reviewer comment (Experiments):
The paper lacks some analysis on how the proposed architecture captures spatial context and increases receptive field size. The introduction describes this as a key motivation, but there is little discussion in the experiments. Some visualization or measurements of the receptive field size could provide more insight.

Author response:
We use the script in RepLKNet(https://github.com/DingXiaoH/RepLKNet-pytorch/tree/main/erf) to visualize the receptive field, and put these images in the **appendix (page 12)**. For models trained in imagenet-1k, all of them could capture a large receptive field. After adding more convolutions and GSiLU, the features become more discriminative and the final model focuses more on center and edge areas.

Reviewer comment (Novelty):
The paper does not discuss in detail some other related work on improving convolutional backbones, such as [1,2]. Comparing and contrasting with these methods could highlight the novelty of SCNN's approach.  
[1] Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu:Visual attention network. Comput. Vis. Media 9(4): 733-752 (2023)  
[2] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao: PVT v2: Improved baselines with Pyramid Vision Transformer. Comput. Vis. Media 8(3): 415-424 (2022)

Author response:
We cite and compare with PVT in instance segmentation tasks, but we overlook PVT v2 and VAN because they are published in a newly established journal (2015) we overlook. We will cite them in the final version. Compared with PVT v2, SCNN-T gets **the same accuracy (83.2%) with only 65% FLOPs (4.5G vs 6.9G)**. Compared with VAN, SCNN-T obtains **better results (83.2% vs 82.8%) with fewer FLOPs (4.5G vs 5.0G)**.

Reviewer comment (Novelty):
For global context modeling, how does GSiLU compare to other approaches like SENet? The global context modeling in SCNN does not appear to be very different from widely used techniques like SE modules (Channel Attention for Convolution). The paper could benefit from more comparison and discussion about the relative merits of GSiLU.

Author response:
However, both GSiLU and SE belong to gated activation functions. And replacing GSiLU with SE, the performance gain is marginal (from 83.2% to 83.3). For a fair comparison, we utilize GSiLU because both SiLU and GSiLU are parameter-free activation functions, and we only add a global average pooling. SE requires an extra MLP, which introduces a large number of parameters.

Reviewer comment (Writing):
The paper could benefit from a more polished and professional tone in its presentation and writing style.

Author response:
Thank you, we will polish the manuscript.

Reviewer comment (Experiments):
Some of the claims or hypotheses put forth lack empirical validation through experiments, which could strengthen the paper's assertions.

Author response:
Thank you, we will clearly check our claims or hypotheses and provide more references, more experiments, and more visualizations in the final version.

Reviewer comment (Experiments):
While it's understandable given potential computational constraints, the model was not trained on ImageNet 21K, a limitation that might affect generalization claims.

Author response:
ImageNet 21K is too large for our limited resources. We need more time to ask for help from technology companies to do this experiment in the final version.

Reviewer comment (Experiments):
The paper doesn't explore the performance of larger-sized versions of their model. Although this might be due to resource limitations, such exploration could provide additional insights into the model's scalability and robustness.

Author response:
Due to the limitation of resources, we train the base version of the model (15.4G). We will try to ask for help from some technology companies, to train a larger version (e.g. 50G FLOPs).

Reviewer comment (Experiments):
The paper describes the use of both Layer Normalization (LN) and Batch Normalization (BN) within the architecture, with LN explicitly employed as the initial normalization layer. Could the authors explain the reason behind this specific arrangement? Furthermore, it would be beneficial if the authors could provide an ablation study examining the impact of the positioning of these normalization layers within the network.

Author response:
To train the model faster, we use apex in PyTorch to train an fp16-fp32 mixed model. We find removing the LN will lead to a numeric overflow problem in the base version (15.4G). In tiny version (4.5G) and small version (8.7G), with or without LN, the result remains unchanged.

Reviewer comment (Experiments):
The paper describes the use of both SiLU and GSiLU activation functions within the architecture, with the use of SiLU as the first activation layer. Could the authors provide insight into the reason underpinning this specific sequence?

Author response:
SiLU has a similar formulation to GELU and Swish, which are widely used in most modern backbones. Thus, it is our default setting to use SiLU in all positions. And, we find replacing the second SiLU with GSiLU will obtain a better result (+0.1%) compared with replacing the first one.

Reviewer comment (Experiments):
The claim that GSiLU enhances the performance of small 3x3 kernels by capturing global spatial information warrants empirical validation. Could the authors clarify if GSiLU doesn't have similar benefits to larger kernel sizes?

Author response:
We use the script in RepLKNet(https://github.com/DingXiaoH/RepLKNet-pytorch/tree/main/erf) to visualize the receptive field, and put these images in the **appendix (page 12)**. For models trained in imagenet-1k, all of them could capture a large receptive field. After adding more convolutions and GSiLU, the features become more discriminative and the final model focuses more on center and edge areas.

Reviewer comment (Evaluation):
The functionality of GSiLU, which zeros out channels with sufficiently negative averages, prompts a request for statistical analysis. Could the authors provide data on the proportion of channels that are effectively zeroed by the GSiLU activation in practice?

Author response:
In the tiny version, the stage one block has 64 channels, we find there is no zero value in these channels, and more than 50% are between 0.3 and 0.7. We think the sigmoid function makes it difficult to generate a zero-value channel.

Reviewer comment (Reproducibility):
As part of a review process, I believe it is expected to validate the findings through an independent examination of the code and model (Hence, my low confidence score.). To maintain the integrity of my review, I request access to the relevant code and model. This will enable me to verify the results personally.

Author response:
All codes and models will be public after being accepted. If it is rejected, we will make this paper, codes, and models public in arXiv and github soon.

Reviewer comment (Experiments):
In Figure 2, putting aside SILU and GSILU, the architecture of SCNN looks very similar to mobilenet and its variants; Could the authors provide results comparison to MobileNet with the same layers, depth and width, but without SILU and GSILU blocks? I am curious whether the improvement is coming from SILU or GSILU.

Author response:
**A1:** The top-1 accuracy of MobileNetV2 with the same settings you mentioned is **81.9% under 4.5G FLOPs**. Notably, by replacing the ReLU activation with SiLU functions, it becomes **82.6% under 4.4G FLOPs**, which is recorded in Tabel 5 line4. It is clear to demonstrate the effectiveness.

Reviewer comment (Experiments):
It claims GSILU leads to rich spatial information, could you provide a receptive field size analysis/illustration when comparing network with or without SILU and GSILU?

Author response:
**A2:** We update the paper and add a visualization of the receptive field to the **appendix (page 12)**. It is clear that for the model trained in imagenet-1k, replacing SiLU with GSiLU makes the features more discriminative, thus the final model focuses more on center and edge areas. Specifically, we use the script in RepLKNet(https://github.com/DingXiaoH/RepLKNet-pytorch/tree/main/erf) to visualize the receptive field.