Paper: CoMNet: Where Biology Meets ConvNets (ICLR-2024)

Reviewer comment (Experiments):
While there are several relevant comparison systems — especially ResNeXt, which uses group convs — there are no comparisons to many more recent CNNs that may also be relevant. Searching on paperswithcode.com for CNNs on ImageNet yields many from the past couple years. Not all of these are necessarily relevant comparisons, but some likely are: the current leaderboard #1 is RevCol which also uses the term "column" extensively, though in a different way, while MogaNet uses similar parameter counts.

Author response:
Thank you for bringing these papers to our knowledge. Definitely, these papers work in the area of CNN. However, they are different from our motivation that we intend to develop a macro ConvNet design which is *Pure CNN* i.e. no attention mechanisms or distributed information paths causing high branching. *RevCOL*, it is fundamentally different, where a column refers to a deep network which makes the final prediction. Multiples of these deep networks are used, which receive the same input and produce independent final predictions and any two columns have different weights. On the other hand,  in CoMNet, multiple columns do not produce the final output. There is only one output end of CoMNet in contrast to the RevCOL. Then If we look closely at *MogaNet*, it has several branches due to different-sized kernels and a very complex information path, resulting in lower latency. Although it happens to be the reason for its competitive results, we believe that CoMNet is a much purer form of CNN which can also be improved similarly to MogaNet by inheriting the fundamentals of CoMNet design e.g. different kernel-sized columns.

Reviewer comment (Theory):
In Sec 4.7, the sentence "The idea of IR in ResNeXt is more similar to us however, the major difference is that ResNeXt has multiple blocks per stage, and each performs replication" could be spelled out more, as this sentence leaves the similarities vague. ResNeXt uses group convs, but IR is not mentioned in that paper. However, IR followed by group conv is the same as a single regular convolution ($group\_conv(IR(x)) = conv(x)$), so if ResNeXt performs $group\_conv(f(conv(x)) = group\_conv(f(group\_conv(IR(x))))$ where $f$ is normalization+activation, then indeed there is a close link, and stacking more group convs is a major difference.

Author response:
Thank you for the observation. Indeed, there is a close link, but the CoMNet structure is different, as pointed out by you, i.e. *stacking more group convs is a major difference.* Since we try to follow biological observations, we have included IR to be consistent with the theory. IR followed by a group convolution is indeed a plain convolution; however, this is true only if there is one convolution. This does not apply to the entire stack in the CoMNet stage. So in CoMNet, the first 3x3 convolution after IR can be replaced by a plain convolution, but the other convolutions in the column stack have to be group conv. Another difference is that in ResNext, 1x1 convolution is used for this purpose, however, in ConvNet, it is a 3x3 conv.

Reviewer comment (Experiments):
The discussion comparing to group convolutions is good, particularly the second point, but it could be expanded further and perhaps referred to more throughout the paper. If a "CCM" is implemented by group conv + residual, then the fact that there are stacked group convs without between-group recombination could be brought more to the forefront and studied explicitly. Questions to explore include: What happens when varying $l$ to be smaller, so that there is a 1x1 recombination more frequently, even for $l=1$ (and more blocks); or alternatively, introducing 1x1 recombinations after every group conv? Is this model then the same as ResNeXt? If so, then this is a critical difference. It would be good to nail this down more, going incrementally from ResNeXt-like operations to CCM.

Author response:
Indeed l=1 and introducing 1x1 recombinations after every group conv makes a CoMNet bottleneck same as ResNeXt. This is what differentiates CoMNet from ResNeXt. There is a stack of 3x3 and only 8 1x1 layers are used in a CoMNet variant (CoMNet-C2) that is equivalent to ResNeXt-50 with 32 1x1 layers. To verify what happens after frequent 1x1 combinations, we experimented as shown in *Sec.C, Ablation study, Varying-l* of the paper. 

Based on your comments, we further elaborate the text which better shows differences with ResNeXt.

Reviewer comment (Writing):
In Sec 4.7, the phrase "no blocks, only stages" is unclear. It is not evident what constitutes the notion of a "block" and why comnet does not have it, since it does have residual connections between each of the $l$ conv applications (analogous to "blocks"). Clarification is needed on what is meant by "no block" here and why it is advantageous, and whether it has to do with how often there is cross-column combination.

Author response:
Traditionally, a ConvNet block is referred to as a bottleneck which consists of a stack or a combination of different convolutions. Indeed CoMNet have a residual connection between the immediate 3x3 convs, however, it is different from the traditional designs that skip connection is formed across a stack of more than one convolution. For this reason, the earlier VGG also had only stages instead of blocks or bottlenecks. In the same fashion, in Comnet, there is the stage which consists of a stack of convolution, thus in CoMNet, blocks or stages can be used interchangeably.

Reviewer comment (Theory):
In Sec 2 and 4.6, the term "exponential param growth" is unclear. What does this refer to and how exactly is it exponential? Increasing depth produces linear parameter growth, while increasing width yields quadratic growth.

Author response:
By this statement, we mean that as we go deep down, the deeper layers have a large number of channels since the width is increased. As if increase the number of repetitions of deeper blocks, together, the increased depth and the increased width causes drastical changes in the parameters. For example, ResNet-50 has roughly 14M parameters in only the final 3 blocks of stage-5 whereas the remaining 10M parameters out of 25M total parameters come from the remaining 13 blocks of the network. 

We'll add this explanation into the text.

Reviewer comment (Evaluation):
In Table 2, it is unclear if the numbers are correct. None of the comnet models show a difference in parameters or FLOPs for plain vs SR, yet there is still a reported speed increase.

Author response:
Yes. They are correct, and in fact this is an interesting fact about the CoMNet in contrast to the RepVGG. That RepVGG have different paths and different kernels (shortcut, 3x3 and 1x1 in parallel) at the train time, which are fused by a single convolution at the test time via SR. Whereas in CoMNet, we only have one 3x3 Convolution and identity, therefore we do not encounter any parameter change during SR. The speed improvement comes from the fusion of identity or shortcut into the 3x3 Conv via SR which shows how branching can affect speed.

Reviewer comment (Presentation):
In Fig. 1, the abstract mentions parnet next to the reference to Fig. 1, but parnet is not in this figure.

Author response:
We will incorporate the readings of ParNet into Fig.1

Reviewer comment (Writing):
In Fig. 2d, the label "Conv, b=M" includes a parameter "b" that is not described anywhere; it may be the number of groups in a group conv, but this needs clarification.

Author response:
Yes. It is a typo. 'b' should be 'g'. Thank you for pointing this out.

Reviewer comment (Novelty):
Just as the authors mentioned, most of the ideas inspired from the visual cortex have been used in ConvNets, such as weight sharing, shortcut, groups convolution, etc. For each design point of this article, it is not novel. Generally, the novelty of this paper is weak.

Author response:
We appreciate your concern. However, it must be noticed that despite weight sharing, shortcuts, and group convolution are prominent in CNNs, many essential cortex properties are still missing from the CNN designs. Our paper is the first to club almost all of them in one architecture and present multi-dimensional efficiency in CNNs. The major novelty from the previous design is using 3x3 convolutions by majority and following the columnar approach. Also, avoiding the need for bottleneck designs which leads to only stage and no blocks concept in CoMNet. Together, these improvements over the existing models are major novelty which helps comnet achieve multidimensional efficiency, including transformers.

Reviewer comment (Theory):
Inspired from the biological function, there are many frameworks to simulate the structure. The authors and previous works provide different solutions, and the designed modules are based on some hypothesis which are not proved.

Author response:
We agree with your comment and this also applies to well-established CNN models and Transformers. At least at some point, these models draw inspiration from biology but these inspirations are not well connected. However in this paper, we concretely revisit the old works in the area of visual cortex, and try to align the modern ConvNet design process with these ideas. Despite neuroscience has advanced so much, however, a concrete pin-pointed architecture of cortex is still not available in literature. This forces the network design researchers to draw their own conclusions and designs which try to mimic the biological cortex. In the same sense, we do not claim or bring anything new about the cortex, instead borrow simple design ideas of cortex, and try to develop a simple CNN architecture which works well on multiple fronts. We verify the same via a number of experiments.

Reviewer comment (Experiments):
The experiments only compare with partial works, all works should be compared in different level, such as efficientnet is only exist in R1 and Parnet only in R4.

Author response:
The reason behind this choice is the resolution. For example, only efficientnet-b0 model is trained and tested at 224x224 resolution which is a standard. Other variants of efficientnet are trained at higher resolution via compound scaling which improves the accuracy due to the availability of more information. On the other hand, we presented our model only at 224x224 which is a very general trend, and we do not study compound scaling on comnet as it would be a separate line of work.

While in terms of ParNet, these models are big. In order to compare with them, we kept the accuracy as a constant and chose suitable comnet variants which can match in accuracy with ParNet. It can be seen that CoMNet is extremely fast than the recent ParNet series of work.

Reviewer comment (Evaluation):
The results in Table 1 is doubtful, why R0, R1, R2 and R3 have the same latency? From my experience, it is impossible.

Author response:
The results on R0-R3 are absolutely certain and correct. The confusion arises from the fact that it is a general trend to increase the depth to achieve higher accuracy. However in CoMNet, we change the number of modules and slightly the depth. By increasing the number of modules, we are essentially modifying the width. Moreover, since depth is roughly the same, the whole computations of CoMNet remains packed into layers thus there are more parallel computations with high computation density in each layer of CoMNet and less serial operations due to smaller depth. This causes similar latency in different CoMNet variants despite having different capacities.**

Reviewer comment (Presentation):
In Table 2, what is the method that combines CoMNet with Structural Reparameterization?

Author response:
We simply fuse the identity connection in CoMNet unit similar to RepVGG and now run the inference. Fusing the identity connection with 3x3 conv is essentially the SR operation.

Reviewer comment (Experiments):
The article only compares earlier works, with the most recent being from 2021. I believe that comparing CoMNet with some more recent methods could enhance the quality of the paper.

Author response:
The reason for this being that our CoMNet is a pure CNN design i.e. free of advance information paths or attention mechanism. We compared the most recent works uptill 2022 (ParNet) in pure CNN realm.

Reviewer comment (Experiments):
The comparison with Vision Transformer (ViT) could be included in the main body of the paper instead of the appendix. Additionally, some of the latest backbones could be used for ViT.

Author response:
We will move the results to the main text.

Reviewer comment (Experiments):
The paper could also benefit from conducting experiments in more areas, such as segmentation and detection. Such experiments might further enhance the quality of the paper.

Author response:
We agree with you. Since this is a new design with multiple concepts, we tried to arrange our experimental setup such that it can be understood easily by the readers instead of incorporating many areas into one paper.