Paper: Brain-inspired $L_p$-Convolution benefits large kernels and aligns better with visual cortex (ICLR-2024)

Reviewer comment (Evaluation):
In the Sudoku challenge, the main argument seems to be about the imbalance between row/column and block accuracy, but the difference in accuracy is <10% in all cases, which doesn't strike me as a particularly worrisome imbalance. In particular, the difference between p=2 and 7x7 (Large) is <1% (although I'm not sure what the latter model exactly is; see below). Could this be a matter of presentation and the differences would be much clearer if you looked at error rates instead of accuracy? Could you explain why you think these results are important given such small effect size?

Author response:
Thank you for the reviewer's critical point. We understand the reviewer's point on the accuracy gap ($\sim$10%; row/column vs square) may not seem worrisome with respect to the imbalance itself. However, this is slightly different from our intention. We would like to clarify our main claim.

Our main claim here is that ___such a small imbalance can have a significant impact on a greater problem___. Considering the fact that a Sudoku puzzle is solved only when all 81 numbers align correctly, a single error in any of the 18 rows/columns ($\approx$6% error) results in a Sudoku unsolved. We believe our experimental result in __Fig. 3b__ supports this point; small imbalances in row/col and square accuracies (1$\sim$10%) can have a significant impact on overall Sudoku accuracy (20$\sim$60%). 

To convey this point more clearly, we have revised our manuscript by changing the figure, and subsection titles in Section 4 as follows.
- (Fig. 3 title) $L_p$-convolution enhances Sudoku solving efficiency by effectively balancing accuracy between square and row-column puzzles
- (subsection 3 title) $L_p$-convolution in Sudoku solving: balancing square and row-column imbalances

Reviewer comment (Experiments):
In the image classification experiments, there is a clearly significant improvement due to Lp-Conv, but not on all architectures and again the improvement is small (a few percent). Given that architectural modifications alone can now push accuracy on CIFAR-100 >90% (https://arxiv.org/abs/2304.05350v2), the 2–3% improvements in the 60–70% range feel a bit insignificant. Can you explain why you think your approach is still worthwhile? Would we expect similar gains if we included the Gaussian masking inductive bias into more modern architectures?

Author response:
Thank you for sharing the impressive paper. While architectural modifications alone can achieve remarkable performance gains, these results can be seen as the outcome of a combination of various excellent methods. ___Our work can be considered as the discovery of a new method that can be combined with such architectural modifications.___ From this perspective, we believe our work is sufficiently valuable.

In response to the reviewer's question we applied $L_p$-Conv (p=2) on the ConvNeXt-V2 (Woo et al., CVPR 2023) and other higher accuracy range (70~80%) models with CIFAR100 as follows (See __Appendix A.20, Table 5__):

|$L_p$-Conv|ConvNeXt-V2-T|ResNet-50|ResNeXt-50|DenseNet-121|
|:-:|:-:|:-:|:-:|:-:|
|-|64.26±0.41|73.17±0.23|73.55±0.57|74.12±0.16|
|$\checkmark$|65.58±0.25|76.66±0.19|77.38±0.36|77.14±0.18|

Our method enhances performance in 1) modern CNN architecture as well as 2) higher accuracy models with 3~4% improvement. These results highlight the versatility of our $L_p$-convolution method to a diverse range of architectures. We believe that integrating our Gaussian masking inductive bias into various architectures could yield similar gains, further affirming the importance of methodological innovations alongside architectural advancements.

Reviewer comment (Evaluation):
In the representational similarity experiments the improvements due to p=2 are in the range of 0.5–2%. Again, why are such small differences relevant from a scientific point of view if the differences between, e.g., AlexNet and ConvNeXt-T are of the same order of magnitude?

Author response:
Thank you for your thoughtful question. In the field of computational neuroscience, the use of artificial systems as a comparative tool has proven invaluable, particularly in understanding the brain's visual processing [1, 2]. Although AlexNet and ConvNeXt-T show similar levels of improvement, the reasons behind these gains are quite intricate, especially when considering structural differences. Our research focuses on unraveling these complexities. Specifically, the consistent improvement pattern across various models with the adjustment of the $p$ value is compelling. ___It suggests a shared underlying mechanism in visual information processing___, potentially mirroring the Gaussian-like receptive fields observed in the human brain. I hope this answers your question!

[1] Kriegeskorte et al., "Representational similarity analysis - connecting the branches of systems neuroscience," Frontiers in Systems Neuroscience, 2008.  
[2] Cadieu et al., "Deep neural networks rival the representation of primate IT cortex for core visual object recognition," PLoS Computational Biology, 2014.

Reviewer comment (Experiments):
Are the masks applied after training to the trained weights or during training? If the latter, does it change anything about what the network learns? It could just learn larger weights W where the mask is small, leading to the exact same network as without masking. Why is this not happening?

Author response:
In response to the reviewer's question, we clarify that the $ L_p$-mask is trained with weight simultaneously. With proper initialization of $L_p$-mask, ___it imposes a spatial constraint on the weights which leads to a change in learning dynamics___. Since both mask and weights are trained with standard backpropagation after the forward process ($Y = \phi(X*(W \odot M))$, Eqn. 4), $L_p$-mask effectively scales down the influence of peripheral weights.  This setup leads to a distinct learning behavior compared to a scenario without a mask, as the mask evolves (Fig. 2, 4, and 8) together with weight update.

Reviewer comment (Presentation):
It is not clear to me what the different networks in Fig. 3 are. It's clear that 3x3 and 7x7 refer to the kernel size. But what does "Large" refer to and what is the Lp^{\dagger} model?

Author response:
Apologies for the insufficient information. We have revised the manuscript's Fig. 3b to address this.
- (new in Fig. 3b caption) '(3x3)' or '(7x7)' denotes the size kernel. '$L_p^\dagger$' denotes parameters of $L_p$-mask is frozen. 'Large' denotes a simple enlargement of the kernel, without a mask.

Reviewer comment (Reproducibility):
As far as I understand Fig. 3, p=2 refers to p being initialized as 2 but then optimized as a trainable parameter, correct? If so, what was p at the end of training for the three models with p={2, 16, 256}? This question actually applies to all experiments.

Author response:
Thank you for your question regarding the optimization of $p$. 
- In the case of $L_p^\dagger (p=256)$, the parameters are fixed, meaning that $p$ remains constant at 256 throughout the training process.
- For the $p=16$ scenario, we have detailed the properties of $L_p$-masks in the revised manuscript, specifically in __Appendix A.18__. 
- Additionally, the distribution of $p$ values across all experiments is comprehensively visualized in __Appendix A.17__ of the revised manuscript.

Reviewer comment (Experiments):
Table 1: ResNet has larger kernels than 3x3 in the first layer. Are they kept the same in all variants and only the 3x3 kernels in the later layers are modified?

Author response:
All kernels are ___equally modified___.
- In TinyImageNet, ResNet has a 7x7 kernel size in the first layer and is equally enlarged as 15x15 in '(Large)' or '$L_p$-Conv'. 
- In CIFAR-100 due to its small image size (32 x 32), ResNet has a 3x3 kernel size in the first layer [1, 2].  This is enlarged as 7x7 in '(Large)' or '$L_p$-Conv'.  
[1] He, Kaiming, et al. "Deep residual learning for image recognition." CVPR. 2016.   
[2] https://github.com/weiaicunzai/pytorch-cifar100/blob/master/models/resnet.py

Reviewer comment (Presentation):
Fig. 5: I did not fully grasp what "Max. SSM" exactly refers to. In panel a) five brain areas are colored; in panel b) five network architectures. Presumably you compared all architectures against all brain areas. What exactly is being reported here? Why is there not one such plot as in b) per brain area?

Author response:
"Max. SSM" refers to the ___maximum SSM value obtained from our pairwise comparisons___ of all network architectures against each brain area, following the methodology established by Bakhtiari et al. in 2021. As you correctly interpreted, these comprehensive comparisons are detailed in __Appendix A.12, Figure 10__ of our manuscript. Due to limited space, we included this analysis in the Appendix to provide a clear presentation of our result.

- Bakhtiari, Shahab, et al. "The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning." Advances in Neural Information Processing Systems 34 (2021)

Reviewer comment (Theory):
Why is the spatial distribution of synapses the right thing to compare to weights in a CNN? The presynaptic neurons also have spatially extended receptive fields (RFs), which means the V1 neuron's RF envelope in pixel space is not the same as the retinotopically mapped synaptic locations.

Author response:
Thank you for your insightful feedback. We understand the limitations in our approach, particularly that the spatial distribution of V1 synapses might not perfectly mirror the weights in a CNN. The idea of comparing retinotopically mapped synaptic locations is an intriguing alternative. Our current method, aligning V1 synaptic patterns with CNN weight distributions, is an ___exploratory effort aimed at fostering better dialogue between the realms of biological and artificial vision___. Though it may not be a direct equivalence, this approach marks a significant step in our journey to understand the intricate parallels between these systems.

Reviewer comment (Novelty):
I am left wondering if CNNs do this RF masking already, by different means.

Author response:
Thank you for your insightful query regarding receptive fields in CNNs.

You are correct that many CNN studies discuss RFs, similar to what we've illustrated in our revised manuscript, specifically in __Appendix A.14, Figure 11a__. In traditional CNN models, these RFs are shaped by the network's trainable parameters – the kernel weights – and evolve naturally during the training process.

However, our work takes a distinct approach. ___We've integrated structured sparsity constraints directly into the network's architecture___. This method doesn't just allow the RFs to emerge; it actively shapes them. This targeted shaping is crucial because it ensures the development of receptive fields that adhere to a specific pattern of structured sparsity. This approach is more than a byproduct of training; it's a strategic design choice.

We believe that this novel method of guiding RF development represents a significant advancement in the field. It opens new avenues for understanding and utilizing receptive fields in CNN architectures, moving beyond traditional methods. Our hope is that this contribution will offer valuable insights into the potential of structured sparsity in neural networks.

Reviewer comment (Writing):
The paper is quite hard to read. Especially the sections introducing $L_p$ convolution (sections 2 and 3) need to be written with much more clarity to make them more accessible. Currently, there are issues such as an abundance of notations, symbols being introduced after their first use, etc. that make it difficult to understand exactly how $L_p$ convolution works and models the spectrum from biologically resembling to artificial receptive fields.

Author response:
We appreciate your review. We have revised Section 2 and Section 3.
First, we have simplified the content by excluding notations that were previously unused, and have added clear explanations for all notations that are used. (e.g., we remove $\varphi$, and we add description for $C$ and $W_i$) 
Second, we clarify the relationship between RF and $L_p$-convolution in the paragraph below Equation 3.

Reviewer comment (Evaluation):
I don't find the similarity of $L_p$ convolution with small p with mouse visual receptive fields to be very convincing. First of all, this seems like a qualitative comparison and is not an objective way to measure similarity to biological receptive fields. It also seems from Appendix A4 (Fig 7A) that the mouse functional synapses in V1 lack any visible structure representative of selectivity to low-level visual features. This is quite concerning as it makes one wonder whether these neurons are selective to low-level features as one would expect from V1 neurons.

Author response:
Thank you for your insightful feedback regarding the comparison of $L_p$-convolution with small $p$ to mouse visual RFs. We appreciate your concern about the qualitative nature of this comparison and the observations from Appendix A4 (Fig 7A).
In response to your comments, we would like to clarify our approach and make some adjustments to our manuscript for better clarity. ___We realize that our definition of "receptive fields" might have been misleading.___ Our definition aligns more with the engineering perspective prevalent in CNNs, especially for AI/ML audience at ICLR. Our adjustments in the manuscript are as follows:

__Clarify the definition in the main text (Section 2):__ To clarify our definition of RF, we promoted footnote 1 to the main text, with bold highlighted at local connectivity patterns.
- (new): While the concept of RF generally encompasses sensory-level inputs and adjacent layers, in this paper, we specifically refer to __RF as local connectivity patterns__ between neurons in adjacent layers.

__Clarify the description in the appendix (Figure 7 legend):__ To further clarify, we have inserted a note in the figure legend.
- (new): The receptive field discussed in this figure specifically refers to the spatial connectivity patterns of synapses. Note that this differs from receptive fields typically associated with low-level visual feature selectivity.