Paper: The Power of Linear Combinations: Learning with Random Convolutions (ICLR-2024)

Reviewer comment (Novelty):
The idea is trivial and well-known.

Author response:
With all due respect, we remind you of the reviewer's instructions for raising weaknesses: *“Be specific, avoid generic remarks. [...] if you believe the contribution lacks novelty, provide references and an explanation as evidence;”* 


If you believe that our results are well-known, please provide references and explanations as evidence. As a reminder here are our key statements from the conclusion section:

1. Random frozen CNNs can outperform fully trainable baselines in clean and robust accuracy.
2. In modern off-the-shelf networks, learned spatial convolution filters only marginally improve the performance compared to frozen random models in small kernel regimes. The reason for this is an implicit computation of linear combinations between pointwise and spatial filters.
3. Adding learnable pointwise convolutions immediately after spatial convolutions can decrease performance and robustness.
4. Only, under increasing kernel size, learned convolution filters begin to increase in importance but only due to a different spatial distribution of weights that cannot be reflected by currently practiced i.i.d. initializations.

Needless to say, we disagree that these statements are well-known or trivial.

Reviewer comment (Experiments):
The experiments are done on toy models and datasets, which are not convincing.

Author response:
It is not true that we only experiment with toy models and datasets. We ablate the effects on various ResNets scaling linear combinations, width, and depth on CIFAR-10. Following that, we show that the results transfer to multiple other datasets including ImageNet (Sec. 5.2) which is arguably the benchmark in vision, as well as other tasks such as Object Detection and Segmentation in Appendix A.1 containing real-life problems from Kaggle Challenges. Fig. 1 also shows the behavior of popular benchmark CNNs: ResNe(X)t-50 and Wide-ResNet-50. In all of these experiments, we see the same observations - how can the results be unconvincing? 

Please let us know if we have addressed your concerns. If you have any other questions/concerns please feel free to ask. We would be grateful if you would reconsider your score in light of our updates. Thank you again for your time and consideration.

Reviewer comment (Experiments):
Experiments on state-of-the-art models.

Author response:
It is not true that we only experiment with toy models and datasets. We ablate the effects on various ResNets scaling linear combinations, width, and depth on CIFAR-10. Following that, we show that the results transfer to multiple other datasets including ImageNet (Sec. 5.2) which is arguably the benchmark in vision, as well as other tasks such as Object Detection and Segmentation in Appendix A.1 containing real-life problems from Kaggle Challenges. Fig. 1 also shows the behavior of popular benchmark CNNs: ResNe(X)t-50 and Wide-ResNet-50. In all of these experiments, we see the same observations - how can the results be unconvincing? 

Please let us know if we have addressed your concerns. If you have any other questions/concerns please feel free to ask. We would be grateful if you would reconsider your score in light of our updates. Thank you again for your time and consideration.

Please see Tab. 2, where we reached 99.8% of the original ImageNet baseline. On other datasets we even outperform baselines.

Reviewer comment (Evaluation):
My main objection with respect to this manuscript is the clarity of practical impact of this approach. It seems like in most settings, the proposed approach slightly underperforms traditional CNN architectures. For larger expansion rates, the model may outperform the baseline, but to me the trade-off in computational complexity is unclear. Although the author’s findings are interesting in their own right - it is certainly somewhat suprising that a factorization this drastic is still able to perform this impressively - I think the manuscript would greatly benefit from more clear indication of where potential benefits of frozen spatial convolution weights may lie, as the current version of the manuscript doesn’t sufficiently address this in my opinion.

Author response:
Our practical relevance is showing, that not learning the spatial convolutions obtains similar or even better accuracy/robustness in large networks. This comes at the benefit of a significantly reduced parameter count, and thus lower VRAM, faster training etc. 

*Are such large networks realistic?* Generally, we are all aware of the trend for larger models, but even today adversarial robustness SOTA (see https://robustbench.github.io/ under CIFAR-10,  $\ell_\infty$) uses WideResNets-70x16 [1] (16 times wider than a normal ResNet)! We assume that we would obtain cheaper and better training with freezing. Sadly we cannot verify this, due to the necessary GPU VRAM to run experiments.

Just to clarify: generally, we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless (although in some cases it is an easy trick to improve performance - e.g., see Tab. 4 in the Appendix). The LC-ResNets are just proxies for linear combinations in real architectures.

Additionally, we deliver “food for thought”: i) pointwise convolutions after learnable spatial convolutions can impair performance - does this limit current architectures (e.g., ResNet-50)? ii) currently practiced initializations are not ideal when scaling kernel size (as commonly done) - spatial considerations are one thing to consider; iii) any studies of convolution filters must consider the presence of linear combinations.

Does this clarify our intent? Would the reviewer suggest that we update our conclusion with this text?

Reviewer comment (Experiments):
The experiments conducted by the authors in the main body of the paper are quite limited in scope, in that they are all classification problems. Given that the authors themselves indicate that linear combinations of random weights have a hard time forming sharp spatial patterns, I think more attention should also be given to e.g. fine-grained segmentation (I’ve seen the results in the appendix but the authors might consider moving them to the main body and providing a more detailed experimental setup).

Author response:
Due to space limits, we decided to put these results in the appendix (as the trend is similar). Including these results would mean that we have to remove or shorten some other section. Do you have any suggestions on what we should replace?

Regarding the sharp spatial patterns: This only affects larger kernels. Common models such as U-Net with 3x3 are not affected. Besides, “sharp” transitions in masks are generated by the high-frequency information on skip-connections, not the up-convolutions [1]. In fact, the results in the appendix are obtained on a U-Net. 

[1] Olaf Ronneberger, Philipp Fischer, Thomas Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI, 2015.

Reviewer comment (Evaluation):
For the experiments in 5.1, how do the modified models and their baseline counterparts compare in terms of computational requirements? I.e. as you increase the expansion factor it increases computational complexity right? How do the different expansion factors and the original model compare, for example in terms of flops?

Author response:
We deliberately did not report computational requirements, because we do not propose to use the LC-ResNets instead of off-the-shelf architectures (see above). 

Our message is freezing spatial filters reduces training time and does not affect accuracy (or even decreases) it in large networks. Here is an example of common networks. We report the number of trainable parameters (as a proxy for VRAM requirements) and latency (required time for one forward and backward pass). Please note that the inference time will depend on the framework, driver, hardware etc. FLOPs remain unaffected.

| Model     	| Frozen 	| Trainable Parameters 	| Latency |
|-----------|--------|----------------------|---------|
| RN-50     	| no     	| 25.6 M               	| 141 ms  	|
|           	| yes    	| 14.2 M               	| 131 ms  	|
| WRN-50x2  	| no     	| 68.9 M               	| 211 ms  	|
|           	| yes    	| 23.6 M               	| 185 ms  	|
| WRN-101x2 	| no     	| 162.9 M              	| 359 ms  	|
|           	| yes    	| 41.5 M               	| 311 ms  	|

Reviewer comment (Evaluation):
In 5.3 the role of kernel size, you indicate that you investigate the spatial distribution for kernels for different kernel sizes. I assume the visualisations of the kernels shown for the frozen model are obtained after training? You indicate that there is a very clear spatial pattern in trained non-frozen kernels that isn’t present in the frozen model, and you indicate this as the main reason for trailing performance. This seems like a reasonable hypothesis (although somewhat in contrast to Fig 2), but would this mean your model is unable to perform more fine-grained tasks such as segmentation? Why not?

Author response:
Yes, all visualized weights are shown after training.

Fig. 2 shows a smooth resulting kernel - for an example of a “sharp” filter please see Fig. 8 in the Appendix. However, if you had enough filters/LCs, you could reproduce the sharp filters as such there is no contrast to Fig. 2.

Reviewer comment (Novelty):
Could you clarify what you see as the main practical impact of this approach? (see above)

Author response:
Our practical relevance is showing, that not learning the spatial convolutions obtains similar or even better accuracy/robustness in large networks. This comes at the benefit of a significantly reduced parameter count, and thus lower VRAM, faster training etc. 

*Are such large networks realistic?* Generally, we are all aware of the trend for larger models, but even today adversarial robustness SOTA (see https://robustbench.github.io/ under CIFAR-10,  $\ell_\infty$) uses WideResNets-70x16 [1] (16 times wider than a normal ResNet)! We assume that we would obtain cheaper and better training with freezing. Sadly we cannot verify this, due to the necessary GPU VRAM to run experiments.

Just to clarify: generally, we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless (although in some cases it is an easy trick to improve performance - e.g., see Tab. 4 in the Appendix). The LC-ResNets are just proxies for linear combinations in real architectures.

Additionally, we deliver “food for thought”: i) pointwise convolutions after learnable spatial convolutions can impair performance - does this limit current architectures (e.g., ResNet-50)? ii) currently practiced initializations are not ideal when scaling kernel size (as commonly done) - spatial considerations are one thing to consider; iii) any studies of convolution filters must consider the presence of linear combinations.

Does this clarify our intent? Would the reviewer suggest that we update our conclusion with this text?

Reviewer comment (Novelty):
The hypothesis and results are not surprising at all given how many linear combinations the authors learn in replacement of learned spatial filters. We already know that we can learn to reconstruct anything in a space by learning a linear combination of orthogonal basis vectors. While the basis vectors in this case are not designed to be orthogonal, they are randomly sampled from a high-dimensional space (e.g. the kernel space of e.g. 3x3xC, where C is often very large). Randomly sampled vectors in high dimensional spaces are extremely likely to be orthogonal. Furthermore, we don't need to reconstruct perfectly any possible vector in filter space – the filters learned by CNNs likely represent a low-dimensional manifold within filter space, and approximating these filters rather than exact reconstruction is sufficient.

Author response:
The central assumption of the reviewer’s rejection rating regarding the orthogonality does not hold - neither in theory nor in practice! Theoretically (by the law of large numbers), the reviewer's orthogonality assumption only holds for very high dimensional vectors ( $C \rightarrow \infty$ ), but the actual dimensionality of C used in our evaluation (and realistic applications) is far from that. We have $ C \leq 64$ in our experiments.   
To verify this, we have followed your suggestion and computed this on the largest layer in a ResNet-LC20x32 under both, uniform and normal inits. This model breaks even with the baseline - as such it should contain a significant ratio of orthogonality. Yet, even when we conservatively define orthogonality with cosine similarity < 0.2, **0% of the filter pairs are orthogonal** in both cases.

Reviewer comment (Novelty):
Given the fact the hypothesis/results should not be surprising, this paper cannot just be considered an analysis paper pointing out a novel/surprising result to the research community, which is in itself a perfectly valid thing to do. The paper should be motivated outside of the fact that the hypothesis is true — i.e. is it useful to train a CNN in this manner over learning spatial filters? It's obvious that, as explained by the authors in the paper, using frozen LC models that match learned LC generalization performance are much more expensive in terms of compute and perhaps more importantly working memory (i.e. GPU VRAM), so this appears to not be the case at all.

Author response:
Just to clarify: Generally, we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless (although in some cases it is an easy trick to improve performance - e.g., see Tab. 4 in the Appendix). The ResNets-LC are just proxies for linear combinations in real architectures.

Instead, we ask: why would you train all parameters, if not learning the spatial convolutions obtains similar or even better accuracy at a significantly reduced parameter count and lower VRAM, faster training etc.? Our results show that, eventually, the utility of learned convolutions saturates and beyond that deteriorates performance - if networks are sufficiently large. 


*Are such networks realistic?* Yes. SOTA research on adversarial robustness (see https://robustbench.github.io/ under CIFAR-10,  $\ell_\infty$) already uses WideResNets-70x16 [2] (16 times wider than a normal ResNet)! We assume that we would obtain cheaper and better training with freezing. Sadly we cannot verify this, due to the necessary GPU VRAM to run experiments. And, generally, we are all aware of the trend for larger models.


Does this clarify our intent? Would the reviewer suggest that we update our conclusion with this text?

Reviewer comment (Experiments):
Many of the empirical results compare models with drastically different numbers of trainable parameters, while using the same training hyperparameters. However, we know that models with different numbers of trainable parameters usually need different hyperparameters and training setups. For example, models with more trainable parameters typically need longer to converge and require different learning rates, etc. Given this, it's not completely convincing that the model's generalization being compared between frozen LC and learned LC is fair.

Author response:
We have opted for an extra-long schedule relative to the small datasets (200 Epochs) to ensure appropriate convergence. We have ensured that the training accuracy of learnable LC-ResNets at any expansion is near 100%, i.e. they have converged. Further, earlier experiments with shorter schedules (75 Epochs under cosine decay) showed similar trends. Thus we are confident that we appropriately train the models. Tuning is mostly trial-and-error, of course, it may be possible to skew the results in either way - we are confident that using these parameters for all experiments allows for a fairer comparison.

Reviewer comment (Experiments):
No explicit comparison of the compute and VRAM requirements for training models that are compared, in many cases there is quite a drastic difference it would seem. This is especially stark in section 5.1 where models with exponentially increasing compute/parameters are compared with a fixed baseline model, i.e. ResNet-20-16.

Author response:
We deliberately did not report computational requirements, because we do not propose to use the ResNet-LCs instead of off-the-shelf architectures (see above). 

Our message is freezing spatial filters reduces training time and does not affect (or even decreases) accuracy in large networks. Here is an example of common networks. We report the number of trainable parameters (as a proxy for VRAM requirements) and latency (required time for one forward and backward pass). Please note that the inference time will depend on the framework, driver, hardware etc.

| Model     	| Frozen 	| Trainable Parameters 	| Latency |
|-----------|--------|----------------------|---------|
| RN-50     	| no     	| 25.6 M               	| 141 ms  	|
|           	| yes    	| 14.2 M               	| 131 ms  	|
| WRN-50x2  	| no     	| 68.9 M               	| 211 ms  	|
|           	| yes    	| 23.6 M               	| 185 ms  	|
| WRN-101x2 	| no     	| 162.9 M              	| 359 ms  	|
|           	| yes    	| 41.5 M               	| 311 ms  	|

Reviewer comment (Novelty):
While the robustness analysis is appreciated and warrants further attention, this is yet another reason why even when frozen LC models match generalization, they are poorly motivated.

Author response:
We aim to explore the robustness aspect in future work. We have also observed that by using label smoothing we can further amplify the effect. One idea that we can give in this rebuttal is that due to the linear combinations, filters tend to be smoother, and at least for the initial layer this has been linked to improved robustness [1].

[1] Haohan Wang, Xindi Wu, Zeyi Huang, Eric P. Xing, “High-frequency Component Helps Explain the Generalization of Convolutional Neural Networks”, CVPR, 2020.

[2] ShengYun Peng, Weilin Xu, Cory Cornelius, Matthew Hull, Kevin Li, Rahul Duggal, Mansi Phute, Jason Martin, Duen Horng Chau, “Robust Principles: Architectural Design Principles for Adversarially Robust CNNs”, BMVC, 2023.

Reviewer comment (Experiments):
Why is it useful to train a frozen LC model versus a learned spatial model that achieve the same generalization? What is the tradeoff in compute and working memory (i.e. GPU VRAM) utilization for achieving this with frozen LC models versus a baseline learned spatial model for the same generalization performance?

Author response:
We deliberately did not report computational requirements, because we do not propose to use the ResNet-LCs instead of off-the-shelf architectures (see above). 

Our message is freezing spatial filters reduces training time and does not affect (or even decreases) accuracy in large networks. Here is an example of common networks. We report the number of trainable parameters (as a proxy for VRAM requirements) and latency (required time for one forward and backward pass). Please note that the inference time will depend on the framework, driver, hardware etc.

| Model     	| Frozen 	| Trainable Parameters 	| Latency |
|-----------|--------|----------------------|---------|
| RN-50     	| no     	| 25.6 M               	| 141 ms  	|
|           	| yes    	| 14.2 M               	| 131 ms  	|
| WRN-50x2  	| no     	| 68.9 M               	| 211 ms  	|
|           	| yes    	| 23.6 M               	| 185 ms  	|
| WRN-101x2 	| no     	| 162.9 M              	| 359 ms  	|
|           	| yes    	| 41.5 M               	| 311 ms  	|

Reviewer comment (Experiments):
Did you do any hyperparameter tuning for frozen LC versus learnable LC models you compared to ensure using the same hyperparameters/training setup is appropriate?

Author response:
We have opted for an extra-long schedule relative to the small datasets (200 Epochs) to ensure appropriate convergence. We have ensured that the training accuracy of learnable LC-ResNets at any expansion is near 100%, i.e. they have converged. Further, earlier experiments with shorter schedules (75 Epochs under cosine decay) showed similar trends. Thus we are confident that we appropriately train the models. Tuning is mostly trial-and-error, of course, it may be possible to skew the results in either way - we are confident that using these parameters for all experiments allows for a fairer comparison.

Reviewer comment (Experiments):
The experiments in Section 5.2 are not conclusive, which weakens the analysis of the paper that is otherwise very strong up to this point. The reviewer encourages the authors to try and improve this part.

Author response:
Sec. 5.2 (“Scaling to other problems”) shows that our results are not limited to CIFAR-10 but scale to other datasets and tasks. We think it is a very important section - otherwise, readers may be concerned that the findings don’t scale (e.g., see Reviewer gGqG). Could it be that you meant to some other section?

Reviewer comment (Theory):
Do you have any insights with respect to the modus operandi of long convolutional models and how it differs from that of conventional (small kernel) architectures?

Author response:
Long convolutions are more popular in NLP tasks, where it seems reasonable to assume that there is a correlation between distance in the context and relevance, i.e., far tokens are less relevant than closer ones. As such, long convolution kernels often show a decay of magnitude with depth (e.g., see [1]), which is very similar to the large kernel findings on vision we have in Sec. 5.2. Thus we would expect similar implications: I) current init methods won’t allow to us to reconstruct these filters only via LCs from random filters; II) we need better init methods for large/long convolution weights.

[1] Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey, “What makes convolutional models great on long sequence modeling?”, arXiv preprint arXiv:2210.09298, 2022.