Paper: An old dog can learn (some) new tricks: A tale of a three-decade old architecture (ICLR-2024)

Reviewer comment (Novelty):
All the techniques studied in this paper have been proposed in existing works.

Author response:
We do agree with the reviewer. We believe that a retrospective view of existing techniques is valuable and can offer new insights that can be useful to practitioners. For instance, please check the [response to Q1 from reviewer jSdE](https://openreview.net/forum?id=yqAToOgxgf&noteId=qcNLj5PywS) on the surprising components on ViT and MLP-Mixer.

Reviewer comment (Novelty):
The effectiveness of these techniques has been confirmed in existing works.

Author response:
We agree that many of the techniques studied in this work have been popular in the literature. However, there is a *gap between knowledge of useful components and understanding when to use them and whether they would work in other models*. To our knowledge, many of these techniques are used exclusively in contemporary works, making the difference from older models more opaque. Concretely, is there any paper showing that the dated Pi-Sigma (a network proposed over 30 years ago) can perform so well by the addition of those standard techniques? If the reviewer has seen such a paper previously, please let us know and we will adjust our claims.

Reviewer comment (Evaluation):
Even with all the state-of-the-art techniques, Pi-Sigma net cannot outperform even ResNet-50 (with the same FLOPs, see: Table 4). The difference is ~5%, which is extremely large on ImageNet.

Author response:
Let us clarify that our large model does perform favorably to the ResNet50 (given more FLOPs though), please check Table 4. However, the goal of this work is not to claim state-of-the-art behavior. We would like to remind the reviewer that when MLP-Mixer firstly emerged, it could not surpass ResNets. Fortunately though, several brave researchers continued their dedicated effort to improve upon those models and propose improvements on those. That resulted in important improvements in the model design of MLP-based models and has led to several successful models since. Therefore, we do urge the reviewer to consider our contributions as a retrospective analysis that includes several fresh insights offered in the paper.

Reviewer comment (Experiments):
Even though this paper reveals that some techniques are less effective on Pi-Sigma, it may be hard to say that these techniques are not effective on top of other models.

Author response:
We agree with the reviewer. We have made an effort to extend beyond Pi-Sigma, please check sec. D in the Appendix. Having said that, we are open to consider an alternative network if the reviewer has any suggestion for us.

Reviewer comment (Experiments):
The motivation for this article is insufficient. The author aims to provide insights for future model design through experiments, but these findings are all based on the Pi-Sigma model, which may not necessarily apply to future model designs and could even be misleading.

Author response:
> **Q1**: Findings based on the Pi-Sigma model and might not reflect other models. 

We agree with the reviewer that *any* empirical analysis has no guarantees over the design of other models. However, we advocate that our retrospective analysis and insights add a unique value, because our analysis assesses components proposed within the last few years in a model proposed 30 years ago. Therefore, the model is not "tuned" to work well with those components specifically. On the contrary, we offer a detailed account of what works, how much it improves the result and even what does not work.

Reviewer comment (Novelty):
The experiments conducted in this article exclusively employ existing techniques, failing to offer guiding principles for the innovation of new methods in the future.

Author response:
> **Q2**: Guiding principles for future model design

We do believe that our insights can be useful to a practitioner in the field: 

* We do offer an analysis over the trade-offs of components (e.g. accuracy improvement over computational cost of adding them). The cost of those components is less dependent on the architecture. 

* Many of the insights offered in this work are not mentioned in the literature to our knowledge. For instance, the fact that the contribution of each technique depends considerably on the techniques added already. If the reviewer is aware of any previous reference or study in this, we would be happy to include it. 

* Last but not least, the role of skip connections. Even though skip connections are an integral part of model design, their exact role is unclear upon our study. Many works, including ViT, MLP-Mixers are using skip connections by default and do not elaborate on *why* those are included or their role. 

We do believe that those insights along with our findings of reviving an older architecture can have an impact in the community.

Reviewer comment (Experiments):
ViT is currently a widely used model, and it is known for its scalability on data. However, despite the experiments conducted on ImageNet-21k, the author did not include comparative experiments or discussions regarding the ViT model.

Author response:
> **Q3**: Include the ViT model.

We appreciate the remark by the reviewer. We include ViT results in Table 4. Notice that ViT has both more FLOPs and more parameters than our largest model. Additionally, note that the best ViT result reported in the corresponding paper is pre-trained on JFT-300M, which is a proprietary dataset, so a direct comparison with models trained on public datasets might not be fair. We can also try to conduct experiments on the rest of the datasets for a fair comparison. We will update the tables when we have additional results.

Reviewer comment (Experiments):
The author discusses the impact of adding six different techniques to the Pi-Sigma model in various orders. However, in the ablation experiments section, there is no discussion of the individual effects of adding each technique to the Pi-Sigma model separately.

Author response:
> **Q4**: Include the individual effects of adding each technique to the Pi-Sigma model separately.

We are thankful to the reviewer for proposing this important experiment. We have indeed conducted this experiment, where techniques are added in isolation on top of the original Pi-Sigma block trained on AdamW. The results below report the average accuracy over 3 runs (on Cifar10):


|      	| T. 1 | T. 2 | T. 3 | T. 4 | T. 5 | T. 6 | Data augm. |
|----------|------|------|------|------|------|------|------------|
| Accuracy | 52.5 | 42.1 | 50.0 | 44.7 | 41.2 | 49.1 | 39.9   	|

Note that the strongest techniques of Layer Normalization and Deep Network work well in isolation (when trained with AdamW). This is complementary to our key observation that the ordering of adding components can play a key role in the additional performance of the network, as we emphasize in sec. 3.8. 

To provide a more thorough analysis, we also test the components in isolation using SGD as the learning algorithm and the results are exhibited below: 

|      	| T. 1 | T. 2 | T. 3 | T. 4 | T. 5 | T. 6 | Data augm. |
|----------|------|------|------|------|------|------|------------|
| Accuracy | 28.0 | 20.8 | 32.1 | 18.1 | 19.5 | 19.6 | 20.0   |

This experiment has been added in sec. B.6 and highlighted with red in the revised manuscript for visual distinction.

Reviewer comment (Novelty):
Could the author discuss some guiding insights from this research for future model design, particularly for state-of-the-art CNNs and ViTs?

Author response:
> **Q2**: Guiding principles for future model design

We do believe that our insights can be useful to a practitioner in the field: 

* We do offer an analysis over the trade-offs of components (e.g. accuracy improvement over computational cost of adding them). The cost of those components is less dependent on the architecture. 

* Many of the insights offered in this work are not mentioned in the literature to our knowledge. For instance, the fact that the contribution of each technique depends considerably on the techniques added already. If the reviewer is aware of any previous reference or study in this, we would be happy to include it. 

* Last but not least, the role of skip connections. Even though skip connections are an integral part of model design, their exact role is unclear upon our study. Many works, including ViT, MLP-Mixers are using skip connections by default and do not elaborate on *why* those are included or their role. 

We do believe that those insights along with our findings of reviving an older architecture can have an impact in the community.

Reviewer comment (Novelty):
Could the author explain why they chose to use Pi-Sigma and highlight any unique aspects or clear advantages it has over CNNs and ViTs?

Author response:
> **Q4**: Unique aspects of Pi-Sigma over CNNs and ViTs.

Our experiments exhibit a concrete advantage of the revised Pi-Sigma when compared to contemporary models. The experiments in sec. C indicate that the proposed revised Pi-Sigma has significant benefits with respect to robustness in ImageNet-A and ImageNet-C. Robustness is a topic of ever-increasing significance as neural networks are deployed in real-world applications. At the same time, the understanding of robustness is not yet complete with even core insights such as depth or width in feedforward networks being unclear in all learning regimes [1]. Therefore, we do believe that this benefit of the proposed model might result in further insights into the robustness of less traditional models. 

Beyond the aforementioned advantage, let us emphasize that the goal of this work is not to design a network with a certain advantage, but rather to highlight how dated architectures can be "resurrected" using very standard components. Pi-Sigma is a prominent example of those architectures and in fact Pi-Sigma is mentioned in standard textbooks [2] as an example of a multilayer network, therefore making it an abandoned but known to the community usecase. 


# # References

[1] Zhu, et al. “Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)”, NeurIPS’22. 

[2] Mehrotra, et al. “Elements of artificial neural networks”. MIT press, 1997.

Reviewer comment (Experiments):
I think the authors only try out fairly popular techniques for the experiments, for which each of their individual effects is fairly well-studied and well-known. It would have been a much more interesting paper if the authors also included other low-level techniques that such models could try out.

Author response:
We agree with the reviewer that these are popular techniques, we also mention this explicitly in our paper as well. However, we do not share the belief that their effects are well-known. For instance, despite the widespread adoption of skip connection, its role is still not conclusively understood as we also mention in sec. 3.8. In addition, in our [response in Q2 for reviewer jSdE](https://openreview.net/forum?id=yqAToOgxgf&noteId=qcNLj5PywS), we showcase how we could reduce the impact of the skip connection in the network. 

We advocate that there are fresh insights even in existing components: for instance, that the contribution of each technique depends considerably on the techniques added already. If the reviewer is aware of any previous reference or study in this, we would be happy to include it. Why is this significant? Practitioners that design new models can be directly impacted by this observation. 

Regarding additional techniques to try out, we also agree with the reviewer. **We are open to suggestions on techniques to try out** and explore their effect on our framework.

Reviewer comment (Experiments):
I believe a very important question is to understand what kind of techniques do not seem to improve performance, it would be of great interest to understand what kind of techniques are possibly over-reliant or dependent on certain architectures. I would also encourage the authors to include this important detail in their work.

Author response:
We are thankful for the question. We do agree that the list of techniques with negative results is important. In fact, a list of the techniques that we tried and failed to provide any improvement exists in sec. F. Surprisingly, techniques once popular, e.g., dropout or dropblock, fail to provide any improvement in our case. We do believe that this list can be further augmented, we are willing to try out suggested techniques.

Reviewer comment (Writing):
A typo in "Out work aims".

Author response:
We are thankful to the reviewer for the attentive study of our work. The typo has been fixed.

Reviewer comment (Novelty):
Although the analysis is useful to put it out there, I'm not sure if it was surprising at the end. As the current architectures of MLP mixer or ViT indeed have very few components, thus making it easy to say what is important vs not. I would like authors to counter this point and provide examples that they discovered that might not have been that obvious.

Author response:
> **Q1**:  The current architectures of MLP mixer or ViT indeed have very few components. 

Let us explain why both MLP-mixer and ViT rely on *a lot of components* and in fact the contribution of such components is not well-studied. Concretely: 

* In MLP-Mixer that the reviewer mentions, they use Adam as an optimizer for the pre-training (which is the most important contributor for their final accuracy) and never question the impact of the optimizer. There are a number of additional details that are important. For instance, the pretraining on JFT-300M, which is a proprietary dataset. We do believe that assessing the components in isolation can offer fresh insights. The pre-training implementation details as printed from the paper are [here](https://imgur.com/a/mzYBf4Q). 

* In ViT, many of the aforementioned components are still valid. They use Adam for pre-training and JFT-300M for pre-training.

In addition, both the MLP-Mixer and the ViT have plenty of skip connections. Even though the presence of skip connections is prevalent across different architectures, to our knowledge, the role of skip connections is still not conclusively settled as we point out in sec. 3.8. 

Therefore, we do believe that the true value of the components can be obfuscated by the use of a lot of implementation details. This is precisely why we believe that exploring the components in an isolated network can provide a more valuable guide for their generalization.

Reviewer comment (Experiments):
Further since training algorithm (AdamW) in this case plays a very important role, it would be helpful to analyse it more closely by checking previous versions of Adam such as RMSprop or even before.

Author response:
> **Q3**:  Analyze more closely the role of the optimizer.

We are thankful to the reviewer for the suggestion. We already included a version using pure SGD in the original submission (typically the first column/row of results in ablation studies such as in Fig. 2 and Tables 7, 13). However, we conduct a dedicated study where all components remain in-tact apart from the optimizer. The results with various versions of Adam, along with SGD and RMSProp are indicated below (when trained on Cifar10): 


|      	| AdamW | Adam | AdamP | SGD  | Lars | RMSProp |
|----------|-------|------|-------|------|------|---------|
| Accuracy | 88.0  | 84.5 | 84.7  | 37.5 | 29.4 | 87.8	|

We have added the related results in sec. B.8. Once again, we are thankful to the reviewer for strengthening our observations.

Reviewer comment (Experiments):
Same for data augmentation, a more in-depth study regarding this would be useful.

Author response:
> **Q4**: Analyze data augmentation more in-depth. 

We appreciate the suggestion. We have conducted the following ablation study: We assess the performance when one of the augmentation methods is removed and we obtain the following results: 

|      	| No Auto augment | No Mixup | No smoothing | No random erase | Baseline |
|----------|-----------------|----------|--------------|-----------------|----------|
| Accuracy | 86.0        	| 84.6 	| 85.4     	| 85.4        	| 88.0 	|

Reviewer comment (Experiments):
Can authors compare on the benchmarks on which Pi-Sigma paper had shown their result? This will help realise if the improvements are indeed benchmark specific or general.

Author response:
> **Q5**: Is it possible to reproduce the original experiments of Pi-Sigma?

To our knowledge the dataset of “underwater SONAR transient data from the DARPA data set” is not available anymore. If the reviewer has any link to that, we would be happy to reproduce the results. 

However, beyond the architecture, we also believe that the probabilistic learning rule that the original paper used might insert another complication into this comparison. We would be happy to discuss this further if the reviewer believes this is very important.

Reviewer comment (Experiments):
What is the role of convolutions vs MLP mixer. I think the standard convention is that if you have less data you should do convolution, however this is not clearly indicated from the author's experiments.

Author response:
> **Q6**: Role of convolution vs MLP-mixer in the limited data regime. 

We do agree in principle that having less data requires additional inductive bias in order to guide the learning of the network. However, our point here is precisely to emphasize that various components added to the architecture, e.g., layer normalization or data augmentations, alter significantly the inductive bias of the network and result in a substantial improvement in the performance. We do believe this is an understudied topic that warrants a careful analysis from the community. 


We are thankful for the time and effort of the reviewer. If there are any further questions, we are more than happy to clarify.