Paper: CARENET : A NOVEL ARCHITECTURE FOR LOW DATA REGIME MIXING CONVOLUTIONS AND ATTENTION (ICLR-2024)

Reviewer comment (Novelty):
Limited novelty. While the paper brings forward a new model design, its foundation lies in leveraging existing techniques. As a result, its contribution is incremental and not enough for establishing a distinct and well-designed model architecture.

Author response:
We appreciate the reviewer's perspective on the novelty of our CARENet architecture. We acknowledge that CARENet leverages existing techniques; however, we believe that the innovation in our work lies not solely in the creation of entirely new methods, but in the novel integration and optimization of these techniques to address specific challenges.
The development of CARENet was the result of extensive experimentation and a thorough review of existing literature. Our primary objective was to design an architecture that achieves comparable performance to current state-of-the-art models but with significantly fewer parameters and a smaller model size. This goal was driven by the practical need for efficient models in environments with limited computational resources, such as embedded systems like Jetson cards.
CARENet represents a careful and deliberate balance between efficiency and performance. The architecture is tailored to function effectively in low-memory environments, a critical consideration for a wide range of real-world applications, especially in the context of Industry 4.0 and IoT devices.
To further emphasize the novelty and practical significance of CARENet:
1.	Targeted Design for Specific Environments: We will elaborate in the manuscript on how CARENet is specifically designed for environments with limited memory, detailing its application potential in embedded systems.
2.	Comparative Analysis: We will enhance our comparison section to clearly articulate how CARENet stands apart from existing architectures in terms of its balance between model size, parameter count, and performance.
3.	Empirical Evidence: We plan to include additional empirical evidence to demonstrate the effectiveness of CARENet in practical scenarios, further establishing its value in the field.
In summary, while CARENet builds upon existing methods, its contribution lies in the unique combination and optimization of these techniques to create an architecture that is not only efficient but also effective in resource-constrained environments. We believe this presents a valuable contribution to the field, especially considering the growing demand for lightweight yet powerful models in various applications.

Reviewer comment (Experiments):
The experiment comparisons are not convincing enough. The performances are similar to ResNet18 on most datasets and does not show significant superiority. Moreover, the compared methods do not include more modern architectures, such as Swin-T [1], ResNeXt [2], ConvNext [3].

Author response:
We appreciate the reviewer’s feedback on the comparisons in our experiments. We understand the importance of benchmarking our CARENet architecture against not only established models like ResNet18 but also more modern architectures such as Swin-T, ResNeXt, and ConvNext.
Our primary focus in developing CARENet was to achieve a balance between model efficiency (in terms of parameter count) and performance, particularly in low-data regimes. To this end, CARENet has been designed with a total of approximately 12 million parameters. When comparing this with the parameter counts of the mentioned models:
●	Swin Transformers: Approximately 27.5 million parameters
●	ResNeXt: Approximately 25 million parameters
●	ConvNext: Approximately 393 million parameters
It becomes evident that these models have significantly larger sizes compared to CARENet. The primary intent of CARENet is to provide a more parameter-efficient alternative while maintaining competitive performance, especially in environments where computational resources are a constraint.

Reviewer comment (Experiments):
Lacking ablations. The paper could be further strengthened with an ablation study. This would offer clarity on how specific components of the proposed design contribute to performance enhancements.

Author response:
We acknowledge the value of the reviewer's suggestion for an ablation study in our research. We plan to conduct a detailed ablation study on CARENet, removing or altering specific components to gauge their impact on performance. This will help us understand the contributions of elements like the self-attention layer, architectural choices, and any unique integration or optimization techniques. The results will offer clearer insights into CARENet's functionality and its efficacy in image classification tasks, particularly in low-data contexts. The findings will be included in our revised manuscript, adding an important dimension of empirical validation to our work.

Reviewer comment (Experiments):
The evaluation of the model on only three small datasets does not provide a robust validation of its capabilities. A more extensive evaluation, including additional and larger datasets, would be more convincing.

Author response:
Thank you for your valuable feedback concerning the scope of our model evaluation. We understand your concern about the robustness of the validation provided by the limited number of datasets used in our study.  
In our manuscript, we reported the evaluation of CAReNet on four datasets: STL10, Tiny ImageNet, Fashion MNIST, and MNIST. Despite the limitations, we endeavored to demonstrate the effectiveness of CAReNet across a variety of datasets, including those with small and medium-sized images. This was aimed at showcasing the architecture's versatility and performance in different scenarios, especially in low data regimes. However, we acknowledge the importance of evaluating our model on larger and more diverse datasets to further validate its capabilities. To this end, we have planned to extend our evaluation to include more comprehensive datasets like ImageNet-1K.   
Additionally, we are in the process of requesting access to a cluster equipped with 100 NVIDIA A100 80GB GPUs, which would significantly enhance our ability to conduct more extensive and rigorous tests. We also plan to make the weights of our model publicly available once these additional evaluations are completed. This would allow the broader research community to replicate our results and further explore the potential of CAReNet in various applications.   
The choice of these initial datasets was primarily influenced by the computational resources available to us. As a research team, we had access to only a single machine equipped with an RTX 3090 GPU, which constrained the scale of our experiments.

Reviewer comment (Evaluation):
The current evaluation presented in Table 3 does not convincingly demonstrate the model's effectiveness. For example, the marginal improvement over ResNet18 on the MNIST dataset, despite a higher parameter count, calls into question the practical benefits of the proposed method.

Author response:
Thank you for your comments regarding the performance comparison presented in Table 3 of our manuscript. Your observation about the marginal improvement of CAReNet over ResNet18 on the MNIST dataset, especially considering the higher parameter count, is well-taken and merits further explanation.  

Firstly, it is important to note that while CAReNet shows a modest improvement over ResNet18 on the MNIST dataset, the primary objective of our architecture was not just to maximize performance on a single dataset but to demonstrate consistent and robust performance across a variety of datasets. In this regard, as mentioned in our paper, CAReNet achieves an average Top-1 validation accuracy of 82.25% across different architectures, compared to 80.85% for ResNet18. This indicates a more consistent and reliable performance of CAReNet in a wider range of scenarios, which is a significant factor in its practical utility.  

Furthermore, CAReNet's slightly higher parameter count is offset by its versatility and efficiency in handling diverse datasets. The integration of convolutional and attention mechanisms in CAReNet is designed to offer a balanced approach to feature extraction, enabling it to adapt to various types of image data more effectively than traditional architectures like ResNet18. This adaptability is crucial in practical applications where datasets can vary widely in terms of size, complexity, and content.

Reviewer comment (Experiments):
For work of this nature, it is crucial to assess performance on larger-scale datasets to ensure the model's effectiveness and generalizability.

Author response:
Thank you for your valuable feedback concerning the scope of our model evaluation. We understand your concern about the robustness of the validation provided by the limited number of datasets used in our study.  
In our manuscript, we reported the evaluation of CAReNet on four datasets: STL10, Tiny ImageNet, Fashion MNIST, and MNIST. Despite the limitations, we endeavored to demonstrate the effectiveness of CAReNet across a variety of datasets, including those with small and medium-sized images. This was aimed at showcasing the architecture's versatility and performance in different scenarios, especially in low data regimes. However, we acknowledge the importance of evaluating our model on larger and more diverse datasets to further validate its capabilities. To this end, we have planned to extend our evaluation to include more comprehensive datasets like ImageNet-1K.   
Additionally, we are in the process of requesting access to a cluster equipped with 100 NVIDIA A100 80GB GPUs, which would significantly enhance our ability to conduct more extensive and rigorous tests. We also plan to make the weights of our model publicly available once these additional evaluations are completed. This would allow the broader research community to replicate our results and further explore the potential of CAReNet in various applications.   
The choice of these initial datasets was primarily influenced by the computational resources available to us. As a research team, we had access to only a single machine equipped with an RTX 3090 GPU, which constrained the scale of our experiments.