Paper: CNNGEN: A GENERATOR AND BENCHMARK FOR SUSTAINABLE CONVOLUTIONAL NEURAL NETWORK SEARCH (ICLR-2024)

Reviewer comment (Experiments):
Why the performance of networks should be diverse? Note the goal of NAS is to find architecture better than human crafted ones, while human designed architecture perform quite well in nearly all cases. The importance of NAS to distinguish models with similar high performance. For instance, NAS is designed to distinguish ConvNeXt and ResNeXt, rather than ConvNeXt and LeNet. The previous benchmark that has most architectures with very close and high accuracy actually make more sense than the proposed ones.

Author response:
Regarding weakness 1, we refer to our comment to reviewer (vffX): we want to evaluate and design predictors in a general way: these predictors should also be able to rule out bad architectures, which is not the case for the NAS benchmarks which all the topologies have similar good performance.  We agree intending to design (in the long term) better architectures than human-crafted ones.  Our motivation is that it extends to the whole topology and not only to parts (cells) of a given topology (i.e., architecture like LeNet). Therefore, our approach allows us to explore the CNNs’ search space in the large rather than in the small for a predefined topology.

Reviewer comment (Theory):
DSL has limitation in the expressivity of the architecture. The proposed DSL seems only supports sequential architecture without branches (Fig 3 and Fig. 8).

Author response:
Regarding weakness 2, this is not correct. Our grammar can generate architectures with an infinity of branches in theory. In practice,  we constrain the topological space with at most 2 branches per node for a given network. It is true that we only showed linear ones in the original paper for the sake of simplicity, our mistake.  We have added in Appendix A a few more complex architectures generated in our benchmark. You can also see in the Zenodo in the folder output/architecure_img 1300 architectures that our generator synthesizes.

Reviewer comment (Theory):
Intuition of predictor. Why do you think a CNN can know the network's performance by looking at the image of its architecture? What is the intuition behind it? For me, it just overfits a small dataset.

Author response:
Regarding weakness 3, the intuition behind using CNN to predict the network performance is that the image contains the topology and hyperparameters and this information is necessary to predict performance. 

We trained each predictor 10 times (see Section 6.1) and reported an average of the performance (cf code in Zenodo). There is also a CSV file that contains the results of each run.

Reviewer comment (Evaluation):
Why do we need an accurate number of energy consumption? Why not just model parameters and FLOPs? Energy consumption is very sensitive to the setup of machines and can be easily outdated. A model that consumes many energy might be very energe-saving in the next year due to the new software support and hardware update.

Author response:
Regarding question 1, it is common to report energy consumption as it characterizes the environmental impact of these models. We did not see these concerns widely covered in existing benchmarks, which motivated providing a new one (in addition to network diversity). We do agree that it varies depending on hardware and software optimization. We rely on the code carbon library, which reports operations as well and could help design novel predictors in the future.

Reviewer comment (Theory):
The extracted five key concepts used to describe architectures (architecture, featureExtraction, featureDescription, classification) are still limited, not sure why this would lead to better diversity in neural architectures.

Author response:
Thank you for your review. 

Regarding your first question, the five concepts are only the top-level ones in our grammar that represent the different phases of processing in a CNN. They were found based on our analysis of the literature and the function of the different layers (or groups of layers) from usual DNNs. In addition, we want to highlight that, as described in Section 4.1, we decompose these concepts into other ones to describe the network’s functions (convolution, pooling, batchNormalisation, etc.). We added the full Xtext Grammar in Appendix B. It contains 16 terminals and 17 non-terminals.

Reviewer comment (Reproducibility):
It's unclear how the author choose the five key concepts used to describe architecture.

Author response:
Thank you for your review. 

Regarding your first question, the five concepts are only the top-level ones in our grammar that represent the different phases of processing in a CNN. They were found based on our analysis of the literature and the function of the different layers (or groups of layers) from usual DNNs. In addition, we want to highlight that, as described in Section 4.1, we decompose these concepts into other ones to describe the network’s functions (convolution, pooling, batchNormalisation, etc.). We added the full Xtext Grammar in Appendix B. It contains 16 terminals and 17 non-terminals.

Reviewer comment (Novelty):
I feel the method is just trying to use natural language to replace the tradtional symbolic respresentation in NAS search spaces. I'm sceptical of why the proposed method could generate more diverse architectures.

Author response:
Thank you for your review. 

Regarding your second question, we do not use natural language but a formal context-free grammar defining the language of possible CNN architectures. Therefore, it is also a symbolic approach. We achieve diversity by assembling network functions sequentially while respecting grammar rules. Our generator achieves diversity as it does not start with a predefined topology but combines the network functions sequentially while respecting the grammar rules. Thus, it can form combinations of multiple architectures in one network or completely new ones.  This diversity results in the varied CNN accuracies as depicted in Figure 1b.