Paper: Graph is All You Need? Lightweight Data-agnostic Neural Architecture Search without Training (ICLR-2024)

Reviewer comment (Novelty):
I think the paper needs more justification about the importance of the training-free data-agnostic NAS.

Author response:
Neural architecture search finds candidate architectures, trains them to obtain their accuracy and uses the accuracy as feedback to modify its search strategy. NAS is computationally expensive and slow and training the architecture consumes the majority of the time. Training-free NAS techniques developed proxy scores in lieu of accuracy. This avoids the need to train the architecture altogether and the training-free metric is computed on a randomly initialized model. These metrics are relatively fast to compute. For instance, our average degree takes only 1 CPU second per architecture. 
The candidates are now ranked using these proxy scores instead. Thus, training-free NAS accelerates the search drastically. 

To save further time, a lot of these training-free NAS algorithms compute the proxy scores only on a few mini-batches of the data. This score might not generalize well across the entire dataset and has a high variance based on the mini-batches passed. To address all the above mentioned issues, we developed a data-agnostic training-free NAS method.

Reviewer comment (Experiments):
The experimental results are evaluated on only simple benchmarks. To show the practicality of the proposed method, I think it would be more helpful if you provide a comparison with other non-training-free NAS methods such as DARTS-PT on a larger search space such as DARTS space.

Author response:
In NAS, the search algorithm searches for a candidate architecture and uses its validation accuracy to determine the candidate’s efficacy. All the training-free metrics  are used in lieu of accuracy. Instead of ranking the candidate models based on the accuracy and selecting the best model found so far, one could use the proxy scores. These training-free metrics can be combined with any existing NAS algorithm. Thus it is essential that all these training-free metrics have a very high correlation between the ranking of the candidates by their score and the ranking by the validation accuracy. We already reported the correlation metrics on more complex search spaces such as ENAS, PNAS, DARTS etc.  in Table 4 of our paper. 

NDS-DARTS is a NAS Benchmark developed for DARTS search space. The accuracy of the best architecture found by our metric is 93.23% . NASWOT got 90.6% on the same search space. On NAS-Bench-201 search space, DARTS-PT reported a test accuracy of 88.11% (table 4 of the paper [2]). Our method using average degree yielded an architecture with 90.2% test accuracy.

Reviewer comment (Novelty):
The ultimate goal of NAS is to find the optimal architecture in any way possible. Even if the proposed method has a high correlation with model performance, it is uncertain whether it will be significantly helpful in quickly finding the best architecture. This is because one must train selected architectures to evaluate the actual performance and verify whether it is a good architecture, and it seems that most of the time will be spent in this verification process during the NAS procedure. Can you kindly and elaborately explain why training-free NAS is an important issue, along with practical scenarios?

Author response:
In NAS, the search algorithm searches for a candidate architecture and uses its validation accuracy to determine the candidate’s efficacy. The goal of the algorithm is to find the architecture with the best test accuracy. Each candidate architecture is trained and then its validation accuracy is used as the feedback for the search algorithm. NAS is very computationally expensive, requiring every candidate architecture to be trained. To alleviate this problem, training-free NAS was introduced [3, 4, 5].

The training-free metrics are used in lieu of accuracy. The training-free metric is computed on a randomly initialized model and is generally fast to compute. Instead of ranking the candidate models based on the validation accuracy and selecting the best model found so far, the candidate models are now ranked using this training-free metric. 

In particular, our average degree metric just randomly initializes the candidate architecture and computes the average degree of the model. This is extremely fast and takes only a ~1 CPU second for each model.

Reviewer comment (Theory):
It appears that NASGraph assumes the inputs to the neural architecture and subsequent graph blocks is non-negative. Does the analysis hold even with non-negative inputs? Many of modern neural architectures utilize activation functions that could yield negative inputs/outputs (e.g., gelu activation). Can NASGraph generalize beyond relu-based architectures?

Author response:
When there is non-negative input to the graph block involved, we need a threshold to create non-negative edge weights in our graph. When the input is larger than the threshold, we consider it as active. Otherwise, we consider it as inactive. Therefore, our method can be applied to GeLU activation as well.

Reviewer comment (Experiments):
In a similar vein, can the proposed method generalize to non-conv-based architecture spaces? Such as ViTs and MLPMixers?

Author response:
Given that we convert a neural network to a graph and our proposed framework only considers the input/output tensor, in principle, our method is applicable to most search spaces. However, owing to the lack of publicly available NAS benchmarks for those search spaces, we cannot evaluate it on those search spaces right now.

Reviewer comment (Presentation):
The authors define the final score of neural architectures (the average degree of the graph measure) in Section 4.2 under the performance evaluation section. I think it would be more appropriate to move this definition to somewhere towards the end of Section 3 because by the end of Section 3, the reader is left hanging without the knowledge of how NASGraph actually ranks neural architectures.

Author response:
We appreciate this valuable suggestion. We will modify section 4.2 accordingly: the average degree is now introduced at the bottom of Section 3.

Reviewer comment (Evaluation):
In Section 4.2, the authors mention that they explore other graph measures as well. Any idea why the average degree works best out of the compared graph measures? Also, intuitive explanation to what each one of these graph measures actually indicates/implies would be helpful.

Author response:
Intuitively, each graph edge indicates if the previous graph block (defined in the context of the NASGraph) contributes to the current graph block. An extreme case is that the graph block is Zero operation (used in NAS-Bench-201). In this case, the previous graph block does not contribute to the current graph block. There will be no graph edge connecting these two graph blocks. The performance of such an architecture is expected to be inferior. In short, the more the number of edges (higher average degree), the better the performance of the corresponding architecture would be. To further illustrate this, in Section A.12 of our paper, we plotted the average degrees of the architectures with the highest and the lowest accuracy in the NASBench 201 benchmark. As expected, the architecture with the highest accuracy has a very high average degree compared to the other one. 

In addition to the average degree, we explore other graph measures such as wedge, resilience, and density. We refer the reviewer to Section A.8 in the appendix.

Reviewer comment (Presentation):
I went through the example in Figure 1 and the caption and the example itself is still unclear to me. Could the authors please elaborate on this?

Author response:
As detailed in Sec. 3, we group neural layers (or operations such as conv, bn and relu) as graph blocks as shown in Figure 1 (a). Then, we do one single forward propagation to determine if the previous graph block contributes to the current graph block in a channel-wise manner. If there is a contribution, we build a graph edge. Otherwise, there is no graph edge. Each channel of the input tensor (or the output tensor) is aggregated to a graph node. This way, we are able to build a new type of graph given an architecture. The connectivity between graph nodes is determined by the forward propagation, i.e., outputs of graph blocks. We do a neural architecture search by comparing graph measures.

Reviewer comment (Experiments):
I find the evaluation of the method quite weak especially since the authors do not compare against the MeCo proxy https://openreview.net/pdf?id=KFm2lZiI7n. On an initial glance it seems that MeCo outperforms the proposed proxy on most of the benchmarks. The code for MeCo is publicly accessible and I encourage the authors to compare their work with MeCo in terms of the correlation metric and search time.

Author response:
We appreciate the suggestion of making comparisons to MeCo, a recent publication just accepted to NeurIPS 2023 and to be presented in December 2023. However, we want to bring to the reviewer’s attention that this is an unfair ask, because our ICLR submission was made back in Sep. 2023. To our best knowledge, the earliest public publication record of MeCo is Sep. 21st 2023, on NeurIPS’s Openreview system (we did not find any arxiv version). This situation makes it impossible to compare to MeCo at the time of our submission, not to mention that MeCo should be considered as a concurrent work. Despite the impossibilities, we follow the reviewer’s suggestion to include the comparison in the rebuttal and the updated manuscript. But we hope our review rating will not be negatively and unfairly judged by not having comparisons to a concurrent work that is only publicly available around the ICLR submission deadline. 

We compared our method with MeCo on  NAS-Bench-101, NAS-Bench-201, Trans-NASBench 101, NDS-DARTS and NDS-ENAS. While it is true that MeCo has competitive performance on NAS-Bench-101 and NAS-Bench-201 as reported in the MeCO paper, we observed that MeCo does not generalize well on other benchmarks. As stated in [1], [2], there is no one proxy that performs the best across all the search spaces. We compared MeCo on the Trans-NASBench 101 search space and evaluated it on the NDS-DARTS and NDS-ENAS search spaces, as those search spaces were already reported in our paper. We used one A40 NVIDIA GPU and one AMD EPYC 7232P CPU to run Meco. Our method runs only on a CPU and does not require a GPU.

Trans-NASBench-101 results:

|                                   | surface_norm | class_scene | room_layout  | Average Running Time  |
|-------------------------|-----------------|---------------|-----------------| --------------------------- |
| MeCo                        |      0.65           |        0.62       |       -0.25        |     6.88 GPU hours        |
| MeCo opt                  |      0.67           |        0.64       |        0.26        |     6.90 GPU hours        |
| Avg degree (ours)     |      0.66           |        0.70       |        0.37        |     2.93 CPU hours        |
| Wedge (ours)            |      0.72           |        0.74       |        0.42        |     3.04 CPU hours        |

It is important to note that while most metrics perform well on NASBench 101 and 201, their performance drops on Trans-NasBench-101.  We used the publicly available Meco code to evaluate it on NDS search space and the correlation is given in the table below.

|                                |     NDS-DARTS*       |   NDS-ENAS*   | Average Running Time |
| --------------------- | ----------------------- | ----------------- | -------------------------- |
|  MeCo                    |            0.31              |         0.19          |        36.50 GPU hours   |
| Avg degree (ours)  |            0.62              |         0.57           |         3.34 CPU hours    |

*There is no performance reported on NDS search space, so we ran experiments using the publicly available code and computed the correlation. We did encounter some bugs while trying to run Meco’s code, but we fixed it to the best of our knowledge. 

[1] https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/  
[2] NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies, Krishnakumar et al.

Reviewer comment (Experiments):
I currently find the search spaces to be very limited to cell-based spaces or benchmarks. Since recently there have been efforts to scale Neural Architecture Search to weight-entangled spaces like AutoFormer [1], OFA [2] and HAT [3], it would be great to evaluate the method on these spaces. Note though these spaces don't have a tabular benchmark for evaluation, they do provide a pre-trained surrogate model for fast evaluation.

Author response:
As the reviewer rightly pointed out, there are no publicly available benchmarks for AutoFormer, OFA and HAT. While OFA does have a surrogate model, the error introduced by the surrogate model would definitely hamper us from determining the true correlation. AutoFormer and HAT do not even have surrogate models to predict the accuracies given an architecture. We do not have the computational power to evaluate 1000 architectures in these search spaces to compute the correlation.

Reviewer comment (Experiments):
Since there has been growing interest in transformer spaces, is the proxy search space agnostic and directly applicable to transformer spaces? This was unclear to me from the paper.

Author response:
In principle, our method is applicable to any neural architecture as it only considers the input/output tensors. The connectivity between graph nodes is determined by the forward propagation. Thus it can be run on the transformer search space too but we don’t have the computational power to evaluate 1000 architectures on Imagenet to compute the correlation. We understand the great interest in extending our method to transformer spaces. This is beyond the scope of this paper and is considered as one of the promising future works based on our study.

Reviewer comment (Experiments):
Check weaknesses: Could the authors compare against MeCo (https://openreview.net/pdf?id=KFm2lZiI7n) and could the authors evaluated on weight entangled macro-architecture spaces like AutoFormer [1], OFA [2] and HAT [3]?

Author response:
We appreciate the suggestion of making comparisons to MeCo, a recent publication just accepted to NeurIPS 2023 and to be presented in December 2023. However, we want to bring to the reviewer’s attention that this is an unfair ask, because our ICLR submission was made back in Sep. 2023. To our best knowledge, the earliest public publication record of MeCo is Sep. 21st 2023, on NeurIPS’s Openreview system (we did not find any arxiv version). This situation makes it impossible to compare to MeCo at the time of our submission, not to mention that MeCo should be considered as a concurrent work. Despite the impossibilities, we follow the reviewer’s suggestion to include the comparison in the rebuttal and the updated manuscript. But we hope our review rating will not be negatively and unfairly judged by not having comparisons to a concurrent work that is only publicly available around the ICLR submission deadline. 

We compared our method with MeCo on  NAS-Bench-101, NAS-Bench-201, Trans-NASBench 101, NDS-DARTS and NDS-ENAS. While it is true that MeCo has competitive performance on NAS-Bench-101 and NAS-Bench-201 as reported in the MeCO paper, we observed that MeCo does not generalize well on other benchmarks. As stated in [1], [2], there is no one proxy that performs the best across all the search spaces. We compared MeCo on the Trans-NASBench 101 search space and evaluated it on the NDS-DARTS and NDS-ENAS search spaces, as those search spaces were already reported in our paper. We used one A40 NVIDIA GPU and one AMD EPYC 7232P CPU to run Meco. Our method runs only on a CPU and does not require a GPU.

Trans-NASBench-101 results:

|                                   | surface_norm | class_scene | room_layout  | Average Running Time  |
|-------------------------|-----------------|---------------|-----------------| --------------------------- |
| MeCo                        |      0.65           |        0.62       |       -0.25        |     6.88 GPU hours        |
| MeCo opt                  |      0.67           |        0.64       |        0.26        |     6.90 GPU hours        |
| Avg degree (ours)     |      0.66           |        0.70       |        0.37        |     2.93 CPU hours        |
| Wedge (ours)            |      0.72           |        0.74       |        0.42        |     3.04 CPU hours        |

It is important to note that while most metrics perform well on NASBench 101 and 201, their performance drops on Trans-NasBench-101.  We used the publicly available Meco code to evaluate it on NDS search space and the correlation is given in the table below.

|                                |     NDS-DARTS*       |   NDS-ENAS*   | Average Running Time |
| --------------------- | ----------------------- | ----------------- | -------------------------- |
|  MeCo                    |            0.31              |         0.19          |        36.50 GPU hours   |
| Avg degree (ours)  |            0.62              |         0.57           |         3.34 CPU hours    |

*There is no performance reported on NDS search space, so we ran experiments using the publicly available code and computed the correlation. We did encounter some bugs while trying to run Meco’s code, but we fixed it to the best of our knowledge. 

[1] https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/  
[2] NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies, Krishnakumar et al.

Reviewer comment (Experiments):
How does the cuda memory consumption and search time of the method compare against other proxies?

Author response:
In the manuscript, we explicitly claim that we only need CPU for the computation. So there is no cuda memory consumption. In table 6, we report the running time needed for the random search.

Reviewer comment (Reproducibility):
Reproducibility is still am important challenge in the NAS literature. For reproducibility, could the authors make the code publicly available?

Author response:
The code has already been shared publicly. The link for the git repo is listed in the supplementary material.