Paper: Towards Neural Architecture Search through Hierarchical Generative Modeling (ICLR-2024)

Reviewer comment (Experiments):
CIFAR-10, CIFAR-100 and Imagenet are benchmarks which the community has collectively overfit on. If time permits do you think you could provide results on a different modality, e.g. an NLP task?

Author response:
Since generalizability was a common concern among reviewers, we decided to include experiments on NB360. We hope this should be more than enough to satisfy this request. Please see the common reply summarising new experiments.

Reviewer comment (Theory):
As stated in their limitations, the method largely relies on ZC proxies for both G-VAE and the generation of the macro architecture.

Author response:
Recent work in zero-cost metrics has shown that zero-cost metrics-based NAS can produce promising performance networks compared with the training-based NAS approach. Admittedly, the zero-cost metrics performance can vary when ranking models from different tasks and different search spaces. However, in our generator, we are not strictly based on the assumption that model performance ranking strictly follows the zero-cost metrics ranking, but ‘similar zero-cost metrics will lead to relatively similar performance’. This is different from existing zero-cost metrics NAS.

Reviewer comment (Experiments):
How do you estimate the number of components for the GMM?

Author response:
To decide the number of components for the GMM we increase their number, starting from 16, and calculate the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). We stop when they plateau.

Reviewer comment (Reproducibility):
Why is the cost of the ES for training the SG such a major bottleneck when using ZC proxies? Did you parallelize ES?

Author response:
Please note we report GPU-hours, i.e., cost and not wallclock time — to avoid confusion, we will change the term used in the paper accordingly. This cost is subject to parallelization, following common practices. Since all steps can be parallelized trivially, we can achieve almost linear speedup if we use multiple GPUs, e.g., with 4 GPUs we can finish everything within ~8h.

Regarding the relative cost of ES to the rest of the system. The bulk of the cost comes from calculating proxies for models, so it is going to be directly proportional to the amount of “labelled examples” mentioned in Tab. 5 (compared to calculating proxies, operating on graphs is very fast, e.g., when training a G-VAE or running a metrics predictor). As such, getting the training data for G-VAE and the SG should take similar time (50k vs 60k), but note that the micro model uses only up to 64 channels (Eq. 2) and our macro search space allows for <=1024, so the average model is much larger in the macro stage.

Finally, we did not dedicate a lot of time optimising our code and there are some opportunities to achieve non-negligible speedups. To give an example, when computing the vector of proxies we currently compute each of: NASWOT, SNIP-SSNR and FLOPs completely independently from each other, resulting in 3 forward passes and 1 backward pass per model, but it clearly should be possible to optimise it such that only 1 forward and 1 backward is needed per model. This would reduce the time needed for data generation by up to 2x.

Reviewer comment (Novelty):
The proposed method still heavily depends on human prior knowledge (e.g., reference design, zero-cost metric). I do not see any improvements in improving NAS's applicability to less studied domains.

Author response:
Regarding the reference design, we would like to point out that in our experiments we used cluster centres for that, which are decided automatically and do not require any prior knowledge. Although we discussed the possibility of improving results further if a good reference point was known, we did not rely on that when running our method. Therefore, we believe the reviewer’s point is reaching too far in that aspect.

Regarding zero-cost proxies, we agree we rely on them, which involves certain risks, as mentioned in the limitations. Still, the proxies are a way of probing into the characteristics of neural networks, like #params or #flops; if anything, we would consider our approach of clustering cell designs according to ZC proxies to be rather automated and require very little human input. We strongly disagree with the implied statement that building on top of the existing knowledge should be considered a weakness.

Having said that, we understand the reviewer is also making a point about the applicability of our method to less-studied tasks. While we disagree with the points above, we do agree with concerns about generalizability, considering our original submission focused on well-studied tasks only. To address this, we have run our method on a suite of tasks from NB360, please find the details in the common reply summarising new experiments.

Reviewer comment (Experiments):
The generalization of the proposed method is a big question. As far as I know, the generalization ability of zero-cost metrics is a bit limited. As the proposed method is based on zero-cost metrics, I think it is necessary to verify the proposed method's generalization ability.

Author response:
Please see the results on NB360 in the common reply.

Reviewer comment (Novelty):
The idea of using a hierarchical design to reduce the space is not novel.

Author response:
We never claim the idea of using a hierarchical design is novel on its own — if the reviewer got an impression that’s the case, please let us know which part of the paper made them think so and we will revise it. The novelty of the paper lies in designing a hierarchical generative system, which is the first of its kind, to the best of our knowledge.

Reviewer comment (Experiments):
I feel the current experiment design is not a good fit for this paper. I am more interested in seeing experiments on less studied domains instead of these standard benchmark datasets. If the author can show one practical case where their method can clearly outperform conventional NAS, this paper will be much stronger.

Author response:
Please see the results on NB360 in the common reply.

Reviewer comment (Novelty):
The design conflicts the target pain point. The target pain point raised in the introduction is to eliminate the need of designing search space manually. However, this paper still manually designs a search space and develop an algorithm upon it, see page 4.

Author response:
Thank you for the time and effort you have invested in reviewing our manuscript. We greatly appreciate your insights and acknowledge the points you have raised. Regarding your concerns, we have provided additional clarification as follows and we are committed to enhancing our experiments and manuscript, hope that the revision will satisfy your concern.

> The design conflicts the target pain point.

We apologise for not making our point sufficiently clear — this is a side effect of limited space, so please let us explain our reasoning in more detail.

First, it is impossible to have an operational codebase that would be able to train models without defining its ramifications. This means any NAS system requires a search space that is designed manually — even if this search space is implicit — because of the need to write functioning code that would be able to execute networks. This is a fundamental issue and we do not see a way around it. We want to make this point clear because the way in which the reviewer raised their point (“this paper still manually designs a search space and develops an algorithm upon it”) directly touches upon it; there is no way to perform NAS differently than designing a search space and developing an algorithm upon it, any deviations from this rule are likely to be nothing more than semantic nuances (e.g., what exactly is called a search space).

Instead, what is usually understood behind the term “manually designed” search space is the amount of manual effort that is needed to construct a search space in which good models can be found fast enough. The trade-off between search space size and time needed to search is again fundamental — we can easily think about a search space of all possible models that can be expressed in, e.g., Pytorch, but searching in such a search space is going to be extremely difficult. Consequently, every NAS method performs some kind of regularisation of this huge search space by considering only its subset, to make the problem tractable. The question becomes: how big can the subset be and if there are any constraining factors. This leaves us with a spectrum, where a method can be considered more “automated” if it retains strong searching performance without limiting the number or the qualitative properties of models in the search space. We would argue that our method advances in both aspects. On one hand, our search space is extremely expressive, to the best of our knowledge the largest among any works that achieve competitive ImageNet performance. On the other hand, we do not impose constraints such as the ability to construct a supernet (with the size of our search space, we would run out of GPU memory very fast if we tried that), etc. As such, the effort to design our search space is rather minimal — we simply include as many operations from the literature as possible, expose as many parameters (channels, stride) as possible and only make sure any network can be executed correctly. Considering all that and the fact that we can obtain strong performance fast, we would argue it is justified to say our method is “more automated” than the existing ones and that it helps minimise manual effort (we will revise our paper to make it clear it is not our goal to completely eliminate manual design, since this cannot be done).

Reviewer comment (Experiments):
The proposed method seems not generic and time-consuming. The proposed methods require a transformer generator to produce macro architectures which seems require time-consuming pretraining, fine-tuning, and may be task and search space specific.

Author response:
>The proposed method seems not generic and time-consuming. The proposed methods require a transformer generator to produce macro architectures which seems require time-consuming pretraining, fine-tuning, and may be task and search space specific.


Regarding genericness, we admit our initial experiments did not necessarily make a strong point behind this aspect. However, with the addition of NB360, we hope this concern is now alleviated.

Regarding time, in our opinion, the amortised cost of 30 GPUh puts us in the regime of very efficient NAS methods. We are not sure why the reviewer says it seems time-consuming. Sure, it might not be the very fastest NAS method but it is still very efficient, especially if we consider the size of our search space. In case this was missed, please note: 1) it is GPU hours, not days (more commonly reported), 2) this is further subject to parallelization (every step is trivially parallelizable) - with 4 GPUs we can finish everything within ~8 hours.

The only really time-consuming element could be the mentioned pretraining of the GPT model, but we think it is somewhat questionable if this should be considered a direct cost of our method (see our response to reviewer vDUp for more details).

If we disregard the cost of pretraining the GPT (since it is not related to NAS in any way) and consider updated results, it is clear that our method is very efficient and not exactly task-specific. With a single pre-training costing us 30 GPU hours, we are able to conduct 13 different searches (C-10, C-100, IN-1k with 3 budgets + 8 tasks from NB360), advancing state-of-the-art on well-established tasks and matching/advancing on 4/8 understudied tasks, all at the same time. To the best of our knowledge, this is a clear advancement compared to the existing methods.

Reviewer comment (Evaluation):
The method uses a pretrained GPT-Neo-125M model. Hence the actual search cost also implicitly inclues the pre-training cost of this model, which should ideally be added to the search cost computation (table 1,2,3).

Author response:
We agree there is a hidden cost in needing a pre-trained GPT model - we will include a note about that in the paper for the sake of completeness (note that the authors of GPT-Neo did not disclose the time needed to train their models, so we will rely on how costly it is to train similar ones). However, we are not sure if a straightforward attribution of this cost to our method is completely fair. GPT models are trained to perform NLP tasks and there is no clear relation between those and NAS. This is in contrast to some NAS methods that might require starting from, e.g., an open-sourced pretrained supernet - in this case, the connection to performing NAS is clear and including supernet pretraining cost is justified. The point is: if we consider GPT training cost to be part of the search cost, it gives a false impression of having to pay it each time we search for a new model (which is clearly not the case); even if we consider it a pre-cost, it gives a false impression that it has to be paid each time we want to extend/change our NAS settings (such as including more models in the search space, etc.) - again, this is not the case. As such, we believe it should be classified as “pre-pre-cost”, but there is no clear consensus within the NAS community on how (or even if) such cost should be reported, as far as we know. In particular, many existing methods might have paid analogous cost (i.e., with similar level of indirection) but did think about reporting it. Of course, because of that we are happy to engage in discussion with the reviewer to work out the best way of doing so - please let us know your thoughts about our arguments and the current way in which we include the cost in our paper.

Reviewer comment (Experiments):
Study is limited to convolutional spaces. It becomes natural to question the robustness of the proxies to the recent transformer baeed search spaces in NAS

Author response:
We agree questioning the robustness of proxies is justified, and we also mentioned that in the paper when we talk about the limitations of our work (not only in the context of transformers). As such, extending and testing our method to work with transformers is very sensible. Unfortunately, given the amount of additional experiments, the extra work needed to do so is beyond our current capabilities. For the time being, we will include it in the limitations of our work. However, please note that the core methodology proposed in the paper should be easily compatible with transformers.