Paper: Encodings for Prediction-based Neural Architecture Search (ICLR-2024)

Reviewer comment (Experiments):
The authors point out that "GCNs are prone to an over-smoothing problem", although really this issue affects Graph Neural Networks (GNNs) in general. The authors then attempt to validate the efficacy of FLAN's predictor using the DGF and GAT in Table 1. These experiments are unconvincing, as GCN was proposed in 2016 and since then other GNN-types like GAT, GIN, GATv2, etc., have demonstrated superiority over GCN, so it is unsurprising that for FLOW, GAT > GCN. It is also unclear whether the DGF module actually solves the oversmoothing mechanism at all; a simpler explanation is that adding skip-connections and more weights leads to better performance. Thus, results in Table 1 are inconsistent and do little to alleviate this concern, and it seems the paper is undecided if it focuses on correcting GNN problems or neural predictors in NAS.

Author response:
`it is unclear whether the DGF module used actually solves the oversmoothing mechanism at all.`

You are indeed right to point out that this issue affects GNNs in general, and that DGF does not necessarily solve the oversmoothing mechanism. We completely agree with this, and instead use empirical evidence to state that the skip connection in DGF may aid performance, and GAT ensemble seems to help. We believe there are 2 places where we make these claims that are not fully supported, we have now edited the text to remove these parts. Our paper’s key focus is on encodings for accuracy predictors within NAS, and our notes on predictor design were mainly done as a step to create a predictor to test out the performance of different encodings.

Reviewer comment (Novelty):
In discussing Unified Encodings for transferable NAS prediction, the authors cite Multi-Predict and GENNAPE but appear unaware of earlier related work such as CDP, which is older than both cited works. The form of encoding in this paper is weaker than related work for three reasons: (1) there is no guarantee an architecture encoding will be unique, as two different architectures could have similar ZCP scores/latency/FLOPs/etc. but differ structurally; 'score-based' would be a better descriptor for this encoding. (2) The use of "unique numerical indices" limits FLAN to a small number of predetermined search spaces, preventing it from addressing the need to "validate NAS improvement without incurring the large compute cost of performing NAS on a new search space." (3) The statement about global graph search being intractable is weak, as large search spaces have not been an impediment even for early NAS methods, and it is still possible to perform search within cell-based NAS-Benchmarks with transferable encodings like CDP or GENNAPE.

Author response:
`there is no guarantee an architecture encoding will be unique as it possible for two different architectures to have almost the same ZCP scores/latency/FLOPs/etc. but be different structurally. '`

We completely agree. In fact, MultiPredict solely used ZCPs to encode architectures. These encodings do not provide uniqueness guarantees. This is one of the key reasons why, throughout our paper, we use these as supplemental encodings. These supplemental encodings are used in conjunction with the adjacency and operation matrices to build our accuracy predictors. 

`the use of "unique numerical indices" limits FLAN to a small number of predetermined search spaces `

The training time for our predictor is very small (7.5 minutes). Thus, it is easy to simply _re-index_ the operation matrices and re-train a predictor to transfer to a novel search space. To avoid confusion, we have also added a short sentence discussing the cost of re-indexing a predictor such that it does not need to incur large compute costs on new search spaces. We also add a small section detailing this in the Appendix (A.4).

`"However, search on a global graph structure—one that encompasses all NNs—can be an intractable search problem due to its size." is very weak. ` and `related work in this field called CDP [3] that is older than Multi-Predict/GENNAPE`

We agree, thank you for bringing this to our attention. We have removed this weak claim and instead explicitly state our focus on cell-based NAS. We also acknowledge the work done in CDP for cross-domain prediction. We will try to compare to the CDP work–we are looking into it now.

Reviewer comment (Evaluation):
Results in Tables 3/4 do not inspire confidence given the paper's goals. For Table 3, FLAN+ZCP performs strongly in some scenarios (NASBench-101, DARTS, ENAS (FixWD)) but less so in others (NASBench-201, DARTS_{FixWD}). In Table 4, it dominates only on ENAS<->DARTS. A method targeting transferability with few target samples should either dominate conclusively or offer reasoning for why the best predictor configuration changes with the transfer target. Without this, especially in real scenarios with truly limited labeled data, it is unclear how to choose between predictor configurations, muting the usefulness of FLOW.

Author response:
`you would expect it to either dominate quite conclusively…mutes the usefulness of FLOW.`

We find that ZCP dominates most often, while sometimes CAZ may offer additional information by virtue of providing CATE and Arch2Vec encodings. Further, we justify/offer insights on the lukewarm result for NASBench-101 and NASBench-201 in the Appendix (Table 9) by empirically testing the kendall tau when using a single source space and multiple target spaces (Amoeba) and comparing those results with our current results in Table 4.

Reviewer comment (Presentation):
Presentation is mixed. The introduction and related work are clear and engaging, but the writing declines afterward, especially in Section 4.1 describing the GNN/DGF. Tables from Table 1 onwards contain issues (e.g., DF+GAT x NB201 typo). Figure 4 has text that is too small, red dots without legend labels, and poor differentiation of search spaces; better use of colors and markers is suggested. Figures 5 and 6 have small and confusing markers, some with the same color, making them difficult to interpret. Overall, plots should be simplified with fewer entries to aid readability.

Author response:
`Figure 4 is a mess: Text is way too small and there are red dots which seemingly have no label in the legend. plots should have fewer entries, only several important variants so the viewer can actually see what is going on.`

We completely agree with these comments. Initially, we aimed to maximize the number of results in the figures/graphs. We have taken this comment to heart, and instead shifted the full figures to the Appendix. We have regenerated Figure 4 and 5 with only a few important variants of our work. Further, we have made an effort to maximize the size of markers and text to improve the clarity of our figures. We have made appropriate changes to other sections as well to improve the quality of our presentation and we appreciate your comments regarding this. 

`DGF-GAT modules for the normal and reduce cells, and adding the aggregated outputs" - why this design choice? In graph frameworks like DGL and PyTorch-Geometric it is entirely possible`

Our framework supports disconnected graphs in the same manner, and for few-shot learning, having a single shared module is beneficial, but when sufficient source-space samples are present, the ensembling helps. 

`"Contrary to prior work, we generate encodings for all architectures in the search space for evaluation." Then why does Table 2 list "Proporations of 7290 samples" for NAS-Bench-101?`

We did this to compare fairly and directly to prior work. For example, TAGATES tests on 7290 samples instead of the entire search space for NB201. Table 2 is solely dedicated to comparing our predictor with existing work. For the rest of the paper, we report results on the entire search space where possible.

In this work, we generate all encodings (individual and unified) for over 1.5 million neural networks, and 13 zero cost proxies for over 0.5 million neural networks and our code (to be released) allows experimentation with all of these encodings, and is no way limited to the 7290 architectures. This will hopefully enable future research on large NAS search spaces. 

In summary, we have slightly modified our text and T-SNE graph+interpretation to have a stronger focus on empirical evidence for the effectiveness of our methodologies, and re-ran key experiments in the tables with more trials upon request. 

We hope you have addressed your concerns, and would love to hear further feedback. Thank you!

Reviewer comment (Evaluation):
The paper starts with strong writing but later contains incorrect/unsupported statements and inferences in methodology and results sections, culminating in a dense results section that fails to instill confidence in the proposed method’s efficacy for future use.

Author response:
`The response to these criticisms was to concede that their claims were false`

The review process helps us refine our claims, and potentially identify unsupported statements. With your help, we have identified claims that are unsupported (not necessarily false), and adjusted them appropriately.

`comment them out (DGF/GENNAPE/CDP) or rephrase what was said while moving it to the Appendix (t-SNE plots). In doing so the authors undermine themselves, `

We benefit from the reviewer feedback on unsupported claims, and adjust our paper such that it better serves the NAS community. We move the plots to Appendix to make space for additional results requested by other reviewers. With your aid, we regenerated improved figures, and thus are able to have a more robust discussion in the appendix, and we add to the body of knowledge, and *not* to undermine ourselves. We also do not undermine our contributions, as we focus on **encodings and their role in cell based predictor-based NAS**, not necessarily designing the best graph neural network or addressing all forms of NAS.

`If the benefit of DGF is just empirically higher performance, are there not other, simpler mechanisms by which this can be achieved?`

Yes, we propose a new encoder that outperforms prior methods, we do not claim it is the best method possible in the field of graph neural network research. We empirically focus on **neural network encodings**.




`(GAT got published by showings its better than GCN)`

Again, we empirically focus on **neural network encodings**. The merit of adding GAT to the ensemble is not the sole contribution of this work.

` 'Score-based' being a better descriptor for FLAN's architecture representation was not convincing and half-hearted. `

It is not a better descriptor, it is a supplementary encoding that was empirically shown to work well, and we use it as such.

` 'We have removed this weak claim and instead explicitly state our focus on cell-based NAS'`

The claim was that global NAS is difficult, we only make this claim once, and thus, we removed it only once. This is part of our wider research question of enabling open-ended NAS but is not specifically addressed in this paper on NN encodings. Thus, we removed this sentence to keep the manuscript focused. 

`A similar rhetoric trick occurs when the author's respond to my question about the 7290 samples for NAS-Bench-101.`

This is **not true**, the other tables present results on ALL NASBench-101 architectures. 7290 is explicitly to compare with TA-GATES and we still report full results.

`If the claim about GENNAPE's global graph structures leading to intractable search is false, why use the unique numerical indices proposed with FLAN, which must be re-designed for new search spaces?`

This is false, it does not need to be re-designed, re-indexing is trivial (counting and converting to one-hot) and search-space agnostic. We can indeed use ZCP or other metrics, but indexing is our selected method differentiating between operations in search space, and it works well as shown by our experiments. We definitely believe that this warrants further investigation in NAS.

`Why not just concat ZCP/CATE/Arch2Vec info to their existing architecture encodings?`

We explicitly do this, it is “CAZ”. However, as you mentioned (and we acknowledge), these representations are not unique, whereas indexes are unique by construction.

`instead scattering NNs according to their computational complexity" has been replaced with "This proximity is influenced by the similarities in their parameter counts".`

Computational complexity == FLOPs/Parameter Count, as described in the CATE [1] paper.

`FLAN has a lot of instability, especially with the NAS-Bench-101 and NAS-Bench-201 plots.`

This is explained in the Appendix, along with results for Amoeba. 

`"FLAN to enhance its accuracy by up to 64%" now being reduced to 47%, is a huge negative change.`

We do not believe it is a huge negative change, as it still shows considerable improvement relative to prior work. This revision is the result of running more seeds as requested by other reviewers.

We thank the reviewer for their detailed and constructive feedback. We hope you will reconsider your score, keeping in mind that our work primarily focuses on the **evaluation of neural architecture encodings in NAS**. We believe that many of your remaining concerns are not regarding our core contribution on studying DNN encodings, rather the predictor design (created as a vehicle to study encodings) and some (valid and helpful) nitpicks on result presentation.

[1] Yan, S., Song, K., Liu, F., & Zhang, M. (2021). CATE: Computation-aware Neural Architecture Encoding with Transformers.

Reviewer comment (Experiments):
The Authors state that improvements gained in sample efficiency by pre-training FLAN on a source search space do not include the cost to pre-train the model. However, this makes sense only if the pre-training is done on a single source space and then transferred to any other space. From table 4, this doesn’t seem to be the case, and each target space has its own source space.

Author response:
`From table 4, this doesn’t seem to be the case, and each target space has its own source space`

In the main paper, we hope to demonstrate that our framework supports transfer from an arbitrary source space to a target space. We believe that demonstrating this ability is key to universal sample efficient NAS. However, we also acknowledge the importance of demonstrating the transfer of a predictor from a single search space to multiple target spaces. With this in mind, we have added a table to the Appendix (Table 9), where we take Amoeba as our choice of source space, and transfer the trained predictor to NASBench-101, NASBench-201, DARTS, DARTS Fix W-D, ENAS, ENAS Fix W-D. Our conclusion further emphasises the trend which favours ZCP and CAZ supplementary encodings during transfer. We hope this helps in evaluating the effectiveness of predictor transfer.

Reviewer comment (Experiments):
To make a better comparison with TA-GATES, NB301 would be a natural choice to include in table 2.

Author response:
`To make a better comparison with TA-GATES, NB301 would be a natural choice to include in table 2.`

We agree, one of the reasons why we did not include NB301 is because we were not able to trace back the networks used for training TA-GATES on NB301 to the original NB301 library. We have now added the NB301 results in Table 2, and FLAN ZCP performs consistently very well in our tests.

Reviewer comment (Experiments):
In Figs.5,6 the source search spaces used for pre-training are not clearly specified. Are these the same as Table.4?

Author response:
`In Figs.5,6 the source search spaces ... specified. ...same as Table.4?`

Thank you for bringing this to our attention. Yes, it is indeed the same as Table 4. We have added a clarification to the caption to explicitly state this.

Reviewer comment (Reproducibility):
In Table.15, for Amoeba, NB301 and TB101, what are the source search spaces used for pre-training FLAN^T?

Author response:
`In Table.15, for Amoeba, NB301 and TB101, what are the source search spaces used for pre-training FLAN^T?`

Thank you for bringing this to our attention, we have added another table that shows the source to target search space mapping. These experiments will also be in ready-to-go scripts in our code for further clarity.

Reviewer comment (Evaluation):
Is Table.15 supposed to generalize Tables.3,4 (or Tables.17,18)? If so, why don’t the numbers match? See e.g. FLAN_ZCP for 128 samples on NB101.

Author response:
`Is Table.15 supposed to generalize ... See e.g. FLAN_ZCP for 128 samples on NB101.`

There is a slight numerical mismatch because the larger table has fewer trials. As mentioned above, to match the trials with TAGATES, we updated our main paper results to be reporting on 9 trials.

Reviewer comment (Experiments):
In the experiments of Table.2 (or Table.16) the number of trials for FLAN seems to be 3. Is this consistent with the number of trials used for TA-GATES reported in the tables?

Author response:
`In the experiments of Table.2 (or Table.16) the ... of trials used for TA-GATES reported in the tables?`

Thank you for bringing this to our attention. To fairly evaluate our work with TA-GATES, we have now re-evaluated our main tables for 9 trials. All our conclusions remain the same despite minor numerical differences.

Reviewer comment (Evaluation):
In the line above Table.4, “Over baseline FLAN, incorporating ZCP encoding improves predictor accuracy by 64% on average”, how can we see this from the tables?

Author response:
`predictor accuracy by 64% on average”, how can we see this from the tables?`

These calculations are made as shown below:
```
>>> flanbase = [0.4424,0.3898,...,0.4876,0.5252]
>>> flanzcp = [0.4996,0.5655,...0.5231,0.5511]
>>> sum([100*(flanzcp[idx] - flanbase[idx])/flanbase[idx] for idx in range(len(flanzcp))])/len(flanzcp)
>>> 64.47
```
The numbers in the flanbase and flanzcp list represent the kendall tau correlation for tests in the table (full list excluded due to character limit but can be provided upon request), and we calculate the average of the percentage improvement in base v.s. ZCP supplementary encoding.

Reviewer comment (Writing):
In the NAS paragraph of page 9: “Our FLAN^T_{CAZ} is able to improve the sample efficiency of end-to-end NAS on NB101 by 2.12× over other methods”, shouldn’t this be FLAN_{CAZ}?

Author response:
` “Our FLAN^T_{CAZ} is able to ... by 2.12× .... shouldn’t this be FLAN_{CAZ}?`

Thank you for the correction, you are indeed right. We have made the appropriate changes.

Reviewer comment (Experiments):
Can the authors provide some experiments where they demonstrate the runtime of their performance predictor?

Author response:
`Demonstrate the runtime of performance predictor`

We believe that runtime performance is very important when prototyping search spaces for accuracy. Our training time for a transfer experiment is 7.5 minutes, and the transfer time (to adapt to a new domain) is less than 1 minute. Further, for inference, we are able to evaluate an average of 160 neural architectures per second.  We have added this to the appendix (A.4).

Reviewer comment (Experiments):
Did authors evaluate their simple NAS method in section 5 (Table 5) on other search spaces except NAS-Bench-101?

Author response:
`Did authors evaluate their simple NAS method in section 5 (Table 5) on other search spaces except NAS-Bench-101?`

Yes, we evaluate our NAS method on three search spaces in the main paper (Figure 5), and on all search spaces in the appendix (Figure 9 and 10). Please note that Table 6 is formatted in such a way to be consistent and directly comparable with prior literature (that also used the same format) in a clear manner.

Reviewer comment (Evaluation):
In Table 4 it seems that the performance of FLAN + ZCP, almost in all settings, especially ENAS -> DARTS and vice versa, deteriorates as you increase the number of samples. Do the authors have any intuition why this is the case? Did they try with more number of samples (say up to 50 as in Table 5)?

Author response:
`deteriorates as you increase the number of samples.`

We test this in more depth, specifically on the task of NDS CIFAR-10 to NDS ImageNet transfer, and find that in Table 5, more samples generally improves performance (On NDS DARTS, ENAS, NASNet and PNAS). There is some ambiguity when there are fewer than 8 samples (As seen on TB101), and this is because for such low sample-counts, the predictor performance variance could also be largely dictated by the specific NN architectures used for adaptation, and not necessarily just the learning algorithm. Future work could look at finding methods to sample better NN architectures for few-shot learning.