Paper: Robustifying and Boosting Training-Free Neural Architecture Search (ICLR-2024)

Reviewer comment (Experiments):
The authors propose to train a BO to capture the relationship between the weight vector and the objective evaluation metric f. However, the queried architecture should be trained from scratch to obtain the objective evaluation during the BO stage, which seems to require large amounts of search costs since a standard BO procedure usually requires tens of queries. BTW: What does $R_f(A)$ denote in Eq. 1? Does it represent the objective evaluation of an architecture? Since Alg.1 directly uses $f$ to denote the objective evaluation metric, I suggest the authors utilize the same notation.

Author response:
Thank you for pointing out the concern regarding the search efficiency of each query in our method. In practice, we usually apply the validation performance of each candidate architecture that is trained from scratch for only a limited number of epochs (e.g., 12 epochs in NAS-Bench-201) to approximate the true performance of this architecture as stated in our Appendix B.2 and C.2, which in fact is quite computationally efficient (e.g., about 110 GPU-seconds for the model training of each candidate in NAS-Bench-201). Of note, such an approximation, which has been applied in the literature [1], is already reasonably good to help find well-performing architectures, as supported by the results in both Table 2 and Table 4. We would like to add this discussion to the main body of our revised paper to make it clearer.

Thank you for addressing the concern regarding our use of the notations $R_f(A)$ and $f$. As presented in the first paragraph of Section 4, $f(A)$ denotes the objective evaluation metric for $A$, while $R_f(A)$ denotes the **ranking** of $A$ in the entire search space based on the objective evaluation metric $f$. Particularly, $R_f(A)=1$ indicates that architecture $A$ is an optimal architecture in the entire search space based on the evaluation metric $f$. Of note, we introduce $R_f$ mainly to facilitate our theoretical analysis detailed in Section 4.3 and Section 5, given that our theoretical analysis mainly focuses on the ranking performance. But in fact, finding the minimum of $R_f(A(w))$ w.r.t $A(w)$ is essentially equivalent to find the maximum of $f(A(w))$ w.r.t $A(w)$, according to our definition of $R_f(A)=1$.

We agree with you that using both $R_f(A)$ and $f$ in our formulation and algorithm could be confusing. Therefore, we would like to follow your suggestion to revise Equation 1 using only $f$ and reserve $R_f(A)$ only for the theoretical sections. This change aims to make our presentation more coherent and accessible to our readers.

Reviewer comment (Evaluation):
I wonder about the effectiveness of the searched robust estimation metric $M^*$. According to Fig. 2, it seems that the optimal architecture has been found in less than 10 queries during the BO procedure. It shows that there is no need to conduct the greedy search through $M^*$, and BO is enough to get the optimal architecture.

Author response:
We would like to clarify that the search costs in Figure 2 include the queries from both the BO stage (e.g., the first 5 queries in Figure 2a that are automatically determined by our RoBoT) and the greedy search procedure (e.g., the last 15 queries in Figure 2a). The results show that greedy search is indeed essential and necessary to help our method achieve competitive search performances, which is further supported by our ablation study in Figure 6. To enhance clarity, we will revise Figure 2 to highlight the search costs incurred by the BO stage and the greedy search procedure separately.

Reviewer comment (Evaluation):
Table 4 shows that RoBoT only requires 0.6 GPU-day to search, does it only count the search cost of the greedy search procedure? I wonder what is the cost of the BO stage, which I am afraid is much larger.

Author response:
We thank you for pointing out this question. We would like to clarify that the search cost in our paper includes the computational costs incurred not only in the greedy search stage but also in the BO stage. In fact, these two stages have the same type of training cost that is used to evaluate the validation performance of each queried candidate architecture. Specifically, such a performance evaluation procedure requires about 0.04 GPU-day for the model training of each queried architecture in the DARTS search space for ImageNet where we have queried approximately 3 architectures in the BO stage and 7 architectures in the greedy search procedure. We would like to add this discussion to the main body of our revised paper to make it clearer.

Reviewer comment (Reproducibility):
Missing details regarding robust metric: It seems that some important details about the BO-searched robust estimation metric are missing. What are the base training-free metrics considered in the search? What are the optimized linear combination weights for them? Do they significantly differ on different datasets/tasks?

Author response:
## About Robust Metric

> What are the base training-free metrics considered in the search?

As reported in our Appendix B.2 and C.2, we borrowed the six widely known training-free metrics (i.e., *grad_norm, snip, grasp, fisher, synflow,* and *jacob_cov*) from [1] for most of the experiments in our paper except for the tasks of *Segment.*, *Normal*, and *Autoenco* in TransNAS-Bench-101 where *synflow* was excluded due to its incompatibility with the tanh activation applied in each candidate architecture within the search space.

> What are the optimized linear combination weights for them? Do they significantly differ on different datasets/tasks?

Thank you for raising this interesting question regarding the optimized linear combination weights in various tasks. We present the optimized weights, along with their similarities and correlations, across four tasks in TransNAS-Bench-101-micro as follows:

Varying Optimized Weights

|  | grad_norm | snip | grasp | fisher | synflow | jacob_cov |
|---|---|---|---|---|---|---|
| Scene | -1.00 | -0.08 | -0.97 | 1.00 | 1.00 | 1.00 |
| Object | 0.03 | -0.21 | -0.76 | 0.51 | 0.95 | 0.16 |
| Jigsaw | -0.74 | 0.18 | 0.04 | -1.00 | -1.00 | 1.00 |
| Layout | -0.65 | -0.27 | 0.57 | -0.48 | 1.00 | 0.67 |

Cosine Similarity

|  | Scene | Object | Jigsaw | Layout |
|---|---|---|---|---|
| Scene | 1.00 | 0.78 | -0.07 | 0.37 |
| Object | 0.78 | 1.00 | -0.55 | 0.19 |
| Jigsaw | -0.07 | -0.55 | 1.00 | 0.2 |
| Layout | 0.37 | 0.19 | 0.2 | 1.00 |

Pearson Correlation

|  | Scene | Object | Jigsaw | Layout |
|---|---|---|---|---|
| Scene | 1.00 | 0.77 | -0.02 | 0.35 |
| Object | 0.77 | 1.00 | -0.52 | 0.16 |
| Jigsaw | -0.02 | -0.52 | 1.00 | 0.3 |
| Layout | 0.35 | 0.16 | 0.3 | 1.00 |

The results show that the optimized weights typically vary for different tasks, which aligns with the observations and motivations in Section 3 and further highlights the necessity of developing robust metrics that can perform consistently well on diverse tasks such as our RoBoT. In addition, for tasks with similar characteristics, e.g., the *Scene* and *Object* tasks, both of which are classification tasks, the optimized weights share a relatively high similarity/correlation, indicating the potential transferability of the optimized linear combination within similar tasks. We would like to add these results and discussions to our revised paper.

Reviewer comment (Experiments):
Recent NAS methods: It is suggested to include some more recent NAS methods into the comparison, e.g., Shapley-NAS [1], $\beta$-DARTS [2].

Author response:
## About Including More Baselines

We appreciate your suggestion on more recent baselines. We show our comparison on NAS-Bench-201 as below. Note that we have applied smaller search budgets for Shaply-NAS and $\beta$-DARTS than the ones presented in their original papers in order to maintain comparable search budgets for different training-based NAS algorithms (e.g., DARTS (2nd), GDAS, DrNAS).  The results show that our RoBoT can still achieve better search performances than these two recent baselines.  We will add these baselines and comparisons in our revised paper.

|  | C10 (Test Acc %) | C100 (Test  Acc %) | IN-16 (Test  Acc %) | Cost (GPU Sec.) |
|---|---|---|---|---|
| Shaply-NAS | 94.05±0.19 | 73.15±0.26 | 46.25±0.25 | 14762 |
| $\beta$-DARTS | 94.00±0.22 | 72.91±0.43 | 46.20±0.38 | 3280 |
| RoBoT | **94.36**±0.00 | **73.51**±0.00 | **46.34**±0.00 | 3051 |

Reviewer comment (Evaluation):
In Table 3, why are the results on TransNAS-Bench-101 presented as the validation ranking? It seems to be inconsistent with the accuracy/error in the other two datasets (Tables 2 and 4). Also, the search costs are not listed in Table 3.

Author response:
## About the Presentation of Table 3

>  Why are the results on TransNAS-Bench-101 presented as the validation ranking?

We would like to clarify that since there is a noticeable gap between the validation and the test performances in TransNAS-Bench-101, we follow the common practice [2] to present the validation ranking, as detailed in Appendix B.2.

> the search costs are not listed in Table 3

We would like to clarify that, in fact, we have included the search costs (measured by the number of queries, i.e., 100) in the caption of Table 3.