Paper: Fast Neural Architecture Search with Random Neural Tangent Kernel (ICLR-2024)

Reviewer comment (Novelty):
I'm not sure about the significance of the contribution. Although the experimental results appear promising, the derivation of the bias-variance decomposition and its associated equations (Eq. 3 and Eq. 4) seems relatively straightforward, which could imply that the contribution is somewhat incremental to previous works. If the novelty of what you're doing can be highlighted even more, I believe it has the potential.

Author response:
The main purpose of our paper is to propose a training-free NAS method that searches appropriate DNN architectures in a very short time compared to existing methods. For that purpose, we use the normalized generalization error, which is efficiently approximated using the normalized 2nd-order moment of NTK. The performance measure based on Eq.3 and Eq.4 is derived by combining some existing theoretical techniques of NTK. ***As shown in Figure 1 in the revised paper, however, our method, NAS-NGE, provides a highly accurate prediction of DNN's long-term training dynamics. Due to that property, we can construct a computationally efficient method for training-free NAS.***

Reviewer comment (Theory):
I believe it's important to carefully explain how much the behavior of models trained outside the NTK-regime can be accounted for by theoretical analysis using NTK, as unrealistic settings like lazy training are implicitly imposed. Currently, only the performance of NAS is reported, but a more direct analysis would increase the level of confidence. Specific points are listed in the "Questions" section.

Author response:
As the reviewer pointed out, there is a gap between NTK theory and practical performance. In the revised paper, we compare the performance of NTK-based training-free NAS methods, NASI, and our method, NAS-NGE. ***Figure 1 in the revised manuscript shows the test errors at several checkpoint epochs.*** Our numerical experiments indicate that our method, NAS-NGE, well predicts the generalization performance than NASI.

Reviewer comment (Experiments):
I believe that NAS-Bench stores performance at multiple checkpoint epochs (e.g., 4, 12, 36, 108 epochs). In this paper, empirical evaluation is conducted with the shortest training time, both for NAS-Bench-101 and NAS-Bench-201. Do you think that extending these checkpoint epochs significantly impacts the results? Do you have results under such settings? Since NTK theory assumes that NTK does not change during training, I imagine that as training epochs increase, it may deviate somewhat from the theory of infinite width. I'm interested in understanding this deviation. If possible, demonstrating its superiority based on differences in trends on epochs compared to other methods using NTKs could potentially strengthen the paper.

Author response:
As the reviewer pointed out, there is a gap between NTK theory and practical performance. In the revised paper, we compare the performance of NTK-based training-free NAS methods, NASI, and our method, NAS-NGE. ***Figure 1 in the revised manuscript shows the test errors at several checkpoint epochs.*** Our numerical experiments indicate that our method, NAS-NGE, well predicts the generalization performance than NASI.

Reviewer comment (Novelty):
A strong motivation is built upon the premise that generalization guarantees are missing in train-free NAS. However, the papers Generalization guarantees for neural architecture search with train-validation split (ICML'21), Generalization Properties of NAS under Activation and Skip Connection Search (NeurIPS’22) are precisely providing generalization guarantees for different cases. Given those papers, I believe one of the core motivations is less clear to me. Could the authors elaborate on the differences from those?

Author response:
In the paper [1], the authors derived an end-to-end generalization bound for DNN learning using training-validation split, and used the bound to training-based method. The proposed method is not training-free NAS. That is the main difference from our method.

In the paper [2], mainly the authors consider theoretical analysis of training-free NAS methods for binary classification problems. On the other hand, we proposed a NTK-based approximation of the normalized generalization error for regression and classification problems. We find that our method provides an accurate prediction of DNN’s long-term training dynamics. Due to that property, we can construct a computationally efficient method for training-free NAS. ***We added numerical experiments in Figure 1 of the revised paper.***

Reviewer comment (Theory):
The paper skips many important details and intuitions that make it quite hard to understand. For instance, why is the second-order moment used in the analysis and why is the NTK assumed here?

Author response:
The second-order moments naturally appear since we consider the squared error as the loss. NTK is used to approximate the learning dynamics of DNN. By approximating learning dynamics using NTK, we can estimate the generalization performance of DNNs after several learning steps without learning. We added the details in the first paragraph of Section 3.

Reviewer comment (Evaluation):
It seems the reported accuracies in this work are lower than existing standard networks, e.g. ResNets. Even if we do not account for those cases, the aforementioned Neurips’22 paper on the same benchmark reports results closer to standard accuracies on cifar10/100. Could the authors elaborate on those discrepancies?

Author response:
Our numerical studies indicate that NAS-NGE can select an appropriate DNN architecture in a very short time. ***Table 4 in the revised paper shows numerical results using NAS-Bench-1shot1, which has a larger search space than NAS-bench-201.*** We see that NAS-NGE is about three times more computationally efficient than NASI and attains a comparable accuracy.

Reviewer comment (Experiments):
Given that one of the main claims is built around the generalization error, I would expect to see how the proposed model extends beyond the test data of the dataset, which is one of the critical points in NAS overall.

Author response:
If the reviewer asks about the relation of NAS to out-of-distribution (OOD) generalization, that’s a very interesting problem. Thank you for the suggestion. ***The theoretical evaluation of NAS under the OOD setting was added to the Conclusion as an interesting future work.***

Reviewer comment (Evaluation):
One of the messages repeatedly relayed throughout the manuscript is the speed of the proposed method. However, I find that the reporting of “about 10 sec.” to be quite rough; could the authors conduct a refined analysis on this?

Author response:
In numerical experiments, the accurate computation time was supplemented in the tables.

Reviewer comment (Evaluation):
Continuing on the aforementioned point, does the reported time include the time to compute the NTK of each architecture?

Author response:
Yes, the computation time of NTK is included. This is supplemented in the caption of Table 2.

Reviewer comment (Theory):
In condition 3.2 it mentions the NTK, however the papers cited mostly focus on Relu-nets, so are there any additional conditions that should hold for the neural network in order for this analysis to hold?

Author response:
As shown in [3], the NTK approximation applies to a wide range of DNNs. In the binary classification problem, Zhu et al.[2] studied generalization properties of NAS for DNNs with skip connections and pseudo-Lipschitz activation functions. ***Description on these works were added to Remark 1 in the end of Section 3.2 of the revised paper.***

Reviewer comment (Experiments):
One of the sentences in the first paragraph of the introduction claims that “Architectures found in NAS are even surpassing the performance of manually designed architectures”. Is there a reference for this? How does the proposed method perform in imagenet that is the standard benchmark in image classification?

Author response:
The paper [4] commented as follows: "In the literature, various search algorithms (Luo et al., 2018; Zoph et al., 2018; Real et al., 2019) have been proposed to search for architectures with comparable or even better performance than the handcrafted ones." We referred to [4] in the first paragraph of Section 2.