Paper: NAP2: Neural Networks Hyperparameter Optimization Using Weights and Gradients Analysis (ICLR-2024)

Reviewer comment (Experiments):
The proposed method is more like a 'performance predictor' than a 'hyperparameter optimization method' [1]. NAP2 is trained with more than 1,000 evaluation results of fully trained neural network samples, but the comparison methods did not use them. It is more reasonable to compare it with performance predictors like BANANAS [2], NPENAS [3], or AG-Net [4].

Author response:
1) **“It is more reasonable to compare it with performance predictors like BANANAS[2], NPENAS[3], or AG-Net[4]”**

While the reviewer is correct in stating that previous studies (Bananas, Npeans, etc.) used performance prediction, we believe the reviewer overlooked a *critical point*: all mentioned studies require the learning model to be trained on the evaluated architectures. *NAP2, on the other hand, can be trained on one set of architectures, then applied to another set* – even one trained on another dataset (CIFAR-100 in our case) without any additional training.

This difference has significant implications: first, we can train the prediction model `offline’, without any connection to the test set architectures. Then, we can use the trained model in multiple experiments, even on other datasets. Secondly, for our model to work at test time, we only need to extract our meta-features once every 100 mini-batches (the interval size we used in our evaluation). Coupled with the fact that we sample values from each layer, this process has negligible computational cost. This means that our setup is the same as the HPO baselines to which we compare: we begin our evaluation with a fixed set of architectures on which we have no knowledge.

Reviewer comment (Evaluation):
Figure 6 shows that the mean square error between performance prediction and final accuracy is about 0.015, which is larger than the mean absolute error of other performance prediction results [5] in NAS-Bench-101 (mean absolute error < 0.01). Also, the result MSE > 0.86 in CIFAR-100 doesn't substantiate NAP2’s transferability if the scale of accuracy is 0 to 1.

Author response:
2) **"Figure 6 shows that the mean square error between performance prediction and final accuracy is about 0.015, which is larger than the mean absolute error of other performance prediction results[5] in NAS-Bench-101 (mean absolute error < 0.01). Also, the result MSE > 0.86 in CIFAR-100 doesn't substantiate NAP2’s transferability if the scale of accuracy is 0 to 1.”**

We would like to begin with the second point: the **reviewer has MISREAD Figure 6**: the MSE error for CIFAR-100 is 0.086, NOT 0.86. This order of magnitude makes all the difference in understanding our results, since they show that a model that was trained on CIFAR-10 architectures is transferable to CIFAR-100.

Secondly, the reviewer is correct that [5] achieved an error of <0.01, but that was achieved after multiple epochs of training and the use of several training seeds. Our approach, on the other hand, achieves MSE of about 0.015 AFTER A SINGLE EPOCH on the evaluation set architectures.  Additionally, as shown in Figures 2-5, NAP2 requires less than an epoch (100-300 mini-batch steps) to significantly outperform all the baselines.

Reviewer comment (Evaluation):
NAS-Bench-201 [7] reports that the best architecture on CIFAR-10, CIFAR-100, ImageNet16-120 are slightly different. I also think the optimal architectures of different datasets should relate to the complexity of the dataset. Why is NAP2 “generic and transferable across datasets and architectures”?

Author response:
2) **"NAS-Bench-201[7] reports that the...I also think the optimal architectures of different datasets..."**

There is a key difference between our work and [7]: the latter analyzed tens of thousands of architectures, and thus were able to reach broad conclusions. Our experiments were conducted on 1,500 architectures that were sampled from NAS-Bench-101. Because of the smaller sample size,  we observe different performance distribution:  in our experiments, some of our top-performing architectures had 30+ layers, while others had around 60.

Because of the relatively small sample sizes, we also cannot form a conclusion regarding the complexities of architectures and their respective datasets. 

3) **"Why NAP2 is “generic and transferable across datasets and architectures”?"**

As we explain in our response to the weaknesses detailed by the reviewer, the NAP2 model is trained on one set of architectures, then applied without any additional learning to another. We conduct two sets of experiments: 

a)	Training on one set of CIFAR-10-trained architectures, then applying the model on another (disjoint) set of architectures trained on the same dataset.

b)	Training the model on CIFAR-10-trained architectures, then applying the model on a disjoint set of architectures trained on a different dataset (CIFAR-100).

As shown in our results (see Figures 2-5), NAP2 significantly outperforms the baselines in both experiments. Moreover, it is clear that our approach only requires 100-300 mini-batches (less than one epoch) to identify top-performing architectures.

It is likely that the reviewer’s question stems from misreading the MSE values in Figure 6 (0.86 instead of the actual value of 0.086). As mentioned above, NAP2 was able to obtain these values after approximately one epoch of training, compared to dozens required by previous studies dealing with prediction.

Reviewer comment (Experiments):
If the focus of the method is for hyperparameter optimization, then the evaluation should follow a HPO pipeline, i.e., there should be no training dataset of models with ground-truth performance. The whole experiment should be in an online fashion. Otherwise, the proposed method is just a prediction model and do not show its usability in HPO.

Author response:
The reviewer touched on a point that is critical to understanding the strength and re-usability of our model. The reviewer would be correct if the prediction model had to be trained anew for each task. This is **NOT** the case: our prediction model can be trained once, then used in other prediction tasks. We demonstrate the re-usability of our trained model in two ways, with varying degrees of difficulty:

a) Training our model on a set of architectures that were trained CIFAR-10, and then performing HPO on a another (disjoint) set of architectures trained on the same dataset.

b) Training our model on a set of architectures that were trained CIFAR-10, then performing HPO on a set of architectures that were trained on CIFAR-100.

While the second setup is more difficult for our model, because of the need to generalize, NAP2 performs very well in both scenarios.

It is precisely for the reasons mentioned above, that we include in our contributions (see Section 1) the release of the trained model along with our dataset and code. The inclusion of the model is designed to enable other researchers to utilize our approach more easily.

We will update our paper to better emphasize this point.

Reviewer comment (Evaluation):
The evaluation of resources used is unclear to me. Is the cost of training and infering NAP2 counted in calculation? Other baseline methods such as Successive Halving do not require extra computation, so you should probably compensate for those methods with extra computation to make it a fair comparison.

Author response:
The cost of training our model is not included in our evaluation. Our reasoning is as follows: 

a) As explained in our response to the previous comment, because the idea behind our model is “train once, use multiple times” (including on other datasets), we do not consider the training cost to be relevant to the evaluation.

b) The training of the NAP2 model can be done ‘offline’, a long time before the online evaluation.

c) Conceptually, our work is similar to studies that use pre-trained LLMs for various tasks. These LLMs require huge amounts of time and compute resources, but these costs are not factored into the evaluation, because they are used `as-is’. The same logic applies to the training of our model.

We will update our paper to better emphasize this point.

Reviewer comment (Experiments):
On CIFAR-100, the comparison methods seem to perform extremely bad, e.g., none of them get any precision > 0. I wonder if the configurations are reasonable according to the original work? It is unlikely that these methods should use the same set of configurations.

Author response:
The explanation for the baselines’ poor performance on CIFAR-100 is simple, and comprises of three reasons:

a)	The number of architectures is larger: in CIFAR-10 we have 1160 architectures but use a four-fold evaluation. In CIFAR-100, we have 440 architectures, and all are used for evaluation. The larger number of architectures makes detecting the top performers  more difficult (esp. a small number like top-3 or top-5).

b)	Performing well on CIFAR-100 is more challenging because of the larger number of classes (see Figure 7 in the appendix for performance distribution). As a result, the baselines (which do not have NAP2 predictive capabilities) perform more poorly.

c)	We show the results for small number of epochs, because NAP2 does not need more in order to perform well. We elaborate on this further in our response to question #5 below.

It should also be noted that none of our baselines used the top-3/5 precision in their reported evaluation. We include it in our evaluation specifically because it is a very challenging task that only NAP2 manages to do well.

Reviewer comment (Experiments):
In Figure 2-5, it seems that all 3 methods do not benefit with more resources, which is not intuitive. I wonder if the authors could include a simple hill climbing or BO baseline to better illustrate the results?

Author response:
The reason for the lack of improvement by the baselines is simple: 100 mini-batches are about a quarter of an epoch (see Section 4.2). For CIFAR-10, we use four-fold cross validation, so one epoch for all evaluated architectures (before pruning architectures) is ~110,000 mini-batches. This means that we present results for about four epochs of training. 

This small number of epochs is more than enough for our approach, which begins to be effective after as little as 100 mini-batches (a quarter of an epoch). This is the result of our approach being a white-box approach, and its ability to obtain insights mid-epoch (as opposed to, for example, curve extrapolation prediction methods). The other approaches are too slow to succeed in such a short time frame.