Paper: Guided Evolution with Binary Discriminators for ML Program Search (ICLR-2024)

Reviewer comment (Evaluation):
The main issue of the paper is that the author claims to improve the efficiency of evolution by reducing the sample size needed for evaluation, but completely ignores the computation cost of the proposed method itself. While the method seems to slightly reduce the sample size, it is unclear if there is really a speed-up in evolution. There are many factors that will affect the actual time, such as the frequency of online training, the size of GNN, and the average inference speed of each prediction. The authors should take into account these computation costs and compensate the baseline for more samples to run for fair comparisons.

Author response:
We agree that the computation cost of the method itself needs to be taken into account. For simple tasks such as symbolic regression, the method will dominate the compute cost because evaluation is very cheap. The symbolic regression benchmark was more used as a cheap toy task to see if this method could speed up evolution. For more expensive tasks like Hero and AutoRL, the evaluation still dominates the computation. First we note that in Figure 3, for AutoRL the x-axis is CPU hours and includes the model prediction time (although not training time which is done on a separate server in parallel).

Second, there are two components to the cost, wall clock time and total compute. Hero and AutoRL use an asynchronous training setup where the binary model is trained asynchronously with evolution so this will not add to the overall wall clock time. For compute there is another GPU server to train the model so this adds an extra GPU to the existing 20 evaluation worker GPUs for Hero and an extra GPU to the 100 existing CPU workers for AutoRL. The number of workers is listed in Table 1.  For Hero, this is a minimal increase (5%) in compute cost. For AutoRL, this is a larger increase. However we note that as you increase the number of workers as is common for running larger evolution experiments, the cost to train the model stays the same.

Reviewer comment (Experiments):
Nguyen is a simple benchmark, and regularized evolution isn't the best method on this benchmark. The authors should try to at least demonstrate the average performance of all problems, or switch to a more recent benchmark (e.g., SRBench) for more convincing results.

Author response:
1. We chose the hardest problems in Nguyen to test but we can add an aggregate metric for all problems.

Reviewer comment (Experiments):
In addition to regularized tournament, the authors should compare to other selection methods to show generality of their mutation method.

Author response:
2. Regularized evolution is a simple and popular method for AutoML. For the mutation method, we do compare Max-List with PAM-RT and PAM (Figure 10 and 11). We are curious what other selection methods you have in mind.

Reviewer comment (Evaluation):
On Hero and AutoRL, the method doesn't show significant improvement, especially when the performance seems to start to converge. It is unclear if those results are actually the best converging results. Since the main concern is the efficiency to achieve the best result, better performance in early stages does not matter too much, raising doubts about whether the method is actually useful in any way.

Author response:
3. Figure 3 still shows better final performance after convergence of our method vs vanilla regularized evolution.

Reviewer comment (Evaluation):
In figure 4, the oracle experiments are great, but it is unclear what the accuracy of the predictor models that are online trained is. It would be useful to include an analysis to further validate the method.

Author response:
4. The accuracy of the predictor models varies during online training but falls between 75-90%. We keep track of this metric during training and can add it to the Appendix.

Reviewer comment (Experiments):
It is unclear whether the mutation rate and the actual mutation method affect the performance of the proposed strategy.

Author response:
5. The mutation method does affect the performance as shown in Section 6.4.