Paper: Multi-fidelity Deep Symbolic Optimization (ICLR-2024)

Reviewer comment (Experiments):
The generality of the proposed method is doubtful, as the proposed method and experiments are primarily based on problems with only two reward fidelities.

Author response:
While our evaluations and methods are more focused on the two-fidelities case (which is more common than many rewards in multi-fidelity applications), our methods are applicable to more fidelities with none or very small adaptions (for example, the symbolic regression experiment is carried out with 3 fidelities, and adding more would be possible with a slightly higher effort into setting hyper parameters correctly).

Reviewer comment (Experiments):
The authors may want to evaluate their method in real-world complex problems with multiple (more than three) reward fidelities and transition fidelities.

Author response:
Thanks for the suggestion. Developing the methods for multiple transition functions is not trivial and is a future work. We contribute in this paper a very complex real-world problem that motivated the development of those methods (antibody development), and a simple evaluation domain with 3 fidelities (Symbolic Regression).

Reviewer comment (Evaluation):
I expect the authors to report the running time of different methods, as calling the low-fidelity estimators also takes time.

Author response:
We apologize for being unclear regarding the running times. For the antibody optimization domain, the running time is completely dominated by the time spent evaluating samples in the high-fidelity. All the algorithms were executed for the same budget of 250 hours (this is why the graphs show a slightly different number of samples per algorithm). The time for training and sampling in the low fidelity is negligible compared to the 15 hours per batch evaluation in the high fidelity. In the symbolic Regression domain, it wouldn’t make sense to report the running time because this is a synthetic domain where we fabricate the rewards. Therefore the running times for each fidelity is artificially the same. The time taken for learning is not significantly different across the different algorithms.

Reviewer comment (Experiments):
It is beneficial to study the calling of different fidelities during the training time.

Author response:
We apologize for being unclear regarding the running times. For the antibody optimization domain, the running time is completely dominated by the time spent evaluating samples in the high-fidelity. All the algorithms were executed for the same budget of 250 hours (this is why the graphs show a slightly different number of samples per algorithm). The time for training and sampling in the low fidelity is negligible compared to the 15 hours per batch evaluation in the high fidelity. In the symbolic Regression domain, it wouldn’t make sense to report the running time because this is a synthetic domain where we fabricate the rewards. Therefore the running times for each fidelity is artificially the same. The time taken for learning is not significantly different across the different algorithms.

Reviewer comment (Experiments):
The baseline methods compared with the proposed methods are very simple and limited.

Author response:
We wish to clarify that the baseline methods added to our empirical comparison are the main general-purpose algorithms currently used as "state of the art" for symbolic optimization applications. For example, the risk-seeking policy gradient (called as upper bound or lower bound depending of the reward used) won the SRBench 2022 competition in the real-world track (https://cavalab.org/srbench/competition-2022/), with some few modifications to make it more specialized for the competition. Therefore, we are rather comparing against the state-of-the-art way of solving SO problems apart from very domain-specific solutions, instead of just comparing against very basic solutions.

While those algorithms are very powerful, they assume that there is just one reward, and before our paper there was no principled way to consider the multi-fidelity scenario. "Upper bound", "Lower bound", and "Sequential" are our interpretation of the ways in which those algorithms could be applied ina. multi-fidelity problem without huge modifications, and therefore those were our baseline methods.

Reviewer comment (Novelty):
The technical novelty of the multi-fidelity treatment is unclear when comparing the existing techniques used in other algorithms.

Author response:
To the best of our knowledge no RL algorithm is capable of solving MF-MDP in a principled and general manner considering that: (i) the algorithm has to take into account that the different rewards are estimations of different precisions for the same measure; (ii) there can be a mixture of fidelities in the learning batch and the algorithm has to consider that the high-fidelity takes precedence over the low fidelity. (iii) sampling in the high fidelity is very expensive and should be done strategically. Our algorithm considers all those points in a principled manner and outperforms the baselines empirically.

Reviewer comment (Novelty):
Could you elaborate on the technical difference and advantage of the proposed method against the existing multi-fidelity handling methods in Bayesian optimization and reinforcement learning?

Author response:
To the best of our knowledge no RL algorithm is capable of solving MF-MDP in a principled and general manner considering that: (i) the algorithm has to take into account that the different rewards are estimations of different precisions for the same measure; (ii) there can be a mixture of fidelities in the learning batch and the algorithm has to consider that the high-fidelity takes precedence over the low fidelity. (iii) sampling in the high fidelity is very expensive and should be done strategically. Our algorithm considers all those points in a principled manner and outperforms the baselines empirically.

Reviewer comment (Experiments):
How did the authors decide and tune the hyperparameters of the proposed and baseline methods? How sensitive is the performance of each algorithm for hyperparameter settings?

Author response:
3) For the empirical evaluation we set the hyperparameter in a way that seemed reasonable considering the "standard Symbolic Regression" configurations in terms of batch size and epsilon value for the risk-seeking gradient set up in previous approaches. In our explorations our algorithms are not overly sensitive to hyperparameters and are only significantly affected in performance for big changes in the parameters. The parameters are fairly interpretable therefore it should be relatively easy to set them for new applications (for example, the elite-picking strategy picks a % of samples to be evaluated in the high fidelity, therefore one just has to set the % that corresponds to the number of high-fidelity evaluations they are budgeting for per batch).

Reviewer comment (Experiments):
The experiment results as they are now are not sufficient evidence of a performance improvement of RSEP and its variants over the baselines. The main experiment results in Figure 2 and Figure 3 show the objective function value v.s. the number of real objective function evaluations. However, the figures do not show how many low-fidelity evaluations are used for each of the compared methods. For example, it is possible that the proposed RSEP method and its variants used more low-fidelity evaluations than the Sequential baseline in the experiments, which could also result in RSEP outperforming Sequential, but that would not be a fair comparison. This seems quite likely to be the case for the experiments on antibody optimization, as the Sequential baseline was trained on only 720 low-fidelity evaluations before switching to only evaluating the real objective function, while Figure 3 shows that the compared methods used over 1000 evaluations of the real objective function. Using only 720 low-fidelity evaluations for Sequential when the low-fidelity evaluations are considered free while doing more evaluations on the expensive real objective function does not make much sense, and this probably indicates that RSEP consumed more evaluations than Sequential. Therefore, it would be more sensible to show the number of low-fidelity evaluations consumed by each of the compared methods as well.

Author response:
We would like to clarify that 720 was selected because it represents 10 training batches, which we empirically observed that is a threshold where the algorithms are getting close to convergence but not overfitted to the low fidelity yet. If the paper is accepted we will include the results with sequential configured with many samples in low fidelity (we didn't do it yet because the experiments take over two weeks and hence there is not enough time during the rebuttal), but we don't expect the performance of the baseline to change significantly. For example, this table is the result for another target virus in which we tried the algorithm (not included in the paper), with sequential configured to 10000 samples in the low fidelity before starting to explore in the high fidelity:

| Algorithm  | MAX  | Avg  |
|------------|------|------|
| SEQUENTIAL | 5.53 | 4.14 |
| RSEP       | 5.61 | 4.45 |

Reviewer comment (Theory):
The problem formulation in the paper considers low-fidelity evaluations as free, no matter what the fidelity is. This does not support the "a whole new family of multi-fidelity SO algorithms that account for multiple fidelities and their associated costs" part of the claim in the abstract. This modeling assumption does not match the application scenarios in reality and causes a problem in the problem formulation because one could just evaluate every possible (exponentially many) token sequence using the low-fidelity approximation at the start of any algorithm. In the case where there are multiple approximations of the real objective function, it seems the zero-cost assumption would make the one approximation with the highest fidelity dominate all other approximations with lower fidelities (more on this in the first question below). As the authors point out in the future work discussion, it would be more reasonable to incorporate the costs of evaluating each fidelity into the problem formulation, where higher fidelity comes with a higher cost.

Author response:
We would like to clarify that the "zero-cost assumption" approximately corresponds to the reality of many applications, such as antibody optimization which inspired the development of the method. Our low-fidelity simulation takes a time in the order of ms to be completed, while the high-fidelity takes 15 hours to evaluate a single antibody sequence. Therefore the low fidelity evolution time is completely negligible compared to the high-fidelity one. While we recognize this is not the case for all applications and the rigorous development of algorithms when the zero cost assumption is not true is future work, it is still possible to use the exact same algorithms with well-thought parameters.

Reviewer comment (Theory):
Because the error rate is defined as $P(R^0 \ge Q^0_\epsilon, R^1 \le Q^1_\epsilon)$ instead of $P(R^1 \le Q^1_\epsilon | R^0 \ge Q^0_\epsilon)$ in Proposition 2, the first observation (Line 150) is quite trivial and does not provide much deep insight for how good the error rate is. If $\epsilon$ is reduced to a very small value, the absolute error rate will be small but we would probably care more about the conditional error rate given that a sample should pass the quantile under the real objective function. The second observation (Line 151) also only tells one side of the story, as a smaller low-fidelity quantile $Q^1_{\epsilon}$ would reduce the error rate, but at the same time, it would reduce the efficiency of using real objective function evaluations.

Author response:
We would like to clarify that, while this Proposition 2 doesn't use extremely complex maths, it brings insights of extreme significance for the algorithm users: how to avoid the case where the algorithm gets stuck in always sampling the same samples in the high fidelity while there are better undiscovered samples that are underestimated by the low fidelity.