Paper: SYMBOL: Generating Flexible Black-Box Optimizers through Symbolic Equation Learning (ICLR-2024)

Reviewer comment (Experiments):
Reliance on the meta-learning stage: The training of the SEG model requires the selection of a training distribution D. The authors rely on 10 BBO optimization problems to construct D. However, in practice, the obtained distribution might not be large enough to guarantee sufficient generalization to fundamentally different tasks.

Author response:
**[Training distribution selection]** We acknowledge the importance of task distribution $\mathbb{D}$ in facilitating generalization during the meta-learning stage of SYMBOL. To address the concern regarding the adequacy of using only 10 synthetic functions for meta-learning a universally effective policy, we have refined the description in “Training distribution $\mathbb{D}$” part in Section 4, and included additional details in Appendix B.4 of our revised paper. Our explanation is threefold:

1. The 10 synthetic functions we have selected encompass a broad spectrum of **representative** landscape features and optimisation challenges, as detailed in the newly added Table 7 in Appendix B.4. These features include uni/multi-modality, separability, smooth yet narrow ridges, and (a)symmetry, etc. In Appendix B.4, the added Figure 9 now includes visualizations of the fitness landscapes of these 10 functions. Through this diverse range and by incorporating fitness landscape analysis-based state representation (refer to Section 3.1 and Appendix A.1), SYMBOL learns not merely to solve specific tasks, but to adapt its update patterns to varying landscape features. This adaptability underpins its impressive generalization capabilities, as demonstrated in our experimental results.
2. We have also augmented the training distribution $\mathbb{D}$ by incorporating shifted and rotated versions of these functions, as outlined at the beginning of Section 4. This methodological augmentation enriches $\mathbb{D}$ with a multitude of diverse landscapes, thereby expanding its scope and ensuring robust generalization.
3. We would also like to note that these selected 10 functions have a **longstanding** history in the literature for evaluating the performance of BBO optimizers. Their proven ability to present diverse optimization properties makes them ideal for training SYMBOL, ensuring its efficacy in generalizing to a wide array of unseen, realistic tasks.

Reviewer comment (Experiments):
Training SYMBOL is still quite computationally demanding as shown in the meta-test column in Table 1. Is training the model via RL the computational bottleneck?

Author response:
**[Is training via RL the computational bottleneck?]** We thank the reviewer for the insightful question. We agree that a detailed examination of the computational costs of SYMBOL’s different sub-routines during training is essential. In SYMBOL, there are three main sub-routines:

1. **Update Rule Generation**: This involves computing the contextual state representation, feeding the state into the SEG (State Encoding Generator) for generating pre-order traversals, and decoding these traversals into the corresponding update rules.
2. **Lower-Level Optimization**: After generating the update rule, SYMBOL applies this rule to update the candidate population and evaluates the objective values of the new generation.
3. **Meta-Level PPO Training**: Concurrently with the lower-level optimization, the collected trajectories are utilized to update the parameters in the SEG via reinforcement learning.

To quantify the computational demands, we measured the wall time for each of these sub-routines during training. The table below presents the consumed time and corresponding proportions for each subroutine, along with the total time, using SYMBOL-S as an example:

| Sub-routines | Update rule generation | Lower-level optimization | Meta-level PPO training | Total time |
| --- | --- | --- | --- | --- |
| Time | 1.5h (14.9%) | 6.3h (62.6%) | 2.2h (22.1%) | 10h |

As the data reveals, **lower-level optimization emerges as the primary computational bottleneck in SYMBOL**. It's important to clarify that high computational costs are intrinsic to BBO (Black-Box Optimization) optimization in the MetaBBO domain. This is evident in the baseline methods we compare against in our paper (like LDE, DE-DDQN, Meta-ES, etc.).

To mitigate these costs, common practice involves parallel processing. Specifically, we simultaneously sample a batch of task instances and collect trajectories in parallel, significantly accelerating both the lower-level optimization and meta-level training processes. For more details on our approach to parallel processing, please refer to Appendix B.2.

We hope the above clarifications help, and we are happy to engage in further discussions. Thanks for taking your time and making contributions to our paper!

Reviewer comment (Experiments):
It is good to see the improvement in the experiment, however, I note SYMBOL-E does not outperform MadDe (2 out of 3 columns). This raises the question of whether the potential of SYMBOL is largely limited by the performance of MadDe. If MadDe does not work well, can the exploration reward actually help the model learn a much better update rule? The authors could try to use a relatively bad baseline as the teacher to verify this and or find a challenging task where MadDe does not work well.

Author response:
**[Whether the potential of SYMBOL is largely limited by the performance of MadDE]** Thanks for raising this concern. According to our experiments, SYMBOL-G and SYMBOL-S both outperform MadDE, clearly demonstrating the potential of SYMBOL to exceed the employed teacher optimizer. However, SYMBOL-E, designed to independently discover novel optimizers without reliance on any particular teacher optimizer, underperforms MadDE. We acknowledge that navigating the complex landscapes of BBO tasks presents significant challenges for SYMBOL-E when learning solely from scratch, which deserves further exploration in the future. So far, SYMBOL-E's strength lies in scenarios where selecting an appropriate teacher optimizer is challenging.

---

**[Can the exploration reward help the model learn a much better update rule?]**  Yes, it helps. Following the suggestions by the reviewer, we address this concern from the following two perspective. 

**i) When the teacher optimizer is relatively bad:** In Section 4.3, we explored the use of a relatively less effective teacher, vanilla DE, within SYMBOL-S. Despite the limitations of the teacher optimizer, SYMBOL-S consistently outperformed it, as evidenced by the performance curves in Figure 5 (Section 4.3) and Figure 8 (Appendix C.1). This indicates the robustness of SYMBOL-S in surpassing teacher capabilities. 

**ii) When the optimization task is challenging for the teacher optimizer**: In the context of more challenging tasks such as HPO-B and Protein-docking (Table 1, Meta Generalization part), SYMBOL exhibits impressive performance even when the teacher optimizer (MadDE) struggles. Both SYMBOL-E and SYMBOL-G show only limited improvements when used independently, yet SYMBOL-S significantly outperforms MadDE, underscoring that SYMBOL is not constrained by its teacher's performance. The synergy between exploration and imitation rewards in SYMBOL not only demonstrates robust generalization across diverse tasks but also ensures rapid convergence.

Reviewer comment (Experiments):
Since it requires training based on reinforcement learning, training may require a time for framework compared with existing BBO that does not require any training. I wonder how long it takes for training compared with other MetaBBO method?

Author response:
**[Training cost comparison between SYMBOL and baselines]** We would like to clarify that the training time of SYMBOL and other MetaBBO baselines have been reported in Section 4.1, Table 1. For your convenience, we list the training time in the table below.

| SYMBOL-E | SYMBOL-G | SYMBOL-S | LDE | DEDDQN | Meta-ES | MELBA | RNN-Opt |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 6h | 10h | 10h | 9h | 38m | 12h | 4h | 11h |

We hope the above clarifications help, and we are happy to engage in further discussions. Thanks for taking your time and making contributions to our paper!

Reviewer comment (Novelty):
It is somewhat unclear to me the key difference and novelty of the proposed SYMBOL because there exists a lot of MetaBBO techniques. It would be very nice if the authors clarified the advantages and key differences of the SYMBOL compared to other existing methods.

Author response:
**[Clarify the advantages and key differences of our SYMBOL]** We thank the reviewer for the suggestion and apologize for any confusion. **To the best of our knowledge,** **SYMBOL represents the initial endeavour to promote the automated discovery of BBO update rules through symbolic equation learning**. Compared to existing MetaBBO techniques, our SYMBOL is **more flexible** (can generate novel update rules beyond hand-crafted rules dynamically depending on the search situation), **more generalizable** (owing to the leveraged fitness landscape analysis as robust state features), and **more interpretable** (the generated update rules are in closed form). Meanwhile, our SYMBOL archives new state-of-the-art performance among other BBO and recently developed MetaBBO methods. Concerning other MetaBBO works, two primary branches exist — one for auto-configuration and the other for candidate solution proposals, which are discussed as follows:

1. **MetaBBO for Auto-Configuration:** This branch involves a neural policy at the meta level that dictates configurations for the **hand-crafted update rules** in the lower-level optimizer, as seen in Table 1 with baselines like LDE, MELBA, and Meta-ES. Alternatively, some other works leverage the meta-level agent to select a proper **hand-crafted update rule** from a predetermined pool of rules, as done by the DEDDQN in Table 1. However, these methods may inherit potential limitations from the hand-crafted rules within the optimizer itself. SYMBOL minimizes dependence on hand-crafted rules and flexibly generates closed-form update rules upon a symbol set, enhancing the exploration capability of efficient update rules and consequently boosting the potential performance of the generated optimizers.
2. **MetaBBO for Candidate Solution Proposal:** This branch employs RNNs to auto-regressively propose the next generation of candidates end-to-end without explicit update rules. Specifically, the input is the position of the current candidate(s), and the output is the next candidate(s). However, these models may easily get overfitted, and their efficiency may be limited to low-dimensional BBOs only, raising generalization concerns. In contrast, SYMBOL incorporates fitness landscape analysis into the state representation, aiming to construct a generalizable model that can produce flexible update rules for various problems, ensuring a more robust optimization. Additionally, while existing works for candidate proposals are entirely "black-box" with limited interpretability, SYMBOL's generated rules are step-by-step transparent, allowing for easier analysis and understanding of the learned processes.

Correspondingly, we have refined our discussions in the second paragraph of the **Introduction** section and the second paragraph of the **Related Works** section.

Reviewer comment (Novelty):
Many techniques regarding MetaBBO, including meta genetic algorithm and meta evolutionary algorithm, have been developed so far [i]. It might be better to mention the traditional approaches related to MetaBBO.

Author response:
**[Q1: Mention traditional MetaBBO works]** We thank the reviewer for the suggestion. We have enriched the **Related Work** section. As introduced in [i], these traditional methods also adhere to the bi-level optimization paradigm. But in their paradigm, both the meta-level and lower-level components are implemented using traditional black-box optimizers. In addition, they generate a single strategy for the entire lower-level optimization process. Our SYMBOL differs in that a) it employs a learning-based RL-Agent as the meta-level method; b) it generates flexible update rules in the lower level; and c) it dynamically adjusts its strategy throughout the lower-level optimization process, adapting to changing conditions and ensuring greater flexibility.

Reviewer comment (Novelty):
The work of [ii] might relate to this paper. Although it only tunes the step-size adaptation in CMA-ES, the concept and used techniques, such as reinforcement learning-based training and guided policy search, are somewhat similar.

Author response:
**[Q2: Add LTO-CMA]** We thank the reviewer for pointing out this reference. We acknowledge that LTO-CMA [ii] is related to SYMBOL in concept, as both approaches aim to search for the optimal policy by leveraging reinforcement learning. However, as the reviewer has pointed out, LTO-CMA is confined to adapting the step-size for CMA-ES, while our SYMBOL generates the update rule from scratch. Additionally, the state representation of LTO-CMA includes CMA-ES-specific entries, making it less generic. In contrast, SYMBOL provides more flexibility in terms of teacher options, as demonstrated in the experimental section of the paper (Section 4.3, Figure 5). Moreover, there are other distinctions between LTO-CMA and our SYMBOL, including the usage of teachers, the learning objective, and the learning algorithm.

In our meta-test and meta-generalization experiments, we further evaluated LTO-CMA on the $\mathbb{D}$, HPO-B and Protein-docking set by directly loading the trained model in their Github Repo. We then compared its performance with our SYMBOL-S. For a fair comparison, we used SYMBOL-S with CMA-ES as the teacher, trained in Section 4.3. The table below reports the average of min-max normalized performance with standard deviation over 5 independent runs. The results underscore that MetaBBO for auto-configuration may inherit limitations from its backbone optimizer, while our SYMBOL generates flexible and novel update rules that generalize well to unseen tasks.

|  | CMA-ES | LTO-CMA | SYMBOL-S |
| --- | --- | --- | --- |
| D | 0.944 (0.018) | 0.956 (0.016) | 0.966 (0.008) |
| HPO-B | 0.917 (0.003) | 0.920 (0.005) | 0.929 (0.020) |
| Protein-docking | 0.979 (0.000) | 0.984 (0.000) | 0.997 (0.000) |

We have added LTO-CMA as a related work in Section 2.

Reviewer comment (Writing):
On page 2, "At the lower lever" should be "At the lower level."

Author response:
**[Minor comments on typos]** We apologize for any confusion caused by typos in our previous submission. We have made the necessary corrections and have thoroughly proofread the revised manuscript.

Reviewer comment (Writing):
In equation (3), the redundant right-side parenthesis exists.

Author response:
**[Minor comments on typos]** We apologize for any confusion caused by typos in our previous submission. We have made the necessary corrections and have thoroughly proofread the revised manuscript.