Paper: Enhancing Sample Efficiency in Black-box Combinatorial Optimization via Symmetric Replay Training (ICLR-2024)

Reviewer comment (Theory):
The need for easily accessible symmetric transformations limit the scope of this BBO solver for BBO problem.

Author response:
We agree with your comment that our target problems are grey-box rather than black-box. Our target problem is the CO problem, whose objective function evaluation is expensive. One example can be a CO problem with a black-box objective function and white-box constraints. The title and manuscript have been revised, as mentioned in the general response.

Reviewer comment (Presentation):
(Minor suggestion) It might be better to put all plots in Figure 3 on the same scale if the reader has to compare them side-by-side.

Author response:
We adjusted the y-scale of plots in Figure 3 and also in Figure 6. Thanks for the suggestion.

Reviewer comment (Novelty):
How does this RL-based combinatorial BBO solution relate to combinatorial BBO problem handled by Bayesian Optimization schemes such as COMBO [B]? In their case, they would never propose the same solution twice (which is something any RL-based scheme might be producing, and SRT is removing this repetition). And if external information regarding the symmetries between solutions are available, that can be utilized (as is done in this paper) to generate new simulations and/or skip repetitions.

Author response:
**Answer**: In reinforcement learning, a policy constructs a new solution by sequentially selecting actions from an empty solution. Consider the case of determining a set with three components, where a solution is constructed as follows: [ ] -> [a] -> [a, b] -> [a, b, c]. This sequential construction gives rise to multiple trajectories (e.g., [] -> [b] -> [b, c] -> [b, c, a]), and our approach leverages this symmetry.
On the other hand, in Bayesian optimization, new solutions are suggested by exploring the solution space starting from an initial solution. For instance, by employing Gibbs sampling within a 1-Hamming ball on discrete space, e.g., [a, b, c] -> [d, b, c] -> [d, b, a], there are no such symmetries that we utilize in RL.

Reviewer comment (Novelty):
Are there any existing literature on RL based BBO that discusses or brings up symmetries between action sequences (not necessarily leverages them)?

Author response:
**Answer:** In DevFormer, they introduce an additional loss term to consider solution symmetries; however, these symmetries are not induced in a structured way. On the other hand, GFlowNet introduces a new kind of loss function that considers solution symmetries by modeling the sequential decision-making in CO on a directed acyclic graph (DAG) and considering flows on DAG as a policy. Our two-step learning procedure can be used along with these DRL methods, and SRT can further induce solution symmetries effectively.
It is noteworthy that our algorithm can also easily cooperate with symmetric architectures like SE3 Transformer [6] and EGNN [7], which exhibit architectural symmetries in state-level representation (e.g., rotation of an image yields the same representation). Since these models do not consider solution symmetries related to the action trajectories, further learning processes like SRT are required to induce solution symmetries.

Reviewer comment (Experiments):
Isn't TSP a problem where we can get a reward after every action (and the final reward is the sum of the partial rewards)? What happens if we compare the proposed scheme to such a setup with a RL-based solution?

Author response:
**Answer:** First of all, as a TSP solution, we need to find a Hamiltonian cycle covering all cities. The cost of the solution can vary based on the last segment, which involves returning to the starting point. Note that the last segment is automatically defined on the previous sequence. In other words, the reward is episodic. Training the model with intermediate rewards could introduce bias into the return value, so performance is insufficient. Even in heuristic algorithms for TSP, the farthest insertion, which starts to form a new cycle by inserting the farthest city, outperforms other heuristics. Therefore, the RL policy should be trained using the Monte Carlo method with episodic rewards to ensure unbiased learning.

Reviewer comment (Theory):
Are we looking at symmetric action sequences leading to the same solution $\mathbf{x}$, or are we talking about action sequences that lead to different solutions $\mathbf{x}'$ but $f(\mathbf{x}) = f(\mathbf{x}')$? The TSP example given on page 3 after Definition 1 seem to fit this latter setup. If it is in fact the latter, how does that affect the proposed scheme and the related motivations?

Author response:
**Answer:** In the TSP, the decision variable $x_{ij}$ takes the value of 1 if city $j$ is visited immediately after city $i$. For instance, when considering four cities, the sequences 1-2-3-4-1 and 2-3-4-1-2 represent the same solution (i.e., x is the same). This implies an identical cycle and, apparently, the same decision variables in both sequences. This is what we intended in the example on page 3. Note that a TSP solution (there is no specific starting point) is a Hamiltonian cycle. We have made revisions to the example in the manuscript.

Reviewer comment (Theory):
How is the shifting related to the starting point? Will a different sequence of actions from the same starting point lead to a solution with the same cycle (and hence value) but different order? The reversed order does lead to the same solution with the same initial state, but it is not clear about the other scenarios.

Author response:
**Answer:** In the TSP, the decision variable $x_{ij}$ takes the value of 1 if city $j$ is visited immediately after city $i$. For instance, when considering four cities, the sequences 1-2-3-4-1 and 2-3-4-1-2 represent the same solution (i.e., x is the same). This implies an identical cycle and, apparently, the same decision variables in both sequences. This is what we intended in the example on page 3. Note that a TSP solution (there is no specific starting point) is a Hamiltonian cycle. We have made revisions to the example in the manuscript.

Reviewer comment (Evaluation):
It is unclear where the 20+% improvement is coming in for PPO+SRT over PPO with K=100k (as discussed in Section 4.1). Table 1 lists PPO at $6.77 \pm 0.12$ vs $6.71 \pm 0.02$ for PPO+SRT, which is more like a 1% improvement. The reviewer is unsure if they are looking at the wrong results.

Author response:
**Answer:** There are some typos; we revised the sentence as follows: The most substantial improvement facilitated by SRT is observed in the case of GFlowNet, where a cost reduction of 3.76% is achieved when the sample size K=100K.

Reviewer comment (Evaluation):
In the presence of symmetry, and the access to symmetric transformations, it seems that we should always see improved sample efficiency (or at least no degradation). Under what conditions might we expect degradation (as we see with 2 cases in Table 3 with REINVENT)?

Author response:
**Answer:** Based on the experimental results in Figure 5 (the replay loops are increased), the model can suffer from overfitting when there are not enough symmetries to utilize. For example, unlike TSP, we cannot explicitly get symmetric solutions in the PMO benchmark. The symmetric transformation highly depends on the API; thus, it is hard to explore symmetric space effectively (the maximum entropy is achieved with the uniform symmetric transformation policy according to Theorem 1), resulting in performance degradation due to the early convergence in some cases. This can be mitigated by designing a more precise symmetric transformation policy or adjusting the frequency of conducting SRT.

Reviewer comment (Experiments):
The paper seems to have a scattered focus. While the abstract suggests a focus on expensive black-box combinatorial optimization, the studied TSP problem and most leveraged baselines (like PPO, AM, GFlowNets) neither belongs to black-box optimization nor expensive optimization problems/algorithms.

Author response:
**Answer**
We agree that there were some misalignments in the manuscript. We believe our work contributes to **deep RL for Combinatorial Optimization** fields that require sample efficiency. We have revised the title and abstract for better alignment in the manuscript.

Reviewer comment (Experiments):
The paper lacks a comparison with other methods specifically designed for expensive black-box optimization tasks, such as [1-4], making positioning the paper within the literature quite challenging.

Author response:
**Answer**

Thanks for the insightful comments and suggestions. We have looked through the literature you recommended and added the literature review about black-box combinatorial optimization based on the suggested research in Appendix E. 

First of all, we additionally provide the results of a BO-based (GPBO) and a genetic algorithm (Graph GA) according to your suggestion ([1,2,4] - Bayesian optimization, [3] - evolutionary computation). The table shows the average AUC with three independent seeds. We provide the detailed results in Appendix D.3.

|  | GPBO | Graph GA | REINVENT | REINVENT + SRT |
| --- | --- | --- | --- | --- |
| Avg. AUC | 0.609 | 0.619 | 0.613 | **0.633** |

Here, we would like to discuss why we solve CO problems using reinforcement learning, especially when the function evaluation is expensive. 

-----

**Challenges of applying evolutionary computation algorithms to CO.** The paper [3] introduces evolutionary computation (EC) algorithms, including genetic algorithms (GA). While EC algorithms have a well-established reputation for delivering powerful performance across diverse tasks, they do exhibit certain limitations. As mentioned in [3], *designing novel operators, which greatly affects EC algorithms' performance, requires domain knowledge.* This implies that careful algorithm design is necessitated whenever there is a change in tasks. For example, though Graph GA achieves promising performance, performance in other CO problems will vary depending on how well-designed crossover and mutation operators are.

**Challenges of Bayesian Optimization Algorithms to CO.** As indicated in [1,2,4], Bayesian optimization (BO) is a promising method for solving black-box optimization problems. The key idea is to employ a surrogate model, usually a Gaussian Process, and optimize an acquisition function built upon the surrogate model to draw samples. BO has demonstrated its powerful performance in row-budget regimes, especially in continuous space.
However, in the case of combinatorial spaces, constructing an accurate surrogate model poses a substantial challenge due to its non-smoothness. Furthermore, the acquisition optimization is modeled as quadratic integer programming problems, which is NP-hard [5]. Strategies to address this NP-hardness involve the utilization of variational auto-encoders [6] or continuous relaxation [7]. Yet, these approximations can result in significant performance degradation. 

**Sampling in high-dimension is hard.** It is challenging to sample solutions in CO, as they are high-dimensional. To effectively sample a solution, the joint distribution can be factorized as follows.
$$x\sim p(x) = \prod_{t=1}^T p(x_i|s_t)$$

Sampling a solution from the factorized distribution can also be regarded as sequential decision-making, which RL effectively models. 

**Benefits of solving CO with RL.** In RL for CO, solutions are constructed by sequentially deciding on $x_i$. This constructive policy is advantageous in satisfying the constraints of CO by restricting decisions that lead to infeasible solutions at each step. The fulfillment of constraints is a key property in CO.

Another benefit of this approach is that we can construct a general solver that addresses various target problem instances instead of exclusively seeking optimal solutions for individual target instances, i.e., 
$$x \sim p(x|c) = \prod_{t=1}^T p(x_i|s_t, c)$$

In this research, we train a general solver for the traveling salesman problems and decap placement problems in Sections 4.1 and 4.2.

Reviewer comment (Experiments):
Reinforcement learning methods, especially on-policy ones like PPO, A2C, are criticized for their general sample efficiency. It remains unclear why on-policy RL is necessary for solving expensive black-box optimization problems.

Author response:
**Answer**
Due to the combinatorial nature, 1) rewards are episodic, and 2) value function estimation is notoriously hard. Here, episodic reward means we cannot evaluate objective function (i.e., reward) in the middle of constructing solutions. Therefore, Monte Carlo-based methods, like REINFORCE, have been employed in DRL for CO literature. **When we use Monte Carlo, importance sampling is required for off-policy methods, leading to high variance.** As evidence, the empirical results show that REINVENT, an on-policy method, outperforms off-policy methods such as GFlowNets and DQN in the PMO benchmark. Though on-policy is known to give superior performance in CO, **we also verified our method with an off-policy method, GFlowNet.** We added supplement explanations for GFlowNet in the manuscript.

Reviewer comment (Experiments):
The paper neglects the comparison with other recent research that explicitly focused on improving the sample efficiency of DRL based on data augmentation and experience replay.

Author response:
**Answer**

**(Augmentation)** We agree that data augmentation like cropping [8, 9], noise injection [8, 10], and Mixup [10, 11] can improve sample efficiency. However, these methods focus on visual RL scenarios (continuous space), hence not applicable to CO. In the case of TSP in Euclidean space, POMO and Sym-NCO propose augmenting graph node features by rotating coordination values. As illustrated in Appendix D.4, SRT improves the sample efficiency of Sym-NCO. Note that augmentations in Sym-NCO are done in input space, while ours utilize symmetries in output (action trajectories) space.

**(Experience Replay)** We have conducted additional experiments with A2C on TSP to compare ours to experience replay (ER) as follows. We set the buffer size to 10000, which is the same as the epoch size. Note that when we use experience replay to on-policy, importance sampling is required as follows.

$$E_{a \sim p(\cdot|s_1)}[R(x)] = E_{a \sim q(\cdot|s_1)}[ \frac{p(a|s_1)}{q(a|s_1)}R(x) ] = E_{a \sim q(\cdot|s_1)} [\prod_{t=1}^{T}\frac{p(a_t|s_t)}{q(a_t|s_t)}R(x)],$$
where $p(\vec{a})$ is the training policy and $q(\vec{a})$ is the behavior policy that made the experiences in the replay buffer, and $x=C(\vec{a})$. The importance weight terms of $\prod_{t=1}^{T}\frac{p(a_t|s_t)}{q(a_t|s_t)}$ gives high variances to estimate objective so that gives poor performances on replay training as shown in the following table. The detailed experimental settings and results are also provided in Appendix D.2. 

| Method | 100K | 2M |
| -------- | -------- | -------- |
| A2C      | 6.630 +- 0.037 | 6.115 +- 0.009 |
| A2C + ER (size = 20) | 7.085 +- 0.326 | 6.156 +- 0.033 |
| A2C + ER (size = 100) | 7.073 +- 0.293 | 6.139 +- 0.010 |
| A2C + SRT (ours) | *6.560 +- 0.051* | *6.037 +- 0.005* |


The results show that experience replay can degrade the DRL method due to the high variance introduced by the importance sampling. In contrast, our method does not suffer from the importance sampling because SRT updates the policy to maximize the likelihood of symmetric samples.

Additionally, we provide the results on GFlowNet, off-policy RL, on molecular optimization tasks. Here, the experience replay improves the sample efficiency of GFlowNet, but our method shows more effective results. Average AUC with three independent seeds is reported.

| GFlowNet | + Experience Replay | + Reward-prioritized ER | + SRT |
| -------- | -------- | -------- |-------- |
| 0.436 | 0.491 | 0.497 | *0.511* |

*Note that on-policy RL methods with Monte Carlo estimation have shown notable performance in the CO domain. Though augmentation and experience replay are well-known general approaches to improve sample efficiency, directly applying these into DRL for CO can give insufficient performance gain and even lead to performance degradation.*

Reviewer comment (Novelty):
It is slightly unclear which community the paper could contribute to: DRL for CO, DRL's sample efficiency, expensive optimization, or expensive black-box optimization.

Author response:
**Answer**
We agree that there were some misalignments in the manuscript. We believe our work contributes to **deep RL for Combinatorial Optimization** fields that require sample efficiency. We have revised the title and abstract for better alignment in the manuscript.

Reviewer comment (Novelty):
The key idea is, at least in my opinion, kind of obvious. I'm not an RL expert so I can't comment on how previous works in RL have exploited symmetries, but at least in chemistry the existence of these symmetries is well-known and has been exploited in prior work (e.g. the paper "All SMILES Variational Autoencoder for Molecular Property Prediction and Optimization"). To the authors' credit though, I am not aware of work which has done this specifically for RL methods.

Author response:
**Answer for W1: symmetries in chemical literature**

We agree that there are prior works about symmetries both in RL communities and chemical discovery communities. The major novelty of our works to compare with prior works is we reuse existing samples (which is expensive) given from the RL training phase and exploit prior knowledge of symmetries so that we make "free" samples to make replay training with imitation learning for support RL training more sample efficient. In particular, the paper "All SMILES Variational Autoencoder for Molecular Property Prediction and Optimization" focuses on how to make VAE representations invariant to symmetric transformation while we focus on the architecture agonistic method to enhance sample efficiency of RL training.

Reviewer comment (Experiments):
At least for molecules, the experimental results are not as impressive as they may seem. A recent paper showed that tuning genetic algorithms actually can achieve an average score of `0.639` on the PMO benchmark, which is a bit better than the results shown in this paper (http://arxiv.org/abs/2310.09267). Furthermore, the bolding in Table 3 is a bit misleading: they bold the tasks which achieve the best results just on the methods considered, but not necessarily overall SOTA results on the PMO benchmark. For example, drd2, which the authors bolded with a score of `0.960` is beaten by SynNet with a score of `0.969` (Gao et al 2022).

Author response:
**Answer for W2: Experimental results for molecules may not be as impressive compared to the recent research**

We agree about that. On the scientific discovery side, there can be better candidate algorithms, such as a genetic algorithm. 
On the other hand, RL is a general tool that can cope with general-purpose combinatorial discovery and optimization without the need for specific domain knowledge. In addition, we revised the caption of Table 3 to avoid misleading.

Reviewer comment (Experiments):
In TSP you assume distances are Euclidean. However, is this not an easier subtype of TSP problems? I thought the NP-hard version is occurs when there is no such distance heuristic. I guess it is a toy problem so it doesn't matter much, but it might be more impactful to show result on a harder type of TSP.

Author response:
**Q1: hardness of TSP.** Euclidean TSP is one of the cases of TSP where the distance is defined as Euclidean distance. Generally, the cost function of TSP is defined as $\sum_{i,j} d_{ij}x_{ij}$, and regardless of how to define $d_{ij}$, TSP is NP-hard.

Reviewer comment (Theory):
Theorem 1 was unclear to me: what specifically is $p(x|s_1)$? More generally, the importance of this theorem's result was not clear to me...

Author response:
**Q2: Unclear explanation of Theorem 1.** $p(x|s)$ is the distribution over the solution. Since we have multiple action-trajectories that give the same solution, $p(x|s_1) = \sum_{a \in A_x} \pi_\theta(a|s_1)$ holds
Here, $A_x$ denotes the space of trajectory associated with the solution x, i.e., $C(a) = x$ for all $a \in A_x$. This Theorem gives a theoretical background for employing a randomized symmetric transformation policy, which is advantageous. For example, in TSP, there are $2N$ symmetric trajectories. If we use the fixed symmetric transformation, e.g., flip only, the entropy of the policy is bounded. Therefore, to maximize the entropy of the symmetric transformation policy, we uniformly sample the symmetric trajectories over all possible $2N$ trajectories. The experimental results also support this as follows. Also, we revised the manuscript for better readability.

| Method | K=100K | K=1M | K=2M |
| --- | --- | --- | --- |
| A2C | 6.630 +- 0.037 | 6.514 +- 0.049 | 6.115 +- 0.009 |
| + SRT (fixed) | 6.775 +- 0.079 | 6.198 +- 0.024 | 6.069 +- 0.014 |
| + SRT (ours) | 6.560 +- 0.051 | 6.171 +- 0.025 | 6.037 +- 0.005 |

Reviewer comment (Writing):
Typo: Page 4: "sale of loss function" -> "scale [....]"

Author response:
**Q3:** We revised the typo.

Reviewer comment (Experiments):
The ablation study in Figure 4a is a bit unclear to me. How is it in principle possible to do simultaneous updates, given that we must first collect the high-reward samples in Phase 1? I think the idea of a 2-phase algorithm and alternating steps is inevitable, given that we must fist isolate the high-reward samples before applying the symmetric replay training. The authors could explain this part better to avoid confusion. Did they for example only store the high-reward samples in the replay pool for the simultaneous update, and then performed simultaneous updates once the replay buffer was sufficiently large?

Author response:
**Answer for W3 and Q2: how simultaneous updates were performed.**

In our two-step training, the model is updated to minimize the policy gradient loss in Step A. Then, the model is updated to minimize imitation loss in Step B. These updates are conducted alternatively. On the contrary, in the ablation, after calculating RL loss, we collect the high-rewarded samples using greedy rollout without model updating. We compute $L_{SRT}$ using the high-rewarded samples and then update the model with $L = L_{RL}+L_{SRT}$ at once. To clarify this, we revised the manuscript.

Reviewer comment (Experiments):
Have the authors tried other symmetric architectures besides the DevFormer? Does symmetric replay training on top of symmetric policy and value network yield additional benefit, and why? The proposed technique could complement other ideas such as symmetric NNs, so their combined impact would be interesting to understand.

Author response:
**W2, Q1: why does applying SRT on top of such symmetric architectures have additional benefits?**

Solution symmetries, defined as situations where multiple trajectories can lead to identical solutions, are challenging to incorporate directly into architectural structures. Therefore, loss optimization is needed to induce solution symmetries. In DevFormer, they introduce an andditional loss term to consider solution symmetries; however, these symmetries are not induced in a structured way. On the other hand, GFlowNet introduces a new kind of loss function that considers solution symmetries by modeling the sequential decision-making in CO on a directed acyclic graph (DAG) and considering flows on DAG as a policy. Our two-step learning procedure can be used along with these DRL methods, and SRT can further induce solution symmetries effectively.
It is noteworthy that our algorithm can also easily cooperate with symmetric architectures like SE3 Transformer [1] and EGNN [2], which exhibit architectural symmetries in state-level representation (e.g., rotation of an image yields the same representation). Since these models (including the permutation-invariant NN) do not consider solution symmetries related to the action trajectories, further learning processes like SRT are required to induce solution symmetries.

Reviewer comment (Experiments):
Can the authors explain how simultaneous updates were performed in Figure 4a? Did they first conduct standard RL to fill the replay buffer with high-reward samples, and then switched to simultaneous updates? This was unclear from the text.

Author response:
**Answer for W3 and Q2: how simultaneous updates were performed.**

In our two-step training, the model is updated to minimize the policy gradient loss in Step A. Then, the model is updated to minimize imitation loss in Step B. These updates are conducted alternatively. On the contrary, in the ablation, after calculating RL loss, we collect the high-rewarded samples using greedy rollout without model updating. We compute $L_{SRT}$ using the high-rewarded samples and then update the model with $L = L_{RL}+L_{SRT}$ at once. To clarify this, we revised the manuscript.

Reviewer comment (Evaluation):
The paper emphasizes the fact that evaluating the black box function is an expensive operation. However, Table 8 shows that the proposed method can outperform other sample-efficient algorithms such as Sym-NCO in the synthetic settings, where function evaluation is fast and cheap. In that case, it seems that symmetric replay training could be positioned as a general technique for improved sample efficiency that can be used not only with black-box optimization but also more generally. The authors state "broad spectrum of CO problems and methods" and "a new generic DRL method" in the main text, so perhaps symmetric replay training is positioned as a general-purpose technique. On the other hand, the black-box nature (with expensive function evaluations) is mentioned several times and even in the title. It would make sense for the authors to disambiguate this point and position their method very clearly (i.e., as a general-purpose algorithm or a technique that is better suited for black box optimization with expensive function evaluations).

Author response:
**Answer for Q3: overemphasizing black-box nature (SRT is positioned as a general-purpose technique)**

We agree that our SRT can give general-purpose replay training on CO. Our intention was that sample efficiency matters to real-world applications of CO, such as hardware optimization and molecule optimization. We revised the manuscript based on your suggestion.