Paper: Learning to Branch with Offline Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
This paper proposes training branch-and-bound strategies using offline reinforcement learning. However, in practice, interacting with solvers is relatively straightforward, and under these circumstances, using online reinforcement learning may yield better performance. The authors need to clarify the necessity of utilizing offline reinforcement learning.

Author response:
We want to thank reviewer pc5v for your questions and constructive suggestions. Here are our responses to all of your comments.

 > The authors need to clarify the necessity of utilizing offline reinforcement learning.

We have briefly discussed the comparison in the third paragraph in our introduction part, here we will try to further explain why using offline reinforcement learning is meaningful.

1. Training reinforcement learning for branching is super time-consuming [1,2]. It is because interacting with the solver could also be slow (long interaction time) and the BnB tree could be super large when a RL policy is very bad at the beginning. Here we cite the training time from two recent RL for branching papers, `We set a maximum of 15,000 epochs and a time limit of six days for training` and `we left our retro branching agent to train for ≈ 13 days (≈ 500k epochs)`. In comparison, the offline models can all be trained within hours.

2. Bad performance of current RL methods. The results from the current two papers demonstrate that their performance is still way worse the imitation learning methods. In Table 2 and 3, we have also shown that the RL agent (tMDP) is even worse than RCAC which is trained over sub-optimal or small datasets.

3. There are lots of known problems in training online RL agents for branching, as summarized in [2]. For example, existing RL methods choose the size of the search tree as the reward function, which leads to a sparse reward environment and that is why we propose to use the improvement in the dual-bound as the reward function. A subsequent problem is the credit assignment problem (how to tell which action leads to the good/bad outcomes), which is studied by [1], but the the model performance shows that this problem is still far from being solved.

[1] Lara Scavuzzo, Feng Yang Chen, Didier Chetelat, Maxime Gasse, Andrea Lodi, Neil Yorke-Smith, and Karen Aardal. Learning to branch with tree MDPs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.

[2] Christopher W. F. Parsonson, Alexandre Laterre, and Thomas D. Barrett. Reinforcement learning for branch-and-bound optimisation using retrospective trajectories. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):4061–4069, Jun. 2023.

Reviewer comment (Experiments):
Considering that interacting with solvers online is convenient, is there a necessity to use offline reinforcement learning to train branch-and-bound strategies?

Author response:
We want to thank reviewer pc5v for your questions and constructive suggestions. Here are our responses to all of your comments.

 > The authors need to clarify the necessity of utilizing offline reinforcement learning.

We have briefly discussed the comparison in the third paragraph in our introduction part, here we will try to further explain why using offline reinforcement learning is meaningful.

1. Training reinforcement learning for branching is super time-consuming [1,2]. It is because interacting with the solver could also be slow (long interaction time) and the BnB tree could be super large when a RL policy is very bad at the beginning. Here we cite the training time from two recent RL for branching papers, `We set a maximum of 15,000 epochs and a time limit of six days for training` and `we left our retro branching agent to train for ≈ 13 days (≈ 500k epochs)`. In comparison, the offline models can all be trained within hours.

2. Bad performance of current RL methods. The results from the current two papers demonstrate that their performance is still way worse the imitation learning methods. In Table 2 and 3, we have also shown that the RL agent (tMDP) is even worse than RCAC which is trained over sub-optimal or small datasets.

3. There are lots of known problems in training online RL agents for branching, as summarized in [2]. For example, existing RL methods choose the size of the search tree as the reward function, which leads to a sparse reward environment and that is why we propose to use the improvement in the dual-bound as the reward function. A subsequent problem is the credit assignment problem (how to tell which action leads to the good/bad outcomes), which is studied by [1], but the the model performance shows that this problem is still far from being solved.

[1] Lara Scavuzzo, Feng Yang Chen, Didier Chetelat, Maxime Gasse, Andrea Lodi, Neil Yorke-Smith, and Karen Aardal. Learning to branch with tree MDPs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.

[2] Christopher W. F. Parsonson, Alexandre Laterre, and Thomas D. Barrett. Reinforcement learning for branch-and-bound optimisation using retrospective trajectories. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):4061–4069, Jun. 2023.

Reviewer comment (Theory):
In Equation 7, when k is small, the distribution of Q-values over the dataset will be centered around -δ, which is unfavorable for training. How do the authors ensure training effectiveness in this scenario?

Author response:
Thanks for this great question. Actually, we do not directly fit the output of critic network $Q$ to $-\delta$. Note in Equation 8, we only change the output of the target network to $-\delta$. We realize this by first letting the target network output its value, then replace some of those outputs with $-\delta$. The target network needs no training, so it will not be fit to a distribution with values centered around $-\delta$. Then the fitting target of $Q$ in Equation 7 is also not a distribution with values centered around $-\delta$.

Reviewer comment (Experiments):
I believe that the essence of the method proposed by the authors lies in further refining the dataset, specifically selecting the top-k actions in Gω for Bellman operator operations. I am curious to know if, after obtaining the top-k actions in Gω, simple imitation learning on these state-action pairs would yield similar results as the current approach. In other words, my question is whether the key to the effectiveness of this algorithm lies in the dataset refinement rather than offline reinforcement learning. I suggest that the authors conduct further ablation experiments to validate this idea.

Author response:
We think there could be some misconceptions here. $G_{\omega}$ is also a neural network rather than a set, so we actually do not do any dataset refining, i.e., filtering out good transitions from the dataset for training. The ranking is conducted on the scores of the $G_{\omega}$, so the top-k actions do not necessarily have to be in the collected dataset, that is, we do not observe the reward for all these top-k actions. But we can still evaluate these transitions with our learned Q-network.

We believe the most relevant results are in Table 5, since $G_{\omega}$ has already been a neural network, simply imitating $G_{\omega}$ is just a distillation of the model. The better performance of RCAC over $G_{\omega}$ has already demonstrated that the source of improvement is from offline reinforcement learning.

Reviewer comment (Writing):
The paper is a little unclear at some points. For instance, in the last paragraph of Section 2.2: Which variables are the selected ones? Just from the node chosen by the node selection policy, or all variables across the entire tree? In general, the distinction between node selection and variable selection doesn’t become clear: Does the method also do node selection (by picking variables from the entire tree), or just variable selection?

Author response:
Only the node chosen by the node selection policy is used. The node selection policy is fixed. Thanks for the question and we will make this part clear.

Reviewer comment (Experiments):
It is not exactly clear whether there is a single model trained and evaluated on all instances, or multiple independent models trained on and evaluated on individual datasets.

Author response:
We have multiple independent models trained on and evaluated on individual datasets. Actually, the description of the experimental settings exactly follows the descriptions in the previous works, but we will add more clarifications in our final version.

Reviewer comment (Experiments):
The testing set is rather small: 10k training instances, 2k validation instances and 20 test instances is a strange ratio.

Author response:
This ratio is actually a common practice utilized in existing works. 20 test instances x 5 seeds leading to 100 measurements. We follow your suggestion to expand the number of testing instances to 100 in Table 9 and 10, which leads to 500 measurements. The reason for such a small testing set is due to its time measurement, which could be inaccurate under parallel computing, and we believe this is a common problem in the existing research.

Reviewer comment (Evaluation):
The reward function is a little bit strange: Why consider the dual bound, but ignore the primal one completely? Further, these bounds are not scale-invariant, meaning that the same problem, modulo a constant scalar, could have different dual bound improvements. Even if one takes care to normalize the objective vector c beforehand, most solvers like SCIP rescale this vector for increased numerical stability. Depending on which problems are chosen, the range of rewards across different instances might also be massive depending on the duality gap. However, it is agreed with the authors that this metric is still better than tree-size or number of nodes.

Author response:
First, this is not a weird choice since the dual-integral is also a common metric to evaluate the quality of the branching policy, where only the dual bound is used. Second, dual-bound improvement is used to compute the score in the strong branching, which is empirically proven to be a powerful branching policy. It can somewhat directly reflect the quality of the instant branching decision leading to it while primal-dual gap cannot. Finally, the improvement from the primal bound is highly related to the primal heuristics, and is sometimes hard to fit with the node features.

Reviewer comment (Writing):
Which set of variables is being selected from?

Author response:
Only the node chosen by the node selection policy is used. The node selection policy is fixed. Thanks for the question and we will make this part clear.

Reviewer comment (Experiments):
What is the performance of other offline-RL algorithms?

Author response:
See Table 13 and 14 for the comparison with CQL.

Reviewer comment (Experiments):
Can you evaluate on a larger test set?

Author response:
This ratio is actually a common practice utilized in existing works. 20 test instances x 5 seeds leading to 100 measurements. We follow your suggestion to expand the number of testing instances to 100 in Table 9 and 10, which leads to 500 measurements. The reason for such a small testing set is due to its time measurement, which could be inaccurate under parallel computing, and we believe this is a common problem in the existing research.

Reviewer comment (Evaluation):
Why only look at the dual bound improvement (alternative: optimality gap between primal and dual)?

Author response:
First, this is not a weird choice since the dual-integral is also a common metric to evaluate the quality of the branching policy, where only the dual bound is used. Second, dual-bound improvement is used to compute the score in the strong branching, which is empirically proven to be a powerful branching policy. It can somewhat directly reflect the quality of the instant branching decision leading to it while primal-dual gap cannot. Finally, the improvement from the primal bound is highly related to the primal heuristics, and is sometimes hard to fit with the node features.

Reviewer comment (Theory):
In Section 3.2: “In fact, a good action does no harm to policy optimization even if it is an OOD action” – request for elaboration on this point.

Author response:
Offline RL treats those actions with a low frequency in the dataset as OOD actions and avoids the query on these actions since the value estimation on these actions is inaccurate due to the lack of samples. However, if we can verify an action is good, for example, it brings a high dual-bound improvement, then we do not have to worry about its inaccurate value estimation. If its value is underestimated, this action will not be chosen by our actor, then there is no difference. If its value is over-estimated, our actor will choose this action, but since we have verified through the dual-bound improvement that it is highly possible to be a good action, it still brings no harm.

Reviewer comment (Novelty):
The authors claim that the proposed method is the first attempt to apply the offline RL algorithms to MILP solving. However, I found one previous work [1] applies offline RL methods to branching strategies learning as well.

Author response:
The difference between the mentioned paper and ours is actually discussed at the end of section 5.2. Though the name of the mentioned method is similar to our method, we want to point it out that these two methods are actually orthogonal.

First, our assumptions are totally different. Our key assumption is that the interaction with the solver could be expensive, no matter through strong branching (Table 1) or the RL policy (the last part in our general response). This is also the basic assumption for current offline RL algorithms. Howeve, [1] assumes a super cheap cost of using strong branching. They not only collect the strong branching demos, but also use strong branching to roll out the trajectory at each step. 

Consequently, our research questions are also different. [1] solely focuses on improving the performance of neural branching, while we focus on the efficient training and learning from suboptimal data, which we believe is what existing offline RL algorithms do.

Finally, two methods are actually orthogonal. [1] improves the algorithm for dataset collection while we improve the training algorithm. Though [1] claims they formulate the problem as an offline RL problem, they do not utilize any offline RL algorithm as we do. Therefore, we claim we are the first to `apply the offline RL algorithms to MILP solving`.

Reviewer comment (Novelty):
The authors may want to explain the novelty of their method over the work [1] in detail.

Author response:
The difference between the mentioned paper and ours is actually discussed at the end of section 5.2. Though the name of the mentioned method is similar to our method, we want to point it out that these two methods are actually orthogonal.

First, our assumptions are totally different. Our key assumption is that the interaction with the solver could be expensive, no matter through strong branching (Table 1) or the RL policy (the last part in our general response). This is also the basic assumption for current offline RL algorithms. Howeve, [1] assumes a super cheap cost of using strong branching. They not only collect the strong branching demos, but also use strong branching to roll out the trajectory at each step. 

Consequently, our research questions are also different. [1] solely focuses on improving the performance of neural branching, while we focus on the efficient training and learning from suboptimal data, which we believe is what existing offline RL algorithms do.

Finally, two methods are actually orthogonal. [1] improves the algorithm for dataset collection while we improve the training algorithm. Though [1] claims they formulate the problem as an offline RL problem, they do not utilize any offline RL algorithm as we do. Therefore, we claim we are the first to `apply the offline RL algorithms to MILP solving`.

Reviewer comment (Experiments):
The experiments are insufficient. First, the authors may want to evaluate their method on the load balancing dataset from the ML4CO competition as well. Second, the baselines are insufficient. The authors may want to compare their method to the work [1]. Third, the authors may want to evaluate the generalization ability of the learned models.

Author response:
> First, the authors may want to evaluate their method on the load balancing dataset from the ML4CO competition as well. Third, the authors may want to evaluate the generalization ability of the learned models.

The reason we do not evaluate our method on the load balancing dataset is because our assumption is that strong branching is costly to use and we do not have enough good demos. But load balancing is not in this scope, where thousands of strong branching demos could be easily collected in minutes. But we believe 4 datasets have already been enough given most previous works only use 4 or 5 datasets.

> Second, the baselines are insufficient. The authors may want to compare their method to the work [1].

As we discussed above, the methods are orthogonal and thus not comparable. [1] is innovative in their dataset collection algorithm while their training algorithm is still similar to imitation learning. 


> Third, the authors may want to evaluate the generalization ability of the learned models.

Thanks for the suggestion. See Table 11 and 12 in our updated version.

Reviewer comment (Novelty):
Why choose the proposed algorithm over any existing offline RL algorithm like CQL[1], IQL etc.?

Author response:
Using a hard constraint (ranking) rather than the soft constraint adopted in many previous offline RL algorithms, such as CQL. This is because branching performance is somewhat sensitive to the branching decisions, a very bad branching decision would cause a huge subsequent BnB tree. Therefore, the offline RL agent should be trained more conservatively and that is why we use ranking as the constraint.

Please also see Table 13 and 14 for the comparison with CQL in our updated version.

Reviewer comment (Theory):
What are connections of equation 6 to reward weighed regression?

Author response:
Equation 6 can be treated as a coarse-version of the reward weighed regression, since we only use the instant reward rather than the cumulative reward. Using instant rewards does not always make sense in the general RL formulation, but it works here since dual-bound improvement itself can reflect the quality of the branching decisions.

Reviewer comment (Experiments):
**Lack of Scaling-Generalization Results:** A key aim of collecting datasets on smaller instances is to develop policies that excel on larger, more complex instances. It would be beneficial to see how various models perform on scaled-up versions of instances in various problem categories like SC, MIS, CA, or CFL. How do these policies perform on Medium or Hard instances (scaled-up versions) in SC, MIS, CA, or CFL? Does RCAC retain its performance advantage on scaling up to larger instances?

Author response:
See Table 11 and 12 in our updated version

Reviewer comment (Experiments):
**Insufficient Comparison with Existing Methods – GGCN Framework:** The paper lacks a thorough comparison with recent advancements in the GGCN framework, particularly the augmented loss function introduced in "Lookback for Learning to Branch" (Gupta et al. 2022, https://arxiv.org/abs/2206.14987). It would be insightful to see how RCAC compares to this improved GGCN variant.

Author response:
This paper focuses more on a better imitation of the strong branching, which is a bit different from what we study. Besides, the code for this paper is not public, we will try to ask the authors for the code or reproduce it in the final version.

Reviewer comment (Experiments):
**Insufficient Comparison with Existing Methods – RCAC(S) vs GGCN(S) Training Approaches:** If I understand correctly, RCAC (S) and GGCN (S) primarily differ in their approach to training despite similarities in other aspects, such as dataset collection. Specifically, GGCN (S) employs a Cross-Entropy loss function, while RCAC (S) is focused on learning a Q-function (and a corresponding policy). The distinctiveness of the RCAC framework lies in its utilization of rewards instead of directly using FSB selections, as is the case with GGCN. However, an alternative comparison could involve integrating rewards into the GGCN framework as an additional signal. This could be achieved, for instance, by employing rewards to modulate the Cross-Entropy loss at each node, similar to how node depth might be used. Demonstrating RCAC's superior performance in this modified context would further reinforce the effectiveness of its RL-based approach as formulated in the study.

Author response:
Actually, $G_{\omega}$ is trained in the way you suggest. Our result in Table 5 indicates that $G_{\omega}$ has already brought some improvements, but RCAC can further boost the performance. Also, to ensure the reward if informative, we choose the dual-bound rather than the tree size used in previous RL for branching methods.

Reviewer comment (Experiments):
**Comparisons with Other RL Methods:** Comparisons with other RL methods, especially in terms of dataset size and time efficiency, would also be valuable.

Author response:
We actually discuss it in Section 4.2. Here we cite the training time from two recent RL for branching papers, `We set a maximum of 15,000 epochs and a time limit of six days for training` [1] and `we left our retro branching agent to train for ≈ 13 days (≈ 500k epochs)` [2] . In comparison, the offline models can all be trained within hours.

[1] Lara Scavuzzo, Feng Yang Chen, Didier Chetelat, Maxime Gasse, Andrea Lodi, Neil Yorke-Smith, and Karen Aardal. Learning to branch with tree MDPs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.

[2] Christopher W. F. Parsonson, Alexandre Laterre, and Thomas D. Barrett. Reinforcement learning for branch-and-bound optimisation using retrospective trajectories. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):4061–4069, Jun. 2023.

Reviewer comment (Writing):
**Clarification – Section 3.3 Wording:** Should "representation of the B&B tree" be replaced with "representation of the B&B node" for accuracy?

Author response:
Yes, thanks for the suggestion and we have updated it.

Reviewer comment (Experiments):
**Clarification – S Method Training:** Is the S method trained with only 5K transitions?

Author response:
Yes.

Reviewer comment (Experiments):
**Clarification – Reward Distribution:** Could the authors provide details on the distribution of reward values in the dataset, perhaps in the Appendix? Information on how this varies with tree depth and how normalization is handled would be valuable.

Author response:
See Figure 4 in our updated version.

Reviewer comment (Presentation):
**Clarification – Figure 3 Problem Family:** What is the specific problem family represented in Figure 3?

Author response:
As the title suggests, it is the CA (combinatorial auction) problem. We change the ranking constraint $k$ and report of performance change of RCAC.

Reviewer comment (Experiments):
**Clarification – Practicality of H Dataset Collection:** Given that VHB takes longer than FSB (as indicated in column 2), is it still a practical choice since the performance is worse than S?

Author response:
The answer could also be found in the paragraph we refer to in the response to clarification 2. In short, VHB is used to simulate the scenario where the best available heuristics are sub-optimal. Since on these 4 commonly used datasets, SB has already been strong enough, we use VHB to simulate a sub-optimal policy and evaluate the performance of RCAC. Besides, models also have a better performance when trained on the dataset collected by VHB on the CFL problem, so it could also sometimes become a better choice.

Reviewer comment (Writing):
**Clarification – GGCN Abbreviation:** Could the authors clarify the abbreviation GGCN? It seems to be a variation of GCNN (Graph Convolutional Neural Networks) as used in Gasse et al. 2019.

Author response:
We actually use GGCN to refer to the method used in Gasse et al. 2019. Since GCN or GCNN is more like a model name, which is also adopted in our learning method, to discriminate the learning method name from the model name, we try to use the author name G(asse)GCN to name the method.

Reviewer comment (Reproducibility):
**Clarification – Inference Procedure in RCAC:** Are there two forward passes \( G_\omega \) and \( \pi_\phi \) during inference in RCAC? How does this differ from the inference process in GGCN?

Author response:
Only the actor is used during the inference in the current results, as we state in Section 3.2, `During inference, the action arg maxa $\pi_{\phi}(a|s)$ is used at each step. Our algorithm could be summarized as follows`. Note that the actor has already been trained over the Q-values in Equation 7 to avoid those OOD actions, so $G_{\omega}$  is no longer needed. However, we actually observe some improvements when $G_{\omega}$ is used udring the inference time. Theoretically, the inference time remains unchanged if $\pi$ and $G_{\omega}$ are run in parallel, we may add this result in the final version.

Reviewer comment (Reproducibility):
**Clarification – Hyperparameter \( k \) in Figure 3:** Figure 3 suggests that \( k \) has a significant impact on RCAC's performance. Could the authors provide the \( k \) values used for each model and dataset?

Author response:
See Table 7 in our updated version.

Reviewer comment (Evaluation):
**Clarification – Aggregation in Table 4:** How are scores aggregated across 20 instances in Table 4? Assuming this is a cumulative sum, RCAC appears to outperform in WA but not against RPB in AP. Can the authors speculate on which problem types might be more amenable to improvement by RCAC?

Author response:
This is actually the average reward across 20 instances (though the number looks quite large), details about the reward can be found in https://www.ecole.ai/2021/ml4co-competition/ (in one word, negated unshifted dual integral).

Anonymous Problem (AP) is a heterogeneous dataset where problems have different types (for example, it could be a mixture of SC, MIS, CA, CFL and other problems as well). The generalization of such a dataset could be hard since different problems may have different preferences for branching. RCAC can bring improvement to the dataset with a clear distribution.

Reviewer comment (Evaluation):
**Clarification – Reward Ablation Choice:** Could the authors discuss the rationale behind choosing dual bound improvement over primal-dual gap improvement? Understanding the preference for one metric over the other would be enlightening.

Author response:
Dual-bound improvement is used to compute the score in the strong branching, which is empirically proven to be a powerful branching policy. It can somewhat directly reflect the quality of the instant branching decision leading to it while the primal-dual gap cannot. Besides, the improvement from the primal bound is highly related to the primal heuristics and is sometimes hard to fit with the node features.

Reviewer comment (Presentation):
**Suggestion – Dataset Comparison Visualization:** I think it will be pretty helpful to have a section or a figure demonstrating the difference (transition vs. individual nodes) between the dataset collected using the standard IL methods and the one proposed in this work.

Author response:
Thanks for the suggestion. Due to the page limit, we will consider adding such a Figure in our final version.

Reviewer comment (Evaluation):
**Suggestion – Statistical Significance Reporting:** Please include p-values to indicate the statistical significance of differences in Tables 2 and 3.

Author response:
See Table 8 in our updated version.

Reviewer comment (Evaluation):
**Suggestion – Evaluation Methodology and Seeds:** Given that 20 seems a relatively small sample size for testing, it's common practice to evaluate each instance with multiple seeds, as demonstrated in Gasse et al. 2019. Could the authors clarify whether a similar approach can be employed in their study?

Author response:
Yes. We also increased the number of testing instances to 100 in Tables 9 and 10 in the updated version.