Paper: TreeDQN: Learning to minimize Branch-and-Bound tree (ICLR-2024)

Reviewer comment (Theory):
Regarding the Assumption in Theorem 4.1: The paper assumes that the probability of having left and right children does not depend on the state because the pruning decision depends on the global upper bound instead of the parent node. However, the global upper bound can change dynamically during the search, which might influence the probability. Does this paper use optimal solutions as upper bounds? Could the authors provide further clarification on this assumption?

Author response:
However, the global upper bound can change dynamically during the search, which might influence the probability. Does this paper use optimal solutions as upper bounds? Could the authors provide further clarification on this assumption?

In our experiments, we use DFS node selection during training and switch to the default node selection strategy during testing. We also experimented with training using the default node selection strategy. We observed that the agent trained with the default node selection strategy performs slightly less, compared to our final setup, due to higher stochasticity and violation of the Markov property of the training environment.

We present Theorem 4.1 to support the statement that a reinforcement learning agent should be able to converge to an optimal policy in a fully observed tree MDP when the probabilities  $p^+$ and $p^-$ do not depend on the state. Indeed, the branching process in the actual B&B solver may violate some initial assumptions like the Markov property or the independence of $p^+$, $p^-$ probabilities from the state. However, if the state does not contain the exact value of the global upper bound, we can not accurately predict the number of child nodes. Thus, we can consider probabilities $p^+$ and $p^-$ independent from the state to some extent. So, if the violation of initial assumptions is small enough, the agent should be able to converge to a well-performing policy, as we demonstrate in our experiments.

Reviewer comment (Evaluation):
Exploring Limited Generalization Ability: In comparing the results presented in Table 5 and Table 3, it is observed that TreeDQN appears to exhibit less stability in the context of transfer tasks. Could you please offer insights or explanations regarding this phenomenon?

Author response:
Exploring Limited Generalization Ability: In comparing the results presented in Table 5 and Table 3, it is observed that TreeDQN appears to exhibit less stability in the context of transfer tasks. Could you please offer insights or explanations regarding this phenomenon?

Thank you for raising an interesting question! We train the TreeDQN agent to optimize the geometric mean of the expected return. So rare, large trees may have a less significant impact on the final policy. It can be seen from the Probability-probability plot for the Maximum Independent Set task (Fig. 4) that the TreeDQN starts falling behind the IL agent when the tasks become harder (in the upper right corner). Thus, in complex transfer tasks, the TreeDQN agent may underperform.

Reviewer comment (Experiments):
A Traditional vs. RL-based Variable Selection Perspective: Traditional variable selection methods rely on human-designed criteria, such as pseudocosts. One advantage of these traditional approaches is their applicability to various problem types. On the other hand, current RL-based methods require training an optimal policy for each specific problem. Given the noted limitations in generalization ability, RL methods seem to necessitate training on problem instances of a similar size as the target problems. Could you provide any comments or insights on the potential implications of this limitation? (This question is optional, and your input is welcomed purely out of curiosity.)

Author response:
A Traditional vs. RL-based Variable Selection Perspective

RL methods can find a new branching strategy that could perform better than the traditional methods for a specific distribution of tasks. We believe that RL methods can improve the performance of Branch-and-Bound solvers in areas that require frequent solving of similar MILPs. For example, when a logistics company ships goods to customers or when we need to allocate limited resources like human workers or computing resources of a data center. At the present time, human-designed heuristics are better for tasks when you need to solve problems with significantly varying complexity and type or have only several task instances of the same kind.  However, the development of multitask and meta-learning approaches may extend the applicability of reinforcement learning methods.

Reviewer comment (Experiments):
Perhaps the biggest limitation is the assumption that the upper bound has to be derivable from the current node or known ahead of time. The authors assert that this (as well as more intricate node selection policies) lead to at most a moderate distribution shift, but never demonstrate this effect.

Author response:
In our experiments we use DFS during training and switch to the default node selection strategy during evaluation. If we assume that the Strong Branching heuristic is close to optimal in our benchmark tasks (Combinatorial Auction, Set Cover, Maximum Independent Set, Facility Location), we can estimate the maximum value of possible performance degradation as the difference between the performance of TreeDQN and Strong Branching:

Combinatorial Auction - 22%, Set Cover - 30%, Maximum Independent Set - 5%, Facility Location - 10%.

Reviewer comment (Experiments):
Another concern is regarding the difficulty distribution of instances. Random instance generation has been known to generate significant amounts of trivial instances compared to real-world equivalents. However, this is a limitation of most prior work on learnt variable selection rules as well.

Author response:
In our work, we focus on the same tasks as previous works (Gasse, NeurIPS 2020, Scavuzzo, NeurIPS 2022) to fairly benchmark our method with methods from the literature. In the updated version of our paper, we added a more challenging Balanced Item Placement task (see general response and updated paper, Appendix D).

Reviewer comment (Experiments):
TreeDQN is also more expensive in terms of wall-clock-time than prior work (especially the IL agent), which can be seen in Figure 4. The paper does not make it clear whether this is due to TreeDQN using a different architecture, or TreeDQN simply creating more expensive nodes during branching.

Author response:
Thank you for spotting this. We use the same Graph Convolutional Neural Network encoder architecture for all models in our benchmarks (IL, tmdp+DFS, FMCTS, TreeDQN). The TreeDQN is creating more expensive nodes during branching. It is clearly seen from the evaluation results on the Balanced Item Placement task (see Appendix D). In this task, all problem instances were finished by reaching a timeout. The TreeDQN agent solves much fewer LPs but achieves a much higher reward and much lower primal bound than the IL agent.

Reviewer comment (Experiments):
An important missing baseline in their comparisons is out-of-the-box SCIP, acting as an automatic state-of-the-art hand-crafted tradeoff between SB and cheaper heuristics.

Author response:
We added SCIP to our evaluation results. Please see the general response and updated paper.

Reviewer comment (Writing):
The paper needs an extensive re-write in terms of argumentation and clarity.

Author response:
Thank you for providing detailed feedback! We corrected the issues. Please see the updated version of our paper.

Reviewer comment (Reproducibility):
Is the method run on CPU or GPU?

Author response:
GPU

Reviewer comment (Experiments):
What is the model architecture (or more importantly: is it the same for all methods)?

Author response:
We use the same Graph Convolutional Neural Network encoder architecture for all models in our benchmarks (IL, tmdp+DFS, FMCTS, TreeDQN).

Reviewer comment (Writing):
Section 2.2 “Tree MDP” needs way more explanation. It more or less assumes familiarity with the Tree MDP work of Scavuzzo et al., and a more self-contained exposition would be very helpful.

Author response:
We updated the background section and extended the description of Tree MDP. Please see the updated version.

Reviewer comment (Theory):
The theoretical contribution is very hazy. Contraction in mean is not really well-motivated. It is unclear whether the cited theorem (Jaakkola ‘93) applies to the setting of tree operators here, as this seems like a nontrivial assumption that is missing justification. Rather than just including a theorem about contraction in mean, the authors should have a main theorem that states the actual convergence guarantee that follows.

Author response:
The convergence of Q-learning methods can be rigorously proved only for tabular MDPs, which is out of the scope of the present paper. To provide an intuition as to why our method should work, we prove the contraction in mean property of the tree Bellman operator.

Reviewer comment (Novelty):
The paper appears methodologically very similar to Scavuzzo et al., and only differs in the mechanics of how the RL algorithm is trained. In Section 4.2, the main difference is that the authors use a loss function that appears to be selectively picked based on the objective of minimizing the geometric mean of the tree sizes during training/testing. This feels like a specific and brittle design choice.

Author response:
Our method has two significant differences from the work of Scavuzzo et.al.:  
1. Use of MSLE loss function. Distribution of tree sizes in the B&B method will always have a long tail (if we can not prune a tree, it will be a complete binary tree with the size growing exponentially with its depth). The MSLE loss function should generally work better than the MSE when predicting the size of the tree because it is more stable to outliers.  
2. Our method is much more sample efficient than the method of Scavuzzo et.al.(tmdp+DFS). To make one gradient update tmdp+DFS needs to solve a batch of MILP tasks until completion. In our approach, we make as many gradient updates as the size of the tree. To further prove the quality of our method, we tested it on a challenging Balanced Item Problem from the ML4CO competition (NeurIPS 2022, see Appendix D). We show that our TreeDQN agent can get a mean reward higher than the Imitation learning baseline. Since solving a single MILP task in this problem takes 15 minutes, training a tmdp+DFS in a reasonable amount of time would be impossible.

Reviewer comment (Experiments):
The new method is shown to yield smaller branch-and-bound trees than previous RL-based variable selection policies, but no comparison is made to the default settings of any state-of-the-art solver (e.g., Gurobi, CPLEX, SCIP). This is an important comparison that should be included.

Author response:
We added evaluation for the SCIP solver with the default set of parameters. Initially, we did not include it because the internal branching rules can make various modifications to the state of the solver, so it can not be considered a direct competitor to other methods. For more information see:  
1. Discussion by Maxim Gasse (https://github.com/ds4dm/ecole/discussions/286#discussioncomment-2317487)  
2. Gerald Gamrath and Christoph Schubert. Measuring the impact of branching rules for mixed-integer programming. In Operations Research Proceedings 2017, pages 165–170. Springer, 2018.

Reviewer comment (Presentation):
The presentation is possibly the paper's weakest point. The lack of clarity makes me wonder about the value of the contributions of the paper. The main contribution, TreeDQN, is explained in a single paragraph in the main text, and since the text only states that the algorithm is an adaptation of Double Dueling DQN, I assume TreeDQN is a straightforward adaptation of DQN to Tree MDPs.

Author response:
Our method is described in 3 pages (pp. 4 - 6). It includes analysis of distributions of tree sizes, proof of convergence of the tree Bellman operator, discussion of our proposed loss function, implementation details (use of Double Dueling DQN adapted for tree MDP process), and training details.

Reviewer comment (Writing):
The background section could be re-written to use less space and pack more information to help the reader understand the work.

Author response:
We updated the background section and improved the description of the Branch-and-Bound algorithm. Please see the updated version.

Reviewer comment (Theory):
In Section 4, the paper lists properties of a successful RL method for this problem, including being off-policy and "work with tree MDP instead of temporal MDP." I am unclear why it is important to learn off-policy, given that we know of many successful on-policy algorithms for RL. Why do they have to work with tree MDPs?

Author response:
We updated this section. Solving MILP tasks is time-consuming. We need sample efficient methods to learn a variable selection policy. Off-policy methods are generally much more sample-efficient than on-policy since they can reuse arbitrary old experiences during training. In the paper, we updated this requirement from "Be off-policy" to “Be sample efficient”.

When solving a MILP task, the Branch-and-Bound solver splits the problem into subproblems and produces a binary tree. To train an RL agent, we need to map this tree to an episode. Considering the whole tree as a single episode under the tree MDP paradigm lets an agent directly optimize the size of the tree.

Reviewer comment (Experiments):
The empirical setting is described in previous papers, and the current paper relies on that. It is unclear how the training data is generated, whether the problems differ in difficulty, and whether we have to optimally solve the problem to attain the Markov property before training the model. If so, how are the problems solved? Assuming the training instances are easy (one needs to solve them optimally), how does the learned heuristic scale to larger problems?

Author response:
For each task we have a distribution of parameters and randomly sample task instances from that distribution. 

Yes. Depending on sampled parameters, the task could be easy (LP relaxation of the initial problem provides an integer feasible solution, so the size of the resulting B&B tree is 1) or require multiple branching decisions to find an optimal solution.

No. In our work, we use DFS during training and the default node selection strategy during testing.

Reviewer comment (Experiments):
I would like to hear clarifications on the empirical setup and how the training of the branching function is done, as listed in the weaknesses section above.

Author response:
For each task we have a distribution of parameters and randomly sample task instances from that distribution. 

Yes. Depending on sampled parameters, the task could be easy (LP relaxation of the initial problem provides an integer feasible solution, so the size of the resulting B&B tree is 1) or require multiple branching decisions to find an optimal solution.

No. In our work, we use DFS during training and the default node selection strategy during testing.