Learning to Branch with Tree MDPs
Abstract
State-of-the-art Mixed Integer Linear Program (MILP) solvers combine systematic tree search with a plethora of hard-coded heuristics, such as the branching rule.
The idea of learning branching rules from data has received increasing attention recently, and promising results have been obtained by learning fast approximations of the strong branching expert.
In this work, we instead propose to learn branching rules from scratch via Reinforcement Learning (RL).
We revisit the work of Etheve et al. [11] and propose tree Markov Decision Processes, or tree MDPs , a generalization of temporal MDPs that provides a more suitable framework for learning to branch.
We derive a tree policy gradient theorem, which exhibits a better credit assignment compared to its temporal counterpart.
We demonstrate through computational experiments that tree MDPs improve the learning convergence, and offer a promising framework for tackling the learning-to-branch problem in MILPs.
Introduction
Mixed Integer Linear Programs (MILPs) offer a powerful tool for modeling combinatorial optimization problems, and are used in many real-world applications [30].
The method of choice for solving MILPs to global optimality is the Branch-and-Bound (B&B) algorithm, which follows a divide-and-conquer strategy.
A critical component of the algorithm is the branching rule, which is used to recursively partition the MILP search space using a space search tree.
While there is little understanding of optimal branching decisions in general settings [25], the choice of the branching rule has a decisive impact on solving performance in practice [3].
In modern solvers, state-of-the-art performance is obtained using hard-coded branching rules that rely on heuristics designed by domain experts to perform well on a representative set of test instances [18].
In recent years, increasing attention has been given to approaches that use Machine Learning (ML) to obtain good branching rules and improve upon expert-crafted heuristics [5]. Such a statistical B&B tree a b c d e f g h i
Tree MDP episode a b c d e f g h
i
Temporal MDP episodes a c g b d f h i e a c f h i g b d e a b c d e f g h i approach is particularly well-suited in situations where similar problems are solved repeatedly, which is a common industrial scenario.
In such cases, a generic and problem-agnostic branching rule might be suboptimal.
Due to the sequential nature of branching, a natural paradigm to formulate the learning problem is Reinforcement Learning (RL), which allows to directly search for the optimal branching rule with respect to a metric of interest, such as average solving time, or ﬁnal B&B tree size.
Etheve et al.
[11] show that, when using depth-ﬁrst-search (DFS) as the tree exploration (a.k.a. node selection) policy in B&B, minimizing each subtree size is equivalent to minimizing the global tree size.
Using this result, they propose an efﬁcient value-based RL procedure for learning to branch.
In this paper we build upon the work of Etheve et al.
[11], and we propose a generalization of the
Markov Decision Process (MDP) paradigm that we call tree MDP (illustrated in Figure 1, formally discussed in Section 4).
This general formulation captures the key insight of their approach, but opens the door for more general B&B settings, RL algorithms and reward functions.
In particular, we show that DFS is just one way to enforce the “tree Markov” property, and we propose an alternative, more practical solution that simply relies on providing the optimal objective limit to the solver during training.
Our contribution is three-fold:
We introduce the concepts of a tree MDP and of the tree Markov property, and we derive a policy gradient theorem for tree MDPs.
We propose an alternative way to enforce the tree Markov property in learning to branch by imposing an optimal objective limit, which is less computationally demanding than DFS [11].
We show that adopting the tree MDP paradigm improves both the convergence speed of RL and the quality of the resulting branching rules, compared the regular MDP paradigm.
The remainder of this paper is organized as follows.
We formally introduce MILPs, B&B and MDPs in Section 2.
In Section 3 we motivate the need for moving beyond imitation learning in learning to branch, and discuss the challenges it raises.
In Section 4 we introduce the concepts of tree MDPs and the tree Markov property, we derive a policy gradient theorem for tree MDPs, and we present a convenient solution to enforce the tree Markov property in B&B without DFS [11].
In Section 5 we conduct, for the ﬁrst time, a thorough computational evaluation of RL for learning branching rules, on ﬁve synthetic MILP benchmarks with two difﬁculty levels.
Finally, we discuss the signiﬁcance of our results and future directions in Section 6.
Background
In this section, we describe the Branch-and-Bound (B&B) algorithm, and we show how the branching problem naturally formulates as a (temporal) Markov Decision Process.
2.1 The B&B algorithm
Consider a Mixed Integer Linear Program instance as an optimization problem of the form
[Equation]
where c ∈ R n is the objective coefﬁcient vector, A ∈ R m × n is the constraint matrix, b ∈ R m is the constraint right-hand-side, l, u ∈ R n are the lower and upper bound vectors, respectively, and
I ⊆ { 1 , 2 , ..., n } is the subset of variables constrained to take integer values.
Relaxing the integrality constraints, x i ∈ Z ∀ i ∈ I , yields a Linear Program (LP) that can be solved efﬁciently in practice, for example by using the Simplex algorithm.
Solving such a relaxation yields an LP solution ˆ x (cid:63) , and provides a lower bound c T
ˆ x (cid:63) to the original problem (1).
If by chance ˆ x (cid:63) satisﬁes the integrality constraints, ˆ x (cid:63)i ∈ Z ∀ i ∈
I , then it is also an optimal solution to (1).
If, on the contrary, there is at least one variable x
i , i ∈ I such that ˆ x (cid:63)i is not an integer, then one can split the feasible region into two sub-problems, by imposing
[Equation]
This partitioning step is known as branching.
In its most basic form, the vanilla B&B algorithm recursively applies branching, thereby building a search space tree where each node has an associated local sub-MILP, with a local LP solution and associated local lower bound.
If the local LP solution satisﬁes the MILP integrality constraints, then it is termed a feasible solution , and it also provides an upper bound to (1).
At any given time of the B&B process, a Global Lower Bound (GLB) to the original MILP can be obtained by taking the lowest of local lower bounds of the leaf nodes of the tree.
Similarly, a Global Upper Bound (GUB) can be obtained by taking the lowest of the upper bounds so far.
The B&B process terminates when no more B&B leaf node can be partitioned with (2), i.e., all leaf nodes satisfy one of these conditions: the local LP has no solution (infeasible); the local lower bound is above the GUB (pruned); or the local LP solution satisﬁes the MILP integrality constraints (integer feasible).
At termination, we have GLB = GUB , and the original MILP (1) is solved.
The branching problem, a.k.a.
variable selection problem, is the task of choosing, at every B&B iteration, the variable x i that will be used to generate a partition of the form (2).
While there is no universal metric to measure the quality of a branching rule, a common performance measure is the ﬁnal size of the B&B tree (the smaller the better)
2.2 Temporal MDPs
In the following, upper-case letters in italics denote random variables (e.g. S, A ), while their lowercase counterparts denote their value (e.g. s, a ) and their calligraphic counterparts their domain (e.g., s ∈ S , a ∈ A ).
We consider episodic Markov Decision Processes (MDPs) of the form
M = ( S , A , p init , p trans , r ) , with states s ∈ S , actions a ∈ A , initial state distribution p init ( s 0 ) , state transition distribution p trans ( s t +1 | s t , a t ) , and reward function r : S → R .
For simplicity, we assume ﬁnite episodes of length T , that is, τ = ( s 0 , a 0 , . . .
, s T ) , | τ | =
T .
Together with a control mechanism represented by a stochastic policy π ( a t | s t ) , the MDP deﬁnes a probability distribution over trajectories, namely
[Equation]
The MDP control problem is to ﬁnd a policy that maximizes the expected cumulative reward, π (cid:63) ∈ arg max π V π , with
[Equation]
A key property of MDPs is the temporal Markov property S t +1 ⊥⊥ S <t | S t , A t , ∀ t , which guarantees that the future only depends upon the current state and action.
One consequence of this property is the temporal policy gradient theorem [34], which forms the basis of policy gradient methods [Equation]  [Equation]
t = ∇
[Equation]
log π (
[Equation]
t | s t ) | τ | (cid:88) t (cid:48)
=
t + r ( s t (cid:48) ) 
 . 2.3 The branching temporal MDP
Let us now consider the problem of learning a branching policy in a B&B solver.
As noted by [21, 17], branching decisions are made sequentially, thus the problem can naturally be regarded as a temporal
MDP.
In this form, the states s t consist of the entire state of the B&B process (that is, the solver) at time t , which includes the whole tree structure, all sub-MILPs and LP solutions, and all upper and lower bounds.
The actions a t are the branching decisions, and the reward is chosen so that the return matches an objective function of interest (e.g., the ﬁnal B&B tree size).
While appealing by its simplicity, we argue that this formulation is not practical for two reasons.
First, even in the simplest textbook implementation of B&B, the MDP states are complex objects of growing size, which are impractical to handle.
Second, episode length in branching can grow extremely large, in the worst case exponentially with the size of the problem.
This exacerbates the so-called credit assignment problem , i.e., the problem of determining which actions should be given credit for a certain outcome (see Section 3.2 for a more detailed discussion).
In Section 4, we will we will show how the tree MDP formulation addresses both those challenges.
Approaches to learning to branch
Approaches that frame branching as a learning problem have gained signiﬁcant attention recently.
In this section, we review some common trends in the ﬁeld and identify some key challenges.
3.1 Learning to branch with imitation learning
In recent years, it has been demonstrated that efﬁcient branching rules can be obtained by learning fast approximations of strong branching , a globally effective but computationally expensive rule [23, 26, 20, 17, 19, 36, 29].
This approach can be framed as imitation learning, a well-known method for learning in MDPs when an expert is available.
While this approach can lead to improvements over state-of-the-art branching rules on various benchmarks, it also has several drawbacks.
First, strong branching implementations are known to trigger side effects (such as early detection of an infeasible child) that do not map well within the branching MDP framework [14].
This means that branching decisions collected from the strong branching expert might not line up with the environment of the learning agent, which might result in a performance gap between the learning agent and the expert, even if the agent manages to successfully reproduce the expert decisions.
Second, other than these side-effects, strong branching relies on dual bound improvements to make branching decisions.
This can be ineffective in problems where the LP relaxation is not very informative or suffers from dual degeneracy, as pointed out in Gamrath et al.
As an extreme case,
Dey et al.
[9] provide an example where a strong-branching based B&B tree can have exponentially more nodes than the tree obtained with a problem-speciﬁc rule.
Finally, regardless of the expert quality, obtaining strong branching samples can become prohibitively expensive for large instances.
This means that non-trivial engineering solutions and scaling strategies are needed to allow training on larger problems, such as heavy computational parallelization [29].
3.2 Learning to branch with reinforcement learning
The reasons listed above suggest the need for an alternative approach to ﬁnding a good branching policy; an approach that does not rely on the strong branching rule.
One could perform imitation learning on another expert, but no other plausible imitation target is known, and the same performance ceiling issue would remain.
A natural alternative is reinforcement learning, an approach that aims to ﬁnd an optimal policy in a Markov Decision Process, with respect to any desired objective function.
But despite the theoretical appeal of RL for learning to branch, it also comes with its own challenges.
First, common evaluation metrics in B&B, such as solving time or ﬁnal tree size, are inconvenient for RL.
Both require to run episodes to completion, that is, to solve MILP instances to optimality, which leads to very long episodes even for moderately hard instances.
Second, in contrast to many
RL tasks studied in the literature, in B&B the worse the policy is, the longer are the episodes.
These two factors combined give rise to the following problems:
Collecting training data is computationally expensive.
In particular, training from scratch from a randomly initialized policy can be prohibitive for large MILPs.
Due to the length of the episodes, training signals are particularly sparse.
This exacerbates the so-called credit assignment problem [27], i e., the problem of determining which actions should be given credit for a certain outcome.
Two works have tried to tackle the branching problem with RL so far.
Sun et al.
[33] propose an approach based on evolution strategies, a variant of RL, combined with a novelty bonus based on discrepancies in B&B trees.
They show improvements over an imitation learning approach Gasse et al. [17] on instances derived from a common backbone graph, but fail to improve on more heterogeneous ones.
These results emphasize the difﬁculty of a direct application of RL on hard instance sets.
In parallel, Etheve et al.
[11] recently made a contribution that directly addresses the credit assignment problem.
They show that, when depth-ﬁrst-search (DFS) is used as the node selection strategy, minimization of the total B&B tree size can be achieved by taking decisions that minimize subtree size at each node.
Based on this result, they propose a Q-learning-type algorithm [28] where the learned Q-function approximates the local subtree size.
They report improvements over a state-ofthe-art branching rule on collections of small, ﬁxed-size instances.
However, they only evaluate the learned policy with DFS node selection, which matches their training setting, but is not a realistic
B&B setting.
In Section 4, we will show that the method proposed by Etheve et al.
[11] can be interpreted as a speciﬁc instantiation of a more general tree MDP framework, which effectively simpliﬁes the credit assignment problem in learning to branch (point 2 above).
Furthermore, we propose an alternative condition to the one of Etheve et al.
[11] to ensure a tree MDP setting, which results in shorter episodes, hence reducing the cost of data collection (point 1 above).
Branching as a tree MDP
We now detail our tree MDP framework, and show how the branching problem can be cast as a tree
MDP control problem, under some conditions.
Proofs as well as a side-by-side comparison of temporal and tree MDPs are deferred to the Supplementary Material.
4.1 Tree MDPs
We deﬁne tree MDPs as augmented Markov Decision Processes tM =
( S , A , p init , p − ch , p + ch , r, l ) , with states s ∈ S , actions a ∈ A , initial state distribution p init ( s 0 ) , respectively left and right child transition distributions p − ch
( s ch − i | s i , a i ) and p + ch
( s ch
+ i | s i , a i ) 4 , reward function r : S → R and leaf indicator l : S → { 0 , 1 } .
The central concept behind tree MDPs is that each non-leaf state s i (i.e., such that l ( s i ) = 0 ), together with an action a i , produces two new states s ch − i (its left child) and s ch
+ i (its right child).
As a result, the tree MDP generative process results in episodes τ that follow a tree structure (see Figure 1), where leaf states (i.e., such that l ( s i ) = 1 ) are the leaf nodes of the tree, below which no action can be taken and no children state will be created.
For simplicity, just like in
Section 2.2, we assume the tree-like trajectories have some ﬁnite size that we denote T = | τ | .
A tree MDP episode τ consists of a binary 5 tree with nodes N = { 0 , . . .
, | τ |} and leaf nodes
L = { i | i ∈ N , l ( s i ) = 1 } , which embeds a state s i at every node i ∈ N and an action a i at every non-leaf node i ∈ N \ L .
For convenience, in the following we will use pa i , ch − i and ch
+ i to denote the nodes that are respectively parent, left child and right child of a node i
if any, as well as d i and nd i to denote respectively the set of all descendants and non-descendants of a node i in the tree.
Together with a control mechanism π ( a t | s t ) , a tree MDP deﬁnes a probability distribution over trajectories,
[Equation]
As in temporal MDPs, the tree MDP control problem is to ﬁnd a policy that maximizes the expected cumulative reward, as deﬁned by (3).
Due to their speciﬁc generative process, a key characteristic of tree MDPs is the tree Markov property S ch − i , S ch
+ i ⊥⊥ S nd i
| S i ,
A
i , ∀ i , which guarantees, similarly to the temporal Markov property, that each subtree only depends upon the immediate state and action.
This again simpliﬁes the credit assignment problem in RL and results in an efﬁcient tree policy gradient formulation.
Proposition 4.1.
For any tree MDP tM, the policy gradient can be expressed as
[Equation]
4.2 The branching tree MDP
We now show how and under which conditions the vanilla B&B algorithm can be formulated as a tree MDP.
We consider episodes τ that follow exactly the B&B tree structure.
Each node i in the tree embeds a state s i
=
( MILP i , GUB i ) , where MILP i is the local sub-MILP of the node, and GUB i is the global upper bound at the time the node is processed 6 .
Each non-leaf node also embeds an action a i = ( j, x (cid:63)j ) , where j is the index of the branching variable chosen by B&B, and x (cid:63)j is the value used to branch ( x j ≤ (cid:98) x
(cid:63)j (cid:99) ∨ x j ≥ (cid:100) x (cid:63)j (cid:101) ).
Note that such states and actions, embedded in the B&B tree, carry enough information to unroll a vanilla B&B algorithm, as described in Section 2.1.
We now need to make two additional assumptions in order to formulate branching as a tree MDP.
4.2.1 B&B tree transitions
First, and this is our main requirement, B&B state transitions must decompose into p − ch and p + ch .
Assumption 4.2.
For every non-leaf node i , the global upper bounds GUB ch − i and GUB ch
+ i (reached by B&B when the left and right child is processed, respectively) can be derived solely from the current state and action, ( s i , a i ) .
Proposition 4.3.
A vanilla B&B algorithm that satisﬁes Assumption 4.2 forms a tree MDP.
Assumption 4.2 is not always satisﬁed, as the following counterexample shows.
Counter example.
Consider the root problem MILP
= min x s.t. x ≥ 0 .
6 , x ∈ Z , with upper bound
GUB 0 = ∞ .
The root LP solution is ˆ x (cid:63) = 0 .
6 , and the two sub-problems MILP ch − i and MILP ch
+ i follow from the (only) branching decision x ≤ 0 ∨ x ≥ 1 .
Now, the two global upper bound GUB ch − i and GUB ch + i depend on whether the feasible solution x = 1 has been found in the past, which in turn depends on the node processing order.
Going left ﬁrst ( − ) will yield ( GUB ch − i , GUB ch + i ) =
( ∞ , ∞ ) , while going right ﬁrst ( + ) will yield ( GUB ch − i , GUB ch + i ) =
(1 , ∞ ) .
We now provide two conditions under which Assumption 4.2 is true.
Proposition 4.4.
In Optimal Objective Limit B&B (ObjLim B&B), that is, when the optimal solution value of the MILP is known at the start of the algorithm (GUB 0 = GUB (cid:63) ), Assumption 4.2 holds.
Proposition 4.5.
In Depth-First-Search B&B (DFS B&B), that is, when nodes are processed depthﬁrst and left-ﬁrst by the algorithm, Assumption 4.2 holds.
Propositions 4.4 and 4.5 provide two viable options for turning vanilla B&B into a tree MDP, where p − ch and p + ch are deterministic functions.
The ﬁrst variant, ObjLim , requires MILP instances used for training to be solved to optimality once, in order to collect their optimal objective value.
The second variant, DFS , corresponds to the setting in [11].
In this variant there is no need to pre-solve training instances to optimality, however it is expected that the collected episodes might be longer than with a standard node selection rule, which might result in slower training.
4.2.2 B&B tree reward
Last, for branching to formulate as a control problem in a tree MDP, the objective must be compatible.
Assumption 4.6.
The branching objective can be decomposed over the nodes of the B&B tree, with a state-based reward function r : S → R .
Interestingly, a natural objective for branching is the ﬁnal B&B tree size , which expresses naturally as r ( s i ) = − 1 .
Thus, it is trivially compatible with Assumption 4.6.
We will consider this reward in our experiments.
Another common objective is the total solving time, which can also be expressed as a state-based reward r : S → R under mild assumptions.
Indeed, it sufﬁces to considers that solving
LP relaxations and making branching decisions at each node are the main contributing factors in the total running time, while other algorithmic components have a negligible cost.
In vanilla B&B, both these components only depend on the local state s i =
( MILP i , GUB i ) of each node.
4.3 Efﬁciency of tree MDP
Tree MDPs, when applicable, provide a convenient alternative to temporal MDPs for tackling the branching problem.
First, the tree Markov property implies that branching policies in tree branching
MDPs will not beneﬁt from any information other than the local state s i
=
( MILP i , GUB i ) to make optimal decisions, similarly to how control policies in temporal MDP can ignore past states and rely only on the immediate state s t .
Second, the credit assignment problem in the branching tree MDP is more efﬁcient than in the equivalent temporal MDP.
This is showcased in Figure 1, and stems from the fact that in a tree MDP all the descendants of a node i are necessarily processed after that node temporally.
As a consequence, the rewards credited to an action in the tree policy gradient (5), (cid:80) j ∈ d i r ( s j ) , are necessarily a subset of the rewards credited to the same action in the temporal policy gradient (4), (cid:80) t (cid:48) >t r ( s t (cid:48) ) .
Thus, it can be expected intuitively that learning branching policies within the tree MDP framework will be easier, and more sample-efﬁcient than learning within the temporal MDP framework.
We will validate this hypothesis experimentally in Section 5.
4.4 Theoretical limitations
Our proposed B&B variants, ObjLim and DFS , allow for a nice formulation of the branching problem as a tree MDP, which we argue is key to unlocking a more practical and sample-efﬁcient learning of branching policies.
However, usually the end goal is to learn a branching policy that performs well in realistic B&B settings, and the fact that a branching policy performs well in one of those variants does not guarantee that it will perform well in the vanilla setting also.
This discrepancy between the training environment and the evaluation environment is a recurring problem in RL, and is more generally referred to as the transfer learning problem.
While there exist solutions to mitigate this problem, in this paper we leave the question aside and simply assume that the transfer problem is negligible.
We thus directly report the performance obtained from each training setting in the realistic evaluation setting, a default B&B solver.
4.5 Connections with hierarchical RL
The tree MDP formulation has connections with hierarchical RL [10], a paradigm typically applied in the case where the learning task can be decomposed into a set of simpler tasks which can be solved recursively, independently of the parent task.
An example of this is picking up and delivering a package from a point A to a point B, which decomposes into: move to A, pick up, move to B, drop.
Both tree MDP and hierarchical RL exploit a decomposition of the task in order to simplify the credit assignment problem.
However, the two methods also differ on several points. First, in hierarchical RL, the sub-task decomposition must be given a priori by the user for each new task, and is ﬁxed with a limited depth, while in tree MDPs the decomposition holds by construction, and can be applied recursively for virtually inﬁnite depths. Second, the sub-tasks are different (with different reward and optimal policy), while in tree MDPs the reward remains the same.
Lastly, all sub-task states are processed consecutively, while in tree MDPs the temporal processing order of states can vary, and switches between different sub-trees are
Experiments
We now compare the performance of four machine learning approaches: the imitation learning method of Gasse et al.
[17], and three different RL methods.
We also compare against SCIP’s default rule (for a description of this rule we refer to the Supplementary Material A.4).
Code for reproducing all experiments is available online 7 .
5.1 Setup
Benchmarks Similarly to Gasse et al.
[17], we train and evaluate each method on ﬁve NP-hard problem benchmarks, which consist of synthetic combinatorial auctions, set covering, maximum independent set, capacitated facility location and multiple knapsack instances.
For each benchmark we generate a training set of 10,000 instances, along with a small set of 20 validation instances for tracking the RL performance during training.
For the ﬁnal evaluation, we further generate a test set of 40 instances, the same size as the training ones, and also a transfer set of 40 instances, larger and more challenging than the training ones.
More information about benchmarks and instance sizes can be found in the Supplementary Material (A.5).
Training We use the Graph Neural Network (GNN) from Gasse et al.
[17] to learn branching policies, with the same features and architecture.
This state representation has been shown to provide an efﬁcient encoding of the local MILP together with some global features.
Notice that this makes the MDP formulation into a POMDP, given that we do not encode the complete search tree (see
Section 2.3).
We compare four training methods: imitation learning from strong branching (IL); RL using temporal policy gradients (MDP); RL using tree policy gradients with DFS as a node selection strategy (tMDP+DFS), which enforces the tree Markov property due to Proposition 4.5; and RL using tree policy gradients with the optimal objective value set as an objective limit (tMDP+ObjLim), which corresponds to Propositions 4.4.
Other than that, we use default solver parameters, except for restarts and cutting planes after the root node which are deactivated.
We use a plain REINFORCE [35] with entropy bonus as our RL algorithm, for simplicity.
Our training procedure is summarized in Algorithm 1.
We set a maximum of 15,000 epochs and a time limit of six days for training.
Our implementation uses PyTorch [31] together with PyTorch Geometric [12], and Ecole [32] for interfacing to the solver SCIP [15].
All experiments are run on compute nodes equipped with a GPU.
[Figure]
Evaluation For each branching rule evaluated, we solve every validation, test or transfer instancetimes with a different random seed.
We use default solver parameters, except for restarts and cutting planes after the root node which are deactivated (same as during training), and a time limit of 1 hour for each solving run.
For tMDP+DFS and tMDP+ObjLim, the speciﬁc settings used during training (DFS node selection and optimal objective limit, respectively) are not used any more, thus providing a realistic evaluation setting.
We report the geometric mean of the ﬁnal B&B tree size as our metric of interest, as is common practice in the MILP literature [3].
We pair this with the average per-instance standard deviation (in percentage).
We only consider solving runs that ﬁnished successfully for all methods, as in [17].
Extended results including solving times are provided in the Supplementary Material (A.3).
5.2 Results
Figure 2 showcases the convergence of our three RL paradigms MDP, tMDP+ObjLim and tMDP+DFS during training, in terms of the ﬁnal B&B tree size on the validation set (the lower the better).
In order to better highlight the sample efﬁciency of each method, we report on the x-axis the cumulative number of collected training samples, which correlates with the length of the episodes collected during training.
This provides a hardware-independent proxy for training time.
As can be seen, the tree MDP paradigm clearly improves the convergence speed on these three benchmarks, with a clear domination of tMDP+ObjLim on one benchmark (Set Covering).
Training curves for the remaining benchmarks are available in the Supplementary Material (A.3).
Table 1 reports the ﬁnal performance of the branching rules obtained with each method, on both a held-out test set (same instance difﬁculty as training) and a transfer set (larger, more difﬁcult instances than training).
Despite a mismatch between the training and evaluation environments, which is required to enforce the tree Markov property, the tree MDP paradigm consistently produces equal or better branching rules than the temporal MDP paradigm on all 5 benchmarks.
On one benchmark, Multiple Knapsack, the branching rules learned by RL outperform both SCIP’s default branching rule and the strong branching imitation (IL) approach.
The likely reason is that the MILP formulation of Multiple Knapsack provides a very poor linear relaxation, which often results in no dual bound improvement after branching.
This means that strong branching scores are in most cases not discriminative, which is problematic for rules that heavily rely on this criterion (see
Section 3.1), such as SCIP’s default or a policy that imitates strong branching.
This situation makes a strong case for the potential of RL-based methods, which can adapt and devise alternative branching strategies.
On the remaining 4 benchmarks, however, RL methods perform worse than SCIP default or IL, despite being based on the same GNN architecture.
This illustrates the difﬁculty of learning to branch via RL, even on small-scale problems, and the remaining room for improvement.
Additional evaluation criteria (solving times and number of time limits) are available in the Supplementary Material (A.3). [Table]
Conclusions and Future Directions
This paper adds to a growing body of literature on using ML to assist decision-making in several key components of the B&B algorithm (see e.g. [5, 7, 22]).
We contribute to the study of RL as a tool for learning to branch in MILP solvers.
We present tree MDP, a variant of Markov Decision Processes, and show that under some conditions, the B&B branching process is tree-Markovian.
We show that the approach of Etheve et al.
[11] can be naturally cast as Q-learning for tree MDPs, and we propose an alternative, more computationally appealing way to enforce the tree Markov property in B&B, using optimal objective limits.
Finally, we evaluate for the ﬁrst time a variety of RL-based branching rules in a comprehensive computational study, and we show that tree MDPs improve the convergence speed of RL for branching, as well as the overall performance of the learnt branching rules.
These contributions bring us closer to learning efﬁcient branching rules from scratch using RL, which could ultimately outperform existing branching heuristics built upon decades of expert knowledge and experiment.
However, despite the convergence speed-up that our method provides, training without expert knowledge remains very computationally heavy and in general still results in worse performance than its supervised learning counterpart, which reveals a signiﬁcant gap that must be closed.
As future work, we would like to explore ideas to keep improving sample efﬁciency, and generalization across instance size.
This is necessary for RL to scale to larger, non-homogeneous benchmarks, such as MIPLIB [18], which at the moment remain out-of-reach for RL.
Finally, although our concern in this paper is focused on improving variable selection for MILP, our tree MILP construction could be useful in other applications.
Branch-and-bound is a type of divide-and-conquer algorithm, and we expect that, in general, this framework can be applied to any problem where one seeks to control such algorithms more efﬁciently.
Examples would include controlling the order in which a rover explores rooms in a building, or selecting the pivots in a quicksort algorithm. Acknowledgements
The authors acknowledge the generous support of this research.
In particular, the work of Karen
Aardal, Lara Scavuzzo and Neil Yorke-Smith was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under grant number 952215, and by The Netherlands Organisation for Scientiﬁc Research, NWO, Grant OCENW.GROOT.2019.015. The work of Feng Yang Chen, Didier Chételat, Maxime Gasse and Andrea Lodi was supported by the Canada Excellence Research Chair program.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] We claim that tree MDP allows for a better convergence of RL algorithms for solving the branching problem in B&B. (b) Did you describe the limitations of your work?
[Yes] The theoretical limitations are discussed in Section 4.4, and the computational limitations are addressed in Section 5.2.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Our work’s goal is to solve difﬁcult mathematical problems more efﬁciently.
The data was randomly generated and, as a matter of fact, we work with abstraction’s of real-world problems.
For these reasons, our work does not hold potential to have a negative societal impact.
It is true that, similarly to other research in Machine Learning, our work has a non-negligible carbon footprint.
However, given that our goal is more computationally efﬁcient problem-solving, we argue that this work is a wise investment of computational resources.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] These are assumptions 4.2 and 4.6.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs can be found in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] For the purpose of anonymity during the review process, all the code and data will be made available after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Some of these aspects are addressed in the paper (Section 5).
The full details are included in the accompanying repository (to be made available).
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Our evaluation results are averaged over 40 instances solved with 5 different seeds (200 data points total) and we report the standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] As discussed in Section [-5.
-]5.1 we train on GPU-equipped nodes of an internal cluster.
Each experiment took a maximum of six days to run.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We included the appropriate citation for all used models and software.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We (will) provide a link to our code.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Supplementary Material A.1 Side-by-side comparison of MDP and tMDP
A temporal MDP process: [Equation] r )
Probability of a trajectory τ :
[-p init ( s 0 ) (cid:81) | τ |− 1 t =
 π ( a t | s t ) p trans ( s t +1 | s t , a t )-][Equation]
Markov property:
[Equation]
Temporal MDP
A tree MDP process: ( [Equation] [Equation]
Probability of a trajectory τ :
[Equation]
a
[Equation]
Markov property:
[Equation]
Tree MDP A.2 Proofs
Proposition 4.1.
For any tree MDP tM, the policy gradient can be expressed as
[Equation]
Proof.
This proof draws closely to the proof of the temporal policy gradient theorem.
First, let us re-write (3) as
[Equation]
where
[Equation]
if l [Equation] node
[Equation]
and
[Equation]
if
[Equation]
The corresponding gradients when l ( s i ) = 1 and l ( s i ) = 0 are, respectively, ∇ π V π
( s i )
= 0 , and
[Equation]
Let us now write the gradient of V π ,
[Equation]
Either we have l ( s 0 ) =
1 and thus ∇ π V π = 0 , or we can expand ∇ π V π ( s 0 ) to obtain ∇
[Equation]
Then again, each of of the terms ∇ π V π ( s ch − 0 ) and ∇ π V π ( s ch +0 ) can be replaced by 0 if the corresponding node is a leaf node, or can be expanded further in the same way if it is a non-leaf node.
By applying this rule recursively, we ﬁnally obtain
[Equation]
Lemma A.1.
In B&B, both children MILPs MILP ch − i and MILP ch
+ i can be derived from the local
MILP MILP i and branching decision a i
=
( j, x (cid:63)j ) , with j the index of a variable in MILP i , and x (cid:63)j the value to be used for branching.
Proof.
From the deﬁnition of B&B in Section 2, MILP ch − i (resp.
MILP ch + i ) consist of MILP i augmented with the additional constraint x j ≤ (cid:98) x (cid:63)j (cid:99) ( resp.
x j ≥ (cid:100) x (cid:63)j (cid:101) ).
Proposition 4.3.
A vanilla B&B algorithm that satisﬁes Assumption 4.2 forms a tree MDP.
Proof.
We shall now prove that, under Assumption 4.2, the B&B process can be formulated as a tree MDP tM =
( S , A , p init , p − ch , p + ch , r, l ) , with states s i =
( MILP i , GUB i ) and actions a i = ( j, x (cid:63)j ) .
First, the algorithm starts at the root node with an initial MILP, MILP 0 , and an initial global upper bound GUB 0 = ∞ .
Thus, the root state s 0 follows an arbitrary, user-deﬁned MILP distribution p init ( s 0 ) , which is independent of the B&B algorithm.
Second, Lemma A.1, together with Assumption 4.2, ensures the existence of (deterministic) distributions p − ch
( s ch − i | s i , a i ) and p + ch
( s ch
+ i | s i , a i ) , from which the B&B children states s ch − i and s ch
+ i are generated.
Third, the reward function r ( s i ) is not part of the B&B algorithm, and can be arbitrarily deﬁned to match any (compatible) B&B objective.
Last, the leaf node indicator l ( s i ) is exactly the vanilla B&B leaf node criterion, and is obtained by solving the LP relaxation of MILP i constrained with upper bound GUB i , which results in either an infeasible LP (leaf node), a MILP-feasible LP solution (leaf node), or a MILP-infeasible
LP solution (non-leaf node).
This concludes the proof.
Proposition 4.4.
In Optimal Objective Limit B&B (ObjLim B&B), that is, when the optimal solution value of the MILP is known at the start of the algorithm (GUB 0 = GUB (cid:63) ), Assumption 4.2 holds.
Proof.
Because GUB
=
GUB (cid:63) , the initial global upper bound is equal to the optimal solution value to the original MILP.
Then, B&B will never be able to ﬁnd a feasible solution that tightens that bound, and we necessarily have GUB i = GUB 0 , ∀ i .
Hence GUB ch − i =
GUB ch
+ i
= GUB i , and both GUB ch − i and GUB ch
+ i can be directly derived from s i .
This concludes the proof.
Proposition 4.5.
In Depth-First-Search B&B (DFS B&B), that is, when nodes are processed depthﬁrst and left-ﬁrst by the algorithm, Assumption 4.2 holds.
Proof.
First, it is trivial to show that GUB ch − i can be derived from s i .
Because node i is not a leaf node, it has not resulted in an integral solution, and hence processing node i does not change the
GUB.
And since ch − i is processed directly after node i , we necessarily have GUB ch − i =
GUB i .
This, combined with Lemma A.1, shows that s ch − i can be inferred from s i and a i .
Second, we show how
GUB ch
+ i can be derived from s i and a i .
Because node ch + i is processed right after the whole subtree below ch − i has been processed, GUB ch
+ i is necessarily the minimum of GUB i and the optimal solution value of MILP ch
− i .
Now, because s ch − i can be inferred from s i and a i , MILP ch − i can be recovered as well, and solved to obtain its optimal solution value.
Therefore, GUB ch
+ i can be recovered from s i and a i .
This, together with GUB ch − i
=
GUB i , concludes the proof.
[Figure]
A.3 Extended results
Here we provide all training curves (Figure 3) and the extended evaluation results (Table 2) with the geometric mean of the solving times in seconds (Time) and the geometric mean of the ﬁnal B&B tree size (Nodes).
The results are averaged over the solving runs that ﬁnished successfully for all methods.
This is, if a solving run reached the time limit for any method, this is excluded from the average.
Table 3 shows the number of solving runs that timed out per method.
A.4 SCIP’s default branching rule
SCIP assigns maximum priority by default to the hybrid branching rule [2].
This means that the choice of branching variable is based on a weighted sum of different criteria.
The biggest weight is placed on the variable’s pseudocosts.
A variable’s pseudocost is calculated as a function of the change in LP objective value we observe (on each of the branches) as a consequence of branching on that variable.
This value can be explicitly calculated by tentatively branching on candidate variables (in which case the rule is called strong branching), or estimated based on past observed values.
SCIP runs strong branching until it has stored a sufﬁcient amount of observations for each variable, and then switches to the estimation strategy.
Other than pseudocosts, SCIP also considers information about the implied reductions of other variables’ domains and conﬂicts where the variable is involved, though with smaller importance.
It is important to consider that a call to strong branching can trigger a series of side-effects within the solver that are not accounted for in the node count.
This was ﬁrst observed by Gamrath and Schubert [14], who point out that this gives an unfair advantage to methods that use strong branching when comparing branching rules according to the ﬁnal tree size.
A.5 Instance collections
This section presents the models used to generate our instance benchmarks.
The parameters used to generate each benchmark are shown in Table 4.
[Table] [Table]
[Table]
A.5.1 Combinatorial auctions
For m items, we are given n bids {B j } nj =1 .
Each bid B j is a subset of the items with an associated bidding price p j .
The associated combinatorial auction problem is of the following form: maximize
[Equation]
to
[Equation]
where x j represents the action of choosing bid B j .
A.5.2 Set covering
Given the elements 1 , 2 , ..., m , and a collection S of n sets whose union equals the set of all elements, the set cover problem can be formulated as follows:
[Equation]
A.5.3 Maximum independent set
Given a graph G the maximum independent set problem consists in ﬁnding a subset of nodes of maximum cardinality such that no two nodes in that subset are connected.
We use the clique formulation from [6].
Given a collection C ⊆ 2 V of cliques whose union covers all the edges of the graph G , the clique cover formulation is
[Equation]
A.5.4 Capacitated facility location with unsplittable demand
Given a number n of clients with demands { d j } nj =1 , and a number m of facilities with ﬁxed operating costs { f i } mi =1 and capacities { s i } mi =1 , let c ij /d
j be the unit transportation cost between facility i and client j , and let p ij /d
j be the unit proﬁt for facility i supplying client j .
We try to solve the following problem minimize
[Equation]
subject to
[Equation]
where each variable x ij represents the decision of facility i supplying client j ’s demand, and each variable y i representing the decision of opening facility i for operation.
A.5.5 Multiple knapsack
Given n items with respective prices { p j } nj =1 and weights { w j } nj =1 , and m knapsacks with capacities { c i } mi =1 , the multiple knapsack problem consists in placing a number of items in each of the knapsacks such that the price of the selected items is maximized, while the capacity of the knapsacks is not exceeded by the total weight of the items therein.
Formally: maximize m (cid:88) i = n (cid:88)
j = p
j x ij subject to n (cid:88)
[Equation] = w [Equation]
m m (cid:88) i [Equation] [Equation]
n [Equation] [Equation] j where each variable x ij represents the decision of placing item j in knapsack i .