Learning to branch with Tree MDPs
Abstract
State-of-the-art Mixed Integer Linear Program (MILP) solvers combine systematictree search with a plethora of hard-coded heuristics, such as the branching rule.
Theidea of learning branching rules from data has received increasing attention recently,and promising results have been obtained by learning fast approximations of thestrong branching expert.
In this work, we instead propose to learn branching rulesfrom scratch via Reinforcement Learning (RL).
We revisit the work of Etheve et al.[9] and propose tree Markov Decision Processes, or tree MDPs , a generalization oftemporal MDPs that provides a more suitable framework for learning to branch.
We derive a tree policy gradient theorem, which exhibits a better credit assignmentcompared to its temporal counterpart.
We demonstrate through computational experiments that tree MDPs improve the learning convergence, and offer a promisingframework for tackling the learning-to-branch problem in MILPs.
Introduction
Mixed Integer Linear Programs (MILPs) offer a powerful tool for modeling combinatorial optimization problems and are used in many real-world applications [27].
The method of choice forsolving MILPs to global optimality is the Branch-and-Bound (B&B) algorithm, which follows adivide-and-conquer strategy.
A critical component of the algorithm is the branching rule which isused to recursively partition the MILP search space using a space search tree.
While there is littleunderstanding of optimal branching decisions in general settings [22], the choice of the branchingrule has a decisive impact on solving performance in practice [3].
In modern solvers, state-of-the-artperformance is obtained using hard-coded branching rules that rely on heuristics designed by domainexperts to perform well on a representative set of test instances [16].
In recent years, increasing attention has been given to approaches that use Machine Learning (ML)to obtain good branching rules and improve upon expert-crafted heuristics [5].
Such a statisticalapproach is particularly well-suited in situations where similar problems are solved repeatedly, whichis a common industrial scenario.
In such cases, a generic and problem-agnostic branching rule mightbe suboptimal.
Due to the sequential nature of branching, a natural paradigm to formulate the learningproblem is Reinforcement Learning (RL), which allows to directly search for the optimal branchingrule with respect to a metric of interest, such as average solving time, or ﬁnal B&B tree size.
Etheve et al.
[9] show that, when using depth-ﬁrst-search (DFS) as the tree exploration (a.k.a. nodeselection) policy in B&B, minimizing each subtree size is equivalent to minimizing the global treesize.
Using this result, they propose an efﬁcient value-based RL procedure for learning to branch.
Inthis paper we build upon the work of Etheve et al.
[9], and we propose a generalization of the Markov
Decision Process (MDP) paradigm that we call tree MDP (illustrated in Figure 1, formally discussedin Section 4).
This general formulation captures the key insight of their approach, but opens the doorfor more general B&B settings, RL algorithms and reward functions.
B&B tree a b c d e f g h
i
Tree MDP episode a b c d e f g h
i
Temporal MDP episodes a c g b d f h
i e a c f h
i g b d e a b c d e f g h
i
In particular, we show that DFS is just one way to enforce the “tree Markov” property, and we proposean alternative, more practical solution that simply relies on providing the optimal objective limit tothe solver during training.
Our contribution is three-fold:
We introduce the concepts of a tree MDP and of the tree Markov property, and we derive apolicy gradient theorem for tree MDPs.2.
We propose an alternative way to enforce the tree Markov property in learning to branch by
imposing an optimal objective limit, which is less computationally demanding than DFS [9].3.
We show that adopting the tree MDP paradigm improves both the convergence speed of RL
and the quality of the resulting branching rules, compared the regular MDP paradigm.
The remainder of this paper is organized as follows.
We formally introduce MILPs, B&B and MDPsin Section 2.
In Section 3 we motivate the need for moving beyond imitation learning in learning tobranch, and discuss the challenges it raises.
In Section 4 we introduce the concepts of tree MDPsand the tree Markov property, we derive a policy gradient theorem for tree MDPs, and we present aconvenient solution to enforce the tree Markov property in B&B without DFS [9].
In Section 5 weconduct, for the ﬁrst time, a thorough computational evaluation of RL for learning branching rules,on ﬁve synthetic MILP benchmarks with two difﬁculty levels.
Finally, we discuss the signiﬁcance ofour results and future directions in Section 6.
Background
In this section, we describe the Branch-and-Bound (B&B) algorithm, and we show how the branchingproblem naturally formulates as a (temporal) Markov Decision Process.
2.1 The B&B algorithm
Consider a Mixed Integer Linear Program instance as an optimization problem of the form
[Equation]
where c ∈ R n is the objective coefﬁcient vector, A ∈ R m × n is the constraint matrix, b ∈ R m isthe constraint right-hand-side, l, u ∈ R n are the lower and upper bound vectors, respectively, and
I ⊆ { 1 , 2 , ..., n } is the subset of variables constrained to take integer values.
Relaxing the integralityconstraints, x i ∈ Z ∀ i ∈ I , yields a Linear Program (LP) that can be solved efﬁciently in practice,for example by using the Simplex algorithm.
Solving such a relaxation yields an LP solution ˆ x (cid:63) , andprovides a lower bound c
T ˆ x (cid:63) to the original problem (1).
If by chance ˆ x (cid:63) satisﬁes the integralityconstraints, ˆ x (cid:63)i ∈ Z ∀ i ∈
I , then it is also an optimal solution to (1).
If, on the contrary, there is atleast one variable x
i , i ∈ I such that ˆ x (cid:63)i is not an integer, then one can split the feasible region intotwo sub-problems, by imposing
[Equation]
(cid:101) .
This partitioning step is known as branching.
In its most basic form, the vanilla B&B algorithmrecursively applies branching, thereby building a search space tree where each node has an associatedlocal sub-MILP, with a local LP solution and associated local lower bound.
If the local LP solutionsatisﬁes the MILP integrality constraints, then it is termed a feasible solution , and it also providesan upper bound to (1).
At any given time of the B&B process, a Global Lower Bound (GLB) to theoriginal MILP can be obtained by taking the lowest of local lower bounds of the leaf nodes of the tree.
Similarly, a Global Upper Bound (GUB) can be obtained by taking the lowest of the upper bounds sofar.
The B&B process terminates when no more B&B leaf node can be partitioned with (2), i.e., allleaf nodes satisfy one of these conditions: the local LP has no solution (infeasible); the local lowerbound is above the GUB (pruned); or the local LP solution satisﬁes the MILP integrality constraints(integer feasible).
At termination, we have GLB = GUB , and the original MILP (1) is solved.
The branching problem, a.k.a.
variable selection problem, is the task of choosing, at every B&Biteration, the variable x i that will be used to generate a partition of the form (2).
While there is nouniversal metric to measure the quality of a branching rule, a common performance measure is theﬁnal size of the B&B tree (the smaller the better)
2.2 Temporal MDPs
In the following, upper-case letters in italics denote random variables (e.g. S, A ), while theirlower-case counterpart denote their value (e.g. s, a ) and their calligraphic counterpart their domain(e.g., s ∈ S , a ∈ A ).
We consider episodic Markov Decision Processes (MDPs) of the form
M = ( S , A , p init , p trans , r ) , with states s ∈ S , actions a ∈ A , initial state distribution p init ( s 0 ) ,state transition distribution p trans ( s t +1 | s t , a t ) , and reward function r : S → R .
For simplicity, weassume ﬁnite episodes of length T , that is, τ = ( s 0 , a 0 , . . .
, s T ) , | τ | =
T .
Together with a controlmechanism represented by a stochastic policy π ( a t | s t ) , the MDP deﬁnes a probability distributionover trajectories, namely
[Equation]
The MDP control problem is to ﬁnd a policy that maximizes the expected cumulative reward,π (cid:63) ∈ arg max π V π , with
[Equation]
A key property of MDPs is the temporal Markov property S t +1 ⊥⊥ S <t | S t , A t , ∀ t , which guaranteesthat the future only depends upon the current state and action.
One consequence of this property isthe temporal policy gradient theorem [31], which forms the basis of policy gradient methods
[Equation]
2.3 The branching temporal MDP
Let us now consider the problem of learning a branching policy in a B&B solver.
As noted by [19, 15],branching decisions are made sequentially, thus the problem can naturally be regarded as a temporal
MDP.
In this form, the states s t consist of the entire state of the B&B process (that is, the solver) attime t , which includes the whole tree structure, all sub-MILPs and LP solutions, and all upper andlower bounds.
The actions a t are the branching decisions, and the reward is chosen so that the returnmatches an objective function of interest (e.g., the ﬁnal B&B tree size).
While appealing by its simplicity, we argue that this formulation is not practical for two reasons.
First, even in the simplest textbook implementation of B&B, the MDP states are complex objects of growing size, which are impractical to handle.
Second, episode length in branching can growextremely large, in the worst case exponentially with the size of the problem.
This exacerbates theso-called credit assignment problem , i.e., the problem of determining which actions should be givencredit for a certain outcome (see Section 3.2 for a more detailed discussion).
In Section 4, we will wewill show how the tree MDP formulation addresses both those challenges.
Approaches to learning to branch
Approaches that frame branching as a learning problem have gained signiﬁcant attention recently.
Inthis section, we review some common trends in the ﬁeld and identify some key challenges.
3.1 Learning to branch with imitation learning
In recent years, it has been demonstrated that efﬁcient branching rules can be obtained by learningfast approximations of strong branching , a globally effective but computationally expensive rule[20, 23, 18, 15, 17, 33, 26].
This approach can be framed as imitation learning, a well-known methodfor learning in MDPs when an expert is available.
While this approach can lead to improvementsover state-of-the-art branching rules on various benchmarks, it also has several drawbacks.
First, strong branching implementations are known to trigger side effects (such as early detection ofan infeasible child) that do not map well within the branching MDP framework [12].
This meansthat branching decisions collected from the strong branching expert might not line up with theenvironment of the learning agent, which might result in a performance gap between the learningagent and the expert, even if the agent manages to successfully reproduce the expert decisions.
Second, other than these side-effects, strong branching relies on dual bound improvements tomake branching decisions.
This can be ineffective in problems where the LP relaxation is not veryinformative or suffers from dual degeneracy, as pointed out in Gamrath et al.
As an extreme case,
Dey et al.
[8] provide an example where a strong-branching based B&B tree can have exponentiallymore nodes than the tree obtained with a problem-speciﬁc rule.
Finally, regardless of the expert quality, obtaining strong branching samples can become prohibitivelyexpensive for large instances.
This means that non-trivial engineering solutions and scaling strategiesare needed to allow training on larger problems, such as heavy computational parallelization [26].
3.2 Learning to branch with reinforcement learning
The reasons listed above suggest the need for an alternative approach to ﬁnding a good branchingpolicy; an approach that does not rely on the strong branching rule.
One could perform imitationlearning on another expert, but no other plausible imitation target is known, and the same performanceceiling issue would remain.
A natural alternative is reinforcement learning, an approach that aims toﬁnd an optimal policy in a Markov Decision Process, with respect to any desired objective function.
But despite the theoretical appeal of RL for learning to branch, it also comes with its own challenges.
First, common evaluation metrics in B&B, such as solving time or ﬁnal tree size, are inconvenientfor RL.
Both require to run episodes to completion, that is, to solve MILP instances to optimality,which leads to very long episodes even for moderately hard instances.
Second, in contrast to many
RL tasks studied in the literature, in B&B the worse the policy is, the longer are the episodes.
Thesetwo factors combined give rise to the following problems:
Collecting training data is computationally expensive.
In particular, training from scratch
from a randomly initialized policy can be prohibitive for large MILPs.
Due to the length of the episodes, training signals are particularly sparse.
This exacerbates
the so-called credit assignment problem [24], i.e., the problem of determining which actions
should be given credit for a certain outcome.
Two works have tried to tackle the branching problem with RL so far.
Sun et al.
[30] propose anapproach based on evolution strategies, a variant of RL, combined with a novelty bonus based ondiscrepancies in B&B trees.
They show improvements over an imitation learning approach Gasse et al.[15] on instances derived from a common backbone graph, but fail to improve on more heterogeneousones.
These results emphasize the difﬁculty of a direct application of RL on hard instance sets.
In parallel, Etheve et al.
[9] recently made a contribution that directly addresses the credit assignmentproblem.
They show that, when depth-ﬁrst-search (DFS) is used as the node selection strategy,minimization of the total B&B tree size can be achieved by taking decisions that minimize subtreesize at each node.
Based on this result, they propose a Q-learning-type algorithm [25] where thelearned Q-function approximates the local subtree size.
They report improvements over a state-ofthe-art branching rule on collections of small, ﬁxed-size instances.
However, they only evaluate thelearned policy with DFS node selection, which matches their training setting, but is not a realistic
B&B setting.
In Section 4, we will show that the method proposed by Etheve et al.
[9] can beinterpreted as a speciﬁc instantiation of a more general tree MDP framework, which effectivelysimpliﬁes the credit assignment problem in learning to branch (point 2 above).
Furthermore, wepropose an alternative condition to the one of Etheve et al.
[9] to ensure a tree MDP setting, whichresults in shorter episodes, hence reducing the cost of data collection (point 1 above).
Branching as a tree MDP
We now detail our tree MDP framework, and show how the branching problem can be cast as a tree
MDP control problem, under some conditions.
Proofs are deferred to the Supplementary Material.
4.1 Tree MDPs
We deﬁne tree MDPs as augmented Markov Decision Processes tM =
( S , A , p init , p − ch , p + ch , r, l ) ,with states s ∈ S , actions a ∈ A , initial state distribution p init ( s 0 ) , respectively left and right childtransition distributions p − ch
( s ch − i | s i , a i ) and p + ch
( s ch
+ i | s i , a i ) 4 , reward function r : S → R and leafindicator l : S → { 0 , 1 } .
The central concept behind tree MDPs is that each non-leaf state s i (i.e.,such that l ( s i ) = 0 ), together with an action a i , produces two new states s ch − i (its left child) and s ch
+ i(its right child).
As a result, the tree MDP generative process results in episodes τ that follow a treestructure (see Figure 1), where leaf states (i.e., such that l ( s i ) = 1 ) are the leaf nodes of the tree,below which no action can be taken and no children state will be created.
For simplicity, just like in
Section 2.2, we assume the tree-like trajectories have some ﬁnite size that we denote T = | τ | .
A tree MDP episode τ consists of a binary 5 tree with nodes N = { 0 , . . .
, | τ |} and leaf nodes
L = { i | i ∈ N , l ( s i ) = 1 } , which embeds a state s i at every node i ∈ N and an action a i at everynon-leaf node i ∈ N \ L .
For convenience, in the following we will use pa i , ch − i and ch
+ i to denotethe nodes that are respectively parent, left child and right child of a node i
if any, as well as d i and nd ito denote respectively the set of all descendants and non-descendants of a node i in the tree.
Togetherwith a control mechanism π ( a t | s t ) , a tree MDP deﬁnes a probability distribution over trajectories,
[Equation]
As in temporal MDPs, the tree MDP control problem is to ﬁnd a policy that maximizes the expectedcumulative reward, as deﬁned by (3).
Due to their speciﬁc generative process, a key characteristic oftree MDPs is the tree Markov property S ch − i , S ch
+ i ⊥⊥ S nd i
| S i ,
A
i , ∀ i , which guarantees, similarlyto the temporal Markov property, that each subtree only depends upon the immediate state and action.
This again simpliﬁes the credit assignment problem in RL and results in an efﬁcient tree policygradient formulation.
Proposition 4.1.
For any tree MDP tM, the policy gradient can be expressed as
[Equation]
4.2 The branching tree MDP
We now show how and under which conditions the vanilla B&B algorithm can be formulated as atree MDP.
We consider episodes τ that follow exactly the B&B tree structure.
Each node i in the tree embeds a state s i
=
( MILP i , GUB i ) , where MILP i is the local sub-MILP of the node, and GUB i isthe global upper bound at the time the node is processed 6 .
Each non-leaf node also embeds an actiona i =
( j, x (cid:63)j ) , where j is the index of the branching variable chosen by B&B, and x (cid:63)j is the value usedto branch ( x j ≤ (cid:98) x
(cid:63)j (cid:99) ∨ x j ≥ (cid:100) x (cid:63)j (cid:101) ).
Note that such states and actions, embedded in the B&B tree,carry enough information to unroll a vanilla B&B algorithm, as described in Section 2.1.
We nowneed to make two additional assumptions in order to formulate branching as a tree MDP.
4.2.1 B&B tree transitions
First, and this is our main requirement, B&B state transitions must decompose into p − ch and p + ch .
Assumption 4.2.
For every non-leaf node i , the global upper bounds GUB ch − i and GUB ch
+ i (reachedby B&B when the left and right child is processed, respectively) can be derived solely from thecurrent state and action, ( s i , a i ) .
Proposition 4.3.
A vanilla B&B algorithm that satisﬁes Assumption 4.2 forms a tree MDP.
Assumption 4.2 is not always satisﬁed, as the following counterexample shows.
Counter example.
Consider the root problem MILP
= min x s.t. x ≥ 0 .
6 , x ∈ Z , with upper bound
GUB 0 = ∞ .
The root LP solution is ˆ x (cid:63) = 0 .
6 , and the two sub-problems MILP ch − i and MILP ch
+ ifollow from the (only) branching decision x ≤ 0 ∨ x ≥ 1 .
Now, the two global upper bound GUB ch − iand GUB ch
+ i depend on whether the feasible solution x = 1 has been found in the past, which in turndepends on the node processing order.
Going left ﬁrst ( − ) will yield ( GUB ch − i , GUB ch + i ) =
( ∞ , ∞ ) ,while going right ﬁrst ( + ) will yield ( GUB ch − i , GUB ch + i ) =
(1 , ∞ ) .
We now provide two conditions under which Assumption 4.2 is true.
Proposition 4.4.
In Optimal Objective Limit B&B (ObjLim B&B), that is, when the optimal solutionvalue of the MILP is known at the start of the algorithm (GUB 0 = GUB (cid:63) ), Assumption 4.2 holds.
Proposition 4.5.
In Depth-First-Search B&B (DFS B&B), that is, when nodes are processed depthﬁrst and left-ﬁrst by the algorithm, Assumption 4.2 holds.
Propositions 4.4 and 4.5 provide two viable options for turning vanilla B&B into a tree MDP, wherep − ch and p + ch are deterministic functions.
The ﬁrst variant, ObjLim , requires MILP instances used fortraining to be solved to optimality once, in order to collect their optimal objective value.
The secondvariant, DFS , corresponds to the setting in [9].
In this variant there is no need to pre-solve traininginstances to optimality, however it is expected that the collected episodes might be longer than with astandard node selection rule, which might result in slower training.
4.2.2 B&B tree reward
Last, for branching to formulate as a control problem in a tree MDP, the objective must be compatible.
Assumption 4.6.
The branching objective can be decomposed over the nodes of the B&B tree, witha state-based reward function r : S → R .
Interestingly, a natural objective for branching is the ﬁnal B&B tree size , which expresses naturally asr ( s i ) = − 1 .
Thus, it is trivially compatible with Assumption 4.6.
We will consider this reward in ourexperiments.
Another common objective is the total solving time, which can also be expressed as astate-based reward r : S
→ R under mild assumptions.
Indeed, it sufﬁces to considers that solving
LP relaxations and making branching decisions at each node are the main contributing factors in thetotal running time, while other algorithmic components have a negligible cost.
In vanilla B&B, boththese components only depend on the local state s i =
( MILP i , GUB i ) of each node.
4.3 Efﬁciency of tree MDP
Tree MDPs, when applicable, provide a convenient alternative to temporal MDPs for tackling thebranching problem.
First, the tree Markov property implies that branching policies in tree branching
MDPs will not beneﬁt from any information other than the local state s i
=
( MILP i , GUB i ) to makeoptimal decisions, similarly to how control policies in temporal MDP can ignore past states and relyonly on the immediate state s t .
Second, the credit assignment problem in the branching tree MDPis more efﬁcient than in the equivalent temporal MDP.
This is showcased in Figure 1, and stemsfrom the fact that in a tree MDP all the descendants of a node i are necessarily processed after thatnode temporally.
As a consequence, the rewards credited to an action in the tree policy gradient(5), (cid:80) j ∈ d i r ( s j ) , are necessarily a subset of the rewards credited to the same action in the temporalpolicy gradient (4), (cid:80) t (cid:48) >t r ( s t (cid:48) ) .
Thus, it can be expected intuitively that learning branching policieswithin the tree MDP framework will be easier, and more sample-efﬁcient than learning within thetemporal MDP framework.
We will validate this hypothesis experimentally in Section 5.
4.4 Theoretical limitations
Our proposed B&B variants, ObjLim and DFS , allow for a nice formulation of the branching problemas a tree MDP, which we argue is key to unlocking a more practical and sample-efﬁcient learning ofbranching policies.
However, usually the end goal is to learn a branching policy that performs well inrealistic B&B settings, and the fact that a branching policy performs well in one of those variantsdoes not guarantee that it will perform well in the vanilla setting also.
This discrepancy betweenthe training environment and the evaluation environment is a recurring problem in RL, and is moregenerally referred to as the transfer learning problem.
While there exist solutions to mitigate thisproblem, in this paper we leave the question aside and simply assume that the transfer problem isnegligible.
We thus directly report the performance obtained from each training setting in the realisticevaluation setting, a default B&B solver.
Experiments
We now compare the performance of four machine learning approaches: the imitation learningmethod of Gasse et al.
[15], and three different RL methods.
We also compare against SCIP’s defaultrule (for a description of this rule we refer to appendix A.4).
Code for reproducing all experiments isavailable online 8 .
5.1 Setup
Benchmarks Similarly to Gasse et al.
[15], we train and evaluate each method on ﬁve NP-hardproblem benchmarks, which consist of synthetic combinatorial auctions, set covering, maximumindependent set, capacitated facility location and multiple knapsack instances.
For each benchmarkwe generate a training set of 10,000 instances, along with a small set of 20 validation instances fortracking the RL performance during training.
For the ﬁnal evaluation, we further generate a test setof 40 instances, the same size as the training ones, and also a transfer set of 40 instances, larger andmore challenging than the training ones.
More information about benchmarks and instance sizes canbe found in the appendix (A.5).
Training We use the Graph Neural Network (GNN) from Gasse et al.
[15] to learn branchingpolicies, with the same features and architecture.
This state representation has been shown to providean efﬁcient encoding of the local MILP together with some global features.
Notice that this makesthe MDP formulation into a POMDP, given that we do not encode the complete search tree (see
Section 2.3).
We compare four training methods:
imitation learning from strong branching (IL); RLusing temporal policy gradients (MDP); RL using tree policy gradients with DFS as a node selectionstrategy (tMDP+DFS), which enforces the tree Markov property due to Proposition 4.5; and RLusing tree policy gradients with the optimal objective value set as an objective limit (tMDP+ObjLim),which corresponds to Propositions 4.4.
Other than that, we use default solver parameters, except forrestarts and cutting planes after the root node which are deactivated.
We use a plain REINFORCE[32] with entropy bonus as our RL algorithm, for simplicity.
Our training procedure is summarizedin Algorithm 1.
We set a maximum of 15,000 epochs and a time limit of six days for training.
Our implementation uses PyTorch [28] together with PyTorch Geometric [10], and Ecole [29] forinterfacing to the solver SCIP [13].
All experiments are run on compute nodes equipped with a GPU.
Evaluation For each branching rule evaluated, we solve every validation, test or transfer instance 5times with a different random seed.
We use default solver parameters, except for restarts and cuttingplanes after the root node which are deactivated (same as during training), and a time limit of 1 hourfor each solving run.
For tMDP+DFS and tMDP+ObjLim, the speciﬁc settings used during training(DFS node selection and optimal objective limit, respectively) are not used any more, thus providinga realistic evaluation setting.
We report the geometric mean of the ﬁnal B&B tree size as our metric ofinterest, as is common practice in the MILP literature [3].
We pair this with the average per-instancestandard deviation (in percentage).
We only consider solving runs that ﬁnished successfully for allmethods, as in [15].
Extended results including solving times are provided in the appendix (A.3).
5.2 Results
[Figure]
Figure 2 showcases the convergence of our three RL paradigms MDP, tMDP+ObjLim and tMDP+DFSduring training, in terms of the ﬁnal B&B tree size on the validation set (the lower the better).
Inorder to better highlight the sample efﬁciency of each method, we report on the x-axis the cumulativenumber of collected training samples, which correlates with the length of the episodes collectedduring training.
This provides a hardware-independent proxy for training time.
As can be seen, thetree MDP paradigm clearly improves the convergence speed on these three benchmarks, with a cleardomination of tMDP+ObjLim on one benchmark (Set Covering).
Training curves for the remainingbenchmarks are available in the appendix (A.3).
Table 1 reports the ﬁnal performance of the branching rules obtained with each method, on botha held-out test set (same instance difﬁculty as training) and a transfer set (larger, more difﬁcultinstances than training).
Despite a mismatch between the training and evaluation environments,which is required to enforce the tree Markov property, the tree MDP paradigm consistently producesequal or better branching rules than the temporal MDP paradigm on all 5 benchmarks.
On one benchmark, Multiple Knapsack, the branching rules learned by RL outperform both SCIP’sdefault branching rule and the strong branching imitation (IL) approach.
The likely reason is thatthe MILP formulation of Multiple Knapsack provides a very poor linear relaxation which oftenresults in no dual bound improvement after branching.
This means that strong branching scores arein most cases not discriminative, which is problematic for rules that heavily rely on this criterion (see
Section 3.1), such as SCIP’s default or a policy that imitates strong branching.
This situation makes a
[Figure]
[Table]
strong case for the potential of RL-based methods, which can adapt and devise alternative branchingstrategies.
On the remaining 4 benchmarks, however, RL methods perform worse than SCIP default or IL,despite being based on the same GNN architecture.
This illustrates the difﬁculty of learning tobranch via RL, even on small-scale problems, and the remaining room for improvement.
Additionalevaluation criteria (solving times and number of time limits) are available in the appendix (A.3).
Conclusions and Future Directions
In this paper, we contribute to the study of RL as a tool for learning to branch in MILP solvers.
Wepresent tree MDP, a variant of Markov Decision Processes, and show that under some conditions,the B&B branching process is tree-Markovian.
We show that the approach of Etheve et al.
[9] canbe naturally cast as Q-learning for tree MDPs, and we propose an alternative, more computationallyappealing way to enforce the tree Markov property in B&B, using optimal objective limits.
Finally,we evaluate for the ﬁrst time a variety of RL-based branching rules in a comprehensive computationalstudy, and we show that tree MDPs improve the convergence speed of RL for branching, as well asthe overall performance of the learnt branching rules.
These contributions bring us closer to learning efﬁcient branching rules from scratch using reinforcement learning, which could ultimately outperform existing branching heuristics built upondecades of expert knowledge and experiment.
However, despite the convergence speed-up that ourmethod provides, training without expert knowledge remains very computationally heavy and ingeneral still results in worse performance than its supervised learning counterpart, which reveals asigniﬁcant gap that must be closed.
As future work, we would like to explore ideas to keep improvingsample efﬁciency, and generalization across instance size.
This is necessary for RL to scale to larger,non-homogeneous benchmarks, such as MIPLIB [16], which at the moment remain out-of-reach for
RL.
Finally, although our concern in this paper is focused on improving variable selection in branch-andbound MILP solvers, it appears that our tree MILP construction could be useful in other applications.
Branch-and-bound is a type of divide-and-conquer algorithm, and we expect more general thatthis framework could be applied to any problem where one seeks to control such algorithms moreefﬁciently.
Examples would include controlling the order in which a rover explores rooms in abuilding, or selecting the pivots in a quicksort algorithm.
Although these problems could be tackledwith a regular MDP framework, we expect that modeling these problems by a tree MDP, and usingsuitable tree RL algorithms, could result in improved credit assignment, and ultimately better learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] We claim that tree MDP allows for a better convergenceof RL algorithms for solving the branching problem in B&B.(b)
Did you describe the limitations of your work?
[Yes] The theoretical limitations arediscussed in Section 4.4, and the computational limitations are addressed in Section5.2.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Ourwork’s goal is to solve difﬁcult mathematical problems more efﬁciently.
The data wasrandomly generated and, as a matter of fact, we work with abstraction’s of real-worldproblems.
For these reasons, our work does not hold potential to have a negativesocietal impact
It is true that, similarly to other research in Machine Learning, our
work has a non-negligible carbon footprint.
However, given that our goal is more
computationally efﬁcient problem-solving, we argue that this work is a wise investment
of computational resources.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] These are
assumptions 4.2 and 4.6.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs can be
found in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
For the purpose
of anonymity
during the review process, all the code and data will be made available
after acceptance.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] Some of these aspects are addressed in the paper (Section 5).
Thefull details are included in the accompanying repository (to be made available).
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] Our evaluation results are averaged over 40 instances
solved with 5 different seeds (200 data points total) and we report the standard devia-
tion.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] As discussed in Section 5.
we
train on GPU-equipped nodes of an internal cluster.
Each experiment took a maximum
of six days to run.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We included theappropriate citation for all used models and software.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We (will) provide a link to our code.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix
A.1 Side-by-side comparison of MDP and tMDP
A temporal MDP process: ( S , A , p init , p trans , r )
Probability of a trajectory τ : p π ( τ ) =
p init ( s 0 ) (cid:81) | τ |− 1 t =
π ( a t | s t ) p trans ( s t +1 | s t , a t )
Markov property:
S t +1 ⊥⊥ S <t | S t , A t , ∀ t
Temporal MDP
A tree MDP process: ( S , A , p init , p − ch , p + ch , r, l )
Probability of a trajectory τ : p
[Equation]
=
[Equation]
(cid:81) i ∈N\L π ( a i | s i ) p − ch
( s ch − i | s i , a i ) p + ch
( s ch
+ i | s i , a i ) .
Markov property:
S ch − i , S ch
+ i ⊥⊥ S nd i
| S i , A
[Equation]
, ∀ i
Tree MDP
A.2 Proofs
Proposition 4.1.
For any tree MDP tM, the policy gradient can be expressed as
[Equation] [Equation]
Proof.
This proof draws closely to the proof of the temporal policy gradient theorem.
First, let usre-write (3) as
[Equation]
s 0 )] , where
V π ( s i ) :
= r ( s i ) if l ( s i ) =
1 (leaf node), and
V
[Equation]
( s i ) :=
[Equation]
if l ( s i ) = 0 (non-leaf node).
The corresponding gradients when l ( s i ) = 1 and l ( s i ) = 0 are, respectively,∇ π V π
( s i )
= 0 , and ∇
[Equation]
Let us now write the gradient of V π ,
[Equation]
Either we have l ( s 0 ) =
1 and thus ∇ π V π = 0 , or we can expand ∇ π V π ( s 0 ) to obtain∇ π V π
=
E s 0 ,
a
[Equation]
ch
[Equation]
ch
[Equation]
Then again, each of of the terms ∇ π V π ( s ch − 0 ) and ∇ π V π ( s ch +0 ) can be replaced by 0 if the corresponding node is a leaf node, or can be expanded further in the same way if it is a non-leaf node.
By applying this rule recursively, we ﬁnally obtain
[Equation]

[Equation]

[Equation]
Lemma A.1.
In B&B, both children MILPs MILP ch − i and MILP ch
+ i can be derived from the local
MILP MILP i and branching decision a i
=
( j, x (cid:63)j ) , with j the index of a variable in MILP i , and x (cid:63)jthe value to be used for branching.
Proof.
From the deﬁnition of B&B in Section 2, MILP ch − i (resp.
MILP ch + i ) consist of MILP iaugmented with the additional constraint x j ≤ (cid:98) x (cid:63)j (cid:99) ( resp.
x j ≥ (cid:100) x (cid:63)j (cid:101) ).
Proposition 4.3.
A vanilla B&B algorithm that satisﬁes Assumption 4.2 forms a tree MDP.
Proof.
We shall now prove that, under Assumption 4.2, the B&B process can be formulated as atree MDP tM =
( S , A , p init , p − ch , p + ch , r, l ) , with states s i =
( MILP i , GUB i ) and actions a i = ( j, x (cid:63)j ) .
First, the algorithm starts at the root node with an initial MILP, MILP 0 , and an initial global upperbound GUB 0 = ∞ .
Thus, the root state s 0 follows an arbitrary, user-deﬁned MILP distributionp init ( s 0 ) , which is independent of the B&B algorithm.
Second, Lemma A.1, together with Assumption 4.2, ensures the existence of (deterministic) distributions p − ch
( s ch − i | s i , a i ) and p + ch
( s ch
+ i | s i , a i ) ,from which the B&B children states s ch − i and s ch
+ i are generated.
Third, the reward function r ( s i )is not part of the B&B algorithm, and can be arbitrarily deﬁned to match any (compatible) B&Bobjective.
Last, the leaf node indicator l ( s i ) is exactly the vanilla B&B leaf node criterion, and isobtained by solving the LP relaxation of MILP i constrained with upper bound GUB i , which resultsin either an infeasible LP (leaf node), a MILP-feasible LP solution (leaf node), or a MILP-infeasible
LP solution (non-leaf node).
This concludes the proof.
Proposition 4.4.
In Optimal Objective Limit B&B (ObjLim B&B), that is, when the optimal solutionvalue of the MILP is known at the start of the algorithm (GUB 0 = GUB (cid:63) ), Assumption 4.2 holds.
Proof.
Because GUB
=
GUB (cid:63) , the initial global upper bound is equal to the optimal solution valueto the original MILP.
Then, B&B will never be able to ﬁnd a feasible solution that tightens that bound,and we necessarily have GUB i = GUB 0 , ∀ i .
Hence GUB ch − i =
GUB ch
+ i
= GUB i , and both GUB ch − iand GUB ch
+ i can be directly derived from s i .
This concludes the proof.
Proposition 4.5.
In Depth-First-Search B&B (DFS B&B), that is, when nodes are processed depthﬁrst and left-ﬁrst by the algorithm, Assumption 4.2 holds.
Proof.
First, it is trivial to show that GUB ch − i can be derived from s i .
Because node i is not a leafnode, it has not resulted in an integral solution, and hence processing node i does not change the
GUB.
And since ch − i is processed directly after node i , we necessarily have GUB ch − i =
GUB i .
This,combined with Lemma A.1, shows that s ch − i can be inferred from s i and a i .
Second, we show how
GUB ch
+ i can be derived from s i and a i .
Because node ch + i is processed right after the whole subtreebelow ch − i has been processed, GUB ch
+ i is necessarily the minimum of GUB i and the optimal solutionvalue of MILP ch
− i .
Now, because s ch − i can be inferred from s i and a i , MILP ch − i can be recovered aswell, and solved to obtain its optimal solution value.
Therefore, GUB ch
+ i can be recovered from s iand a i .
This, together with GUB ch − i
=
GUB i , concludes the proof.
[Figure]
A.3 Extended results
Here we provide all training curves (Figure 3) and the extended evaluation results (Table 2) with thegeometric mean of the solving times in seconds (Time) and the geometric mean of the ﬁnal B&Btree size (Nodes).
The results are averaged over the solving runs that ﬁnished successfully for allmethods.
This is, if a solving run reached the time limit for any method, this is excluded from theaverage.
Table 3 shows the number of solving runs that timed out per method.
A.4 SCIP’s default branching rule
SCIP assigns maximum priority by default to the hybrid branching rule [2].
This means that thechoice of branching variable is based on a weighted sum of different criteria.
The biggest weightis placed on the variable’s pseudocosts.
A variable’s pseudocost is calculated as a function of thechange in LP objective value we observe (on each of the branches) as a consequence of branching onthat variable.
This value can be explicitly calculated by tentatively branching on candidate variables(in which case the rule is called strong branching), or estimated based on past observed values.
SCIPruns strong branching until it has stored a sufﬁcient amount of observations for each variable, andthen switches to the estimation strategy.
Other than pseudocosts, SCIP also considers informationabout the implied reductions of other variables’ domains and conﬂicts where the variable is involved,though with smaller importance.
It is important to consider that a call to strong branching can trigger a series of side-effects within thesolver that are not accounted for in the node count.
This was ﬁrst observed by Gamrath and Schubert[12], who point out that this gives an unfair advantage to methods that use strong branching whencomparing branching rules according to the ﬁnal tree size.
A.5 Instance collections
This section presents the models used to generate our instance benchmarks.
The parameters used togenerate each benchmark are shown in Table 4.
[Table] [Table]
[Table]
A.5.1 Combinatorial auctions
For m items, we are given n bids {B j } nj =1 .
Each bid B j is a subset of the items with an associatedbidding price p j .
The associated combinatorial auction problem is of the following form:maximize
[Equation]
subject to (cid:88)
[Equation]
where x j represents the action of choosing bid B j .
A.5.2 Set covering
Given the elements 1 , 2 , ..., m , and a collection S of n sets whose union equals the set of all elements,the set cover problem can be formulated as follows:
[Equation]
A.5.3 Maximum independent set
Given a graph G the maximum independent set problem consists in ﬁnding a subset of nodes ofmaximum cardinality such that no two nodes in that subset are connected.
We use the cliqueformulation from [6].
Given a collection C ⊆ 2 V of cliques whose union covers all the edges of thegraph G , the clique cover formulation ismaximize (cid:88)
[Equation]
to
[Equation]
A.5.4 Capacitated facility location with unsplittable demand
Given a number n of clients with demands { d j } nj =1 , and a number m of facilities with ﬁxed operatingcosts { f i } mi =1 and capacities { s i } mi =1 , let c ij /d
j be the unit transportation cost between facility i and client j , and let p ij /d
j be the unit proﬁt for facility i supplying client j .
We try to solve the followingproblemminimize m (cid:88) i = n (cid:88)
j = c ij x ij + m (cid:88) i
= f
i y i subject to n (cid:88)
j = d j x ij ≤ s i
y i , i = 1 , ..., m m (cid:88) i
= x ij ≥ 1 , j = 1 , ..., n x ij ∈ { 0 , 1 } ∀ i, j y i ∈
[Equation]
0 , 1 } ∀ i where each variable x ij represents the decision of facility i supplying client j ’s demand, and eachvariable y i representing the decision of opening facility i for operation.
A.5.5 Multiple knapsack
Given n items with respective prices { p j } nj =1 and weights { w j } nj =1 , and m knapsacks with capacities{ c i } mi =1 , the multiple knapsack problem consists in placing a number of items in each of the knapsackssuch that the price of the selected items is maximized, while the capacity of the knapsacks is notexceeded by the total weight of the items therein.
Formally:maximize m (cid:88) i = n (cid:88)
j = p
j x ij subject to n (cid:88)
j = w j x ij ≤
c
i , i = 1 , ..., m m (cid:88) i = x ij ≤ 1 , j = 1 , ..., n x ij ∈ { 0 , 1 } ∀ i, j where each variable x ij represents the decision of placing item j in knapsack i .