Intelligent Variable Selection for Branch & Bound Methods
Saurabh Jha saurabh jha2@dell com
CFS AI Research
Dell
Sailendu Patra sailendu_patra@dell.com
CFS AI Research
Dell
Abstract
Combinatorial optimisation (Wolsey and Nemhauser [22]) is applied to a wide variety of real-world problems like job scheduling, capacity planning and supply-chain management.
These problems are usually modelled as Mixed Integer Programming (MIP) Problems and solved using the Branch and Bound (B&B) (Land and
Doig [12]) paradigm.
Branch and Bound method partitions the solution space (branching) by creating constrained sub-problems (bounding) and explores those subsets of the solution space which are highly likely to produce optimal solutions.
The efficiency of the Branch and Bound method in finding optimal solutions is heavily influenced by the variable and node selection heuristics used for branching (Linderoth and Savelsbergh [13]).
In this paper, we propose a novel deep reinforcement learning based variable selection strategy.
The proposed solution shows significant improvement over traditional branching strategies like Strong Branching (SB) (Applegate et al.
[3]), which have been traditionally used for variable selection and also outperforms the current state of the art RL-based branching strategies which use PPO (Schulman et al.
[20]) and DQN (Hessel et al.
The results of our experiments show that the proposed solution performs consistently better than the traditional and RL-based strategies across two different NP-hard problems.
Introduction
Mixed Integer Programming (MIP) formulation is an essential step in solving real-world combinatorial optimisation problems.
Real-world optimisation problems are complex in nature and are often NPhard due to the number of integer variables involved in these problems.
An increase in the number of integer variables causes an expansion in the solution space, thus, making the task of finding a solution set that minimises the optimisation objective while obeying the constraints, tedious and computationally costly.
Branch and Bound (Land and Doig [12]) (B&B) method is a widely used strategy to solve Mixed
Integer Programming problems.
Branch and Bound method performs an iterative tree search of the solution space by partitioning the solution space into disjoint subsets creating sub-problems.
The method selects sub-problems in such a fashion that it avoids exploration of subsets unlikely to produce optimal solutions and actively explores subsets of the solution space that are more likely to produce optimal solutions.
Different branching strategies, pruning and cutting rules are used to search the solution space efficiently, to fasten the search and obtain optimal solutions.
Branching strategies are key to solving branch and bound problems, as robust branching can result in reduction in the effective size of the solution space and obtain optimal solutions at a faster rate.
Variable selection based branching (Lodi and Zarpellon [14]) determines the next variable to be used for partitioning the subspace and node selection based branching determines the next subset of the solution space to be explored.
The focus of this paper will be on variable selection based branching.
Traditional strategies for variable selection like Pseudo-Cost Branching (PCB) and Strong Branching (SB) (Applegate et al.
[3]) are manually designed using domain knowledge for each problem instance and do not generalise to other kinds of problems.
Learning to branch using machine learning methods helps in generating variable selection strategies that are non-myopic and can generalise to a broad set of problems.
Supervised Learning and Imitation Learning based approaches are a common choice for learning to branch which are trained using expert demonstrations.
These methods face issues when applied to real-world problems as the data they are exposed to differ from the expert demonstrations that were used during training phase.
Reinforcement Learning has also been adopted for variable selection in the literature.
However, it suffers from cold-start problems and cannot perform optimally until sufficient experience tuples are collected, resulting in sub-optimal early performance.
In this paper, a novel deep reinforcement learning based variable selection strategy devoid of above mentioned drawbacks is proposed.
The key contributions of the paper are as follows,
• A soft-actor critic based deep reinforcement learning framework for branching variable selection has been developed which can be used in branch & bound methods to solve Mixed Integer Programming (MIP) problems.
• The proposed solution uses experience replay, entropy regularized policy updates and double critics to enhance learning stability.
• The proposed solution uses expert demonstration data during beginning of training to avoid cold-start problem.
• The proposed solution uses MaxLP Gain construction heuristic to traverse the search tree and samples experience trajectories, to improve sample efficiency and avoid partial observability.
• We assess the performance of our proposed strategy on non-adaptive heuristic based and adaptive state of the art RL based branching strategies across different kind of MIP formulations and observe that our solution generalizes well to the different scenarios.
Related Work
Traditional Variable Selection Strategies Most Infeasible Branching (MIB) (Achterberg et al.
Pseudo-Cost Branching (PC) and Strong Branching (SB) (Applegate et al.
[3]) are traditional methods which score the candidate variables based on their effectiveness in partitioning the solution space.
Most Infeasible Branching method chooses the variable with the high fractional parts (close to 0.5) and has been shown in Achterberg et al.
[1] to perform worse than when the variables were to be selected randomly.
Pseudo-cost Branching (Falk [8]) estimates an expected gain for each candidate variable based on the history of branching.
The method suffers from cold-start due to lack of history and requires the pseudo-costs to be initialised manually which might induce bias.
Strong Branching is an efficient strategy which selects candidate variables that result in maximum improvement the dual bound at every stage.
However, this strategy is computationally costly even for small problems and becomes intractable even for a slight increase in the size of the problem.
Learning based Variable Selection Strategies As traditional variable branching strategies are computationally heavy and require manual intervention, machine learning based strategies are being studied to develop improved and efficient strategies.
Expert demonstrations of Strong Branching (SB) strategy are used to train Supervised Learning based techniques like offline extra tree regression (Geurts et al.
[10]) and online linear regression (Marcos Alvarez et al.
[16]) to predict branching scores and Imitation Learning based agents which learn to branch by mimicking Strong Branching behavior.
However, these methods require extensively labelled expert data sets for training and cannot generalise well to heterogeneous problems.
Reinforcement Learning has also been used to solve the variable branching problem, Sun et al.
[21] designs a primal-dual policy network with negative unit reward for every time step to encourage the agent to learn to solve the problem with minimum number of steps.
In Parsonson et al.
[17]), Double Deep Q Network architecture with a Superior
Network is used to deal with the large action spaces.
In Ahn et al.
[2], the authors leverage deep reinforcement learning to design solvers for combinatorial optimization problems that can adaptively learn to minimize the number of stages involved in obtaining the solution by deferring the assessment of some vertices to later iterations.
In Chen and Tian [6], the authors formulate the optimization problem as a rewriting problem and perform iterative rewriting an existing solution until optimality.
Background 3.1 Mixed Integer Programming (MIP)
Definition 1 Mixed Integer Programming (MIP) Formulation Given A ∈ R m × n , b ∈ R m and c ∈ R n , and a subset I ⊆ { 1 , 2 , ..., n } , the mixed integer program MIP =
( A, b, c, I ) linear in x is defined as follows,
[Equation]
where the vectors in X are feasible solutions of the MIP if X = { x ∈ R n | Ax ≤ b, x ∈ R n , x j ∈
Z , ∀ j ∈ I } , and x ∗ ∈ X is an optimal solution if c T x ∗ =
z ∗ .
The problem is NP-Hard due to the integer requirements on x. MIP problems are solved by recursively splitting the problem into sub-problems using Branch and Bound method, where the LP relaxation of the problem is used to circumvent the integerality requirements and to obtain the lower bound for the sub-problems.
Definition 2 LP Relaxation for a (MIP) Problem The LP relaxation of the MIP is defined as follows,
[Equation]
The lower-bound for the sub-tree is provided by solving the above LP-relaxation and if the objective value of the LP relaxation ˇ z ≥
ˆ z where ˆ z =
c T ˆ x of the current best solution ˆ x , then the node can be discarded.
The optimal integer solution will always lie between the determined upper-bound and lower-bound for the node to be explored.
The choice of the variable x j to be used for calculating these bounds, the choice of the nodes to be explored and the cutting rules are different strategies used in solving MIP problems.
3.2 Branch & Bound (B&B) Algorithm
For any optimisation problem defined as O =
( W, f ) with an objective function f : W → R and
W as the search space which is the set of all valid solutions to O , the branch and bound algorithm aims to find an optimal solution x ∗ ∈ argmin x ∈ W f ( x ) .
The search space W is partitioned into disjoint sub-spaces by constructing a search tree T , which is then iteratively partitioned by selecting a subspace G ⊂ W from the unexplored sub-spaces and checking for ˆ x ′ ∈ G has f (ˆ x ′ ) <
f (ˆ x ) for a feasible ˆ x ∈ W stored globally.
If above condition is satisfied then the incumbent solution is updated and the subspace G is branched into sub-spaces G 1 , G 2 , ..., G k and the search continues iteratively.
If f (ˆ x ′ ) ≥ f (ˆ x ) ∀ x ∈ G , the subspace is abandoned and not searched further.
Once the search is complete with no unexplored sub-spaces, the best incumbent solution is returned and the search terminates.
Proposed Solution
Variable Selection plays an important role in branch and bound methods as it prescribes the variable on which branching is to be performed.
Selecting a wrong variable for branching can result in increase in the size of the search tree and can increase the computational complexity significantly.
Existing methods score the different variables and perform branching greedily by ordering the variables in terms of the score.
The goal is to develop a high-quality far-sighted variable selection strategy that is capable of generalising to unseen expert data.
The proposed solution leverages reinforcement learning to achieve above goals by modelling the variable selection process as a Markov Decision
Process (MDP) (Puterman [19]).
4.1 MDP Formulation
The sequential decision making process for variable selection is formulated as a Markov Decision
Process.
With the environment represented by the solver, the brancher agent states, actions, transition function and the reward function are described as follows,
State Space S
At each instance t , the problem state s t consists of the bi-partite graph representation for the MIP of the focus node and the index set I t of the branching candidates which posses constraints on integers and have a non-integer solution.
The state s t is defined as below,
[Equation]
where X ∈ R n × d x is a feature matrix describing the variables and the features from the objective function and the constraints.
The nodes of the bi-partite graph represent the n variables to be optimised and the m constraints to be met.
An edge e ij is created if the variable x i has a corresponding
A ij ̸ = 0 in the constraint c j with features d e representing the constraint’s constant term resulting in E ∈ R m × n × d e , the edge feature matrix.
C ∈ R m × d c represents the constraint matrix where the constraints are encoded into d c features like in (Gasse et al.
Thus, the state s t presents a holistic representation of the variables, the objective function and the constraints involved in the MIP at any instance t .
Action Space A
The action a t represents the branching variable selected at instance t from the action space where I t is the index set of branching variable candidates.
[Equation]
Transition Function P
The state transition function P ( s t +1 | s t , a t ) generates the next state provided the current state and action.
The next state s t +1 is given by using a node selection policy π n which selects the next node n t to be branched upon as shown in fig. 1, whose features will be represented by s t +1 .
However, the environment changes independently of the action of the brancher agent as it cannot perceive the actions of the node selection policy resulting in partial observability.
To circumvent partial observability, multiple trajectory paths are constructed such that the root-leaf pairs are used as the source-destination pair and each node on the search tree is mapped to a single trajectory.
This generated trajectory information is then stored in the replay buffer and used during the training process.
Reward Function R
The goal of our solution is to be able to obtain an optimal solution faster with minimum branching steps as possible.
Thus, we define the reward R t at instance t as follows,
[Equation]
(5) where for every step a unit penalty is issued.
The cumulative reward of an episode will represent the number of steps to optimal solution as a penalty, thereby encouraging the method to solve in minimum number of steps.
4.2 Solution
A novel deep reinforcement learning algorithm for variable branching based on Soft Actor Critic
Networks (Christodoulou [7]) is proposed.
The soft actor critic algorithm aims to learn an optimal actor π ∗ which maximises the expected discounted reward and the entropy H associated with it.
[Equation]
where H ( π ( .
| s t ))
= − log π ( .
| s t ) , α is the temperature parameter that manages the explorationexploitation trade off.
The actor network and the critic network are modelled using Graph Convolutional Neural Network (GCNN) as the state s t is represented as a bipartite graph.
Two successive passes to perform a single convolution as below, c ′ p ← f c   c p , ( p,q ) ∈ E (cid:88)
q g c ( x q , e pq , c p ) 
 , (7) x ′ q ← f x   x q , ( p,q )
∈ E (cid:88)
p g x ( x q , e pq , c ′ p ) 
 , (8) ∀ p ∈ C, ∀ q ∈ X , followed by a double-layer perceptrons with ReLU activation.
The actor network uses softmax to estimate the action probability distribution π and the critic network uses the masked sum to estimate the state-action value Q .
The action generated by the soft-actor agent a t represents the branching variable and the next node n t to be branched at instance t is selected by a node selection policy.
The sub-tree with n t as the root node is traversed by selecting the leaf with the largest LP gain until a leaf node not yet fathomed is reached.
Trajectories are generated for each of the nodes present in the sub-tree and are added to the experience replay buffer.
The node selection policy is used when the soft actor-critic agent interacts with the environment and generates a variable for branching during both training and inference phases.
However, the maximum LP-gain trajectory generation happens only during training phase having no additional inference-time overhead.
During the initial training phases, the brancher agent is naive due to lack of experience causing random actions to be taken, which can result in sub-optimal initial performance.
The experience buffer filled with expert demonstration data at the beginning of training process will help close the performance gap during this time.
The proposed solution is provided with tuples of expert demonstration data { s t , a t , r t , s t +1 , done t } in the replay buffer during the beginning of training which is gradually phased it out in a linear matter as new experience tuples are collected by the brancher agent, to avoid cold-start problem during beginning of training.
1 for episode = 1 to E do do
Set
Explore Traj ← MaxLPGain t t
Store
for i to M do
Sample samples
Set y t
= r t + γ min
j =1 ,
[Equation]
Q
[Equation]
( s t +1 , a t +1 )
− α log
[Equation]
( a t +1 | t +
Minimize loss
if t % d == 0 then
Update policy π with gradient ∇
[Equation]
Update target critic network parameters as θ Q ′ j = τθ Q j + (1 −
[Equation]
) θ Q ′ j ; end end end 24 end
Results and Discussion
In this section, the results of our experiments are evaluated and examined for insights and improvement areas.
The performance of our proposed solution is studied under two different NP-hard problem settings where we compare the performance of our solution with existing solutions of different levels of control complexity as follows,
Heuristic-based control – Describes the variable selection methods that are hand-crafted and non-adaptive.
The solution is compared against the traditional variable selection method, strong branching (SB) ; 2.
Adaptive Control - Describes the variable selection methods where the actions are prescribed based on recommendations made by reinforcement learning based controllers.
The performance of our solution which leverages SAC is compared to the methods which use PPO & DQN.
The experiments were run using open-source Ecole (Prouvost et al.
[18]) and PySCIPOpt (Maher et al.
[15]) libraries with SCIP solver.
Two different NP-hard problems were considered: 1) Set
Covering (Balas and Ho [4]) referred to as MIP setting 1 and 2) Maximum Independent Set
(Bergman et al.
[5]) referred to as MIP setting 2, across which the SB and RL-based controllers PPO & DQN were evaluated in comparison to our solution.
The comparison to SOTA tuned commercial solvers is not made, as we do not claim to be competitive with them at this stage.
The PPO and DQN agents have similar MDP formulation and reward definition so that they can be compared effectively.
The node selection policy used for them is DFS search and we can see that DFS search is causing the algorithm to perform poorly as DFS becomes computationally heavy for practical MIPs.
As can be seen in figs 2 & 3, the average reward of our solution improves over PPO by 27 % and 22% in the
MIP setting 1 and MIP setting 2 respectively.
It also significantly outperforms DQN control by 70 % and 58 % for both the MIP settings.
The proposed strategy exhibits steady initial learning behavior unlike the PPO and DQN based branching strategies, this can be attributed to the expert demonstration data used in our solution to avoid cold-start.
The behavior of the different algorithms in the MIP problem 2 are in-line with the performance of the algorithms on the MIP problem 1.
It can be seen that the episodic reward for
SAC in the MIP problem 2 starts at lower levels than that of the PPO and DQN.
However, it exhibits steady episode-over-episode learning behavior until around 100 episodes, after which the agent’s learning stabilises.
DQN exhibits very slow learning as it ascends with much lower slope and takes until around 160 episodes to have stabilised learning.
The efficiency of our framework in solving the CO problems can be inferred from the average number of nodes and the average number of LPs in Table 1.
The proposed strategy is able to reduce the average number of nodes and average number of LPs significantly.
SAC reduces the average number of nodes over PPO by 30% and 16% over the MIP problem 1 and MIP problem 2 respectively.
It also shows significant impact in the average number of nodes over DQN by around 40% and 20% for the different problems.
The average numbers of LPs exhibit a decreasing trend across the different
[Table]
problem settings.
Also, it performs as good as the SB control significantly across the different MIP problems.
Thus, the improvement trend of our proposed strategy over PPO and DQN control is consistent across the different MIP formulations, elucidating that our method is robust and scalable to different kind of problems and significantly outperforms state of the art control strategies.
Conclusion
In this paper, we proposed variable selection strategy for Branch & Bound Methods leveraging
Soft Actor Critic.
Our method is capable of performing variable selection for different kinds of
Mixed Integer Programming problems as it is supplemented by demonstration data to avoid cold-start and trajectory generation that alleviates partial observability and increases sample efficiency.
The proposed method outperforms existing rule-based and reinforcement learning based control strategies like Proximal Policy Optimization (PPO) and Deep Q Networks (DQN) and the traditional variable selection method Strong Branching control (SB) across different MIP problems.
The solution is being studied for larger MIP problems and detailed experiments to observe its scores, wins and performance comparisons with different constructions heuristics are being performed.
The findings will be communicated in a detailed version of this paper.