Neural Large Neighborhood Search
Ravichandra Addanki MIT addanki@mit.edu Vinod Nair
DeepMind vinair@google.com Mohammad Alizadeh MIT alizadeh@csail.mit.edu
Introduction
Large Neighborhood Search (LNS)
[43, 39] is a powerful local search paradigm for solving combinatorial optimization problems.
LNS methods have been successful in many challenging problems, including Mixed Integer Programs (MIPs) [17, 40, 13, 20], Traveling Salesman Problem (TSP) [44],
Vehicle Routing Problem (VRP) [43, 27], and Constraint Programming [38, 15].
Like other local search methods, LNS starts with a feasible assignment of the variables being optimized.
At each iteration, it searches in a neighborhood of the current assignment, and if it ﬁnds an assignment with a better objective value it updates the current assignment.
It repeats this procedure until the search budget is exhausted.
In LNS, the number of neighboring assignments at each iteration is too large for exhaustive search (e.g., exponential in the number of variables).
The choice of the search neighborhood at each iteration is crucial for LNS to be effective.
It should 1) contain assignments that allow large objective value improvements in a few LNS iterations, and 2) be computationally tractable to search.
Designing a neighborhood selection policy that jointly considers these two requirements is challenging.
Successful applications of LNS rely on a domain expert to consider the problem structure and the cost of neighborhood search to time-consumingly handcraft a problem-speciﬁc policy [39].
For example, in the case of MIPs [49], an LNS algorithm called
RINS [17] at each iteration un-assigns the values of the subset of variables whose linear relaxation solution disagrees with the current assignment.
This results in a smaller MIP that can be efﬁciently solved using an off-the-shelf MIP solver to ﬁnd a better assignment.
We propose a Reinforcement Learning (RL) [46] approach to learn a neighborhood selection policy for LNS.
Given a problem instance and a feasible assignment, a learnable policy deﬁnes a search neighborhood around the current assignment.
The resulting search problem has the same form as the original one but is signiﬁcantly smaller, so even an existing (non-learned) solver can be effective on it.
The RL environment runs such a solver to ﬁnd a new assignment.
The reward is a function of both the improvement in objective as well as the computational effort for the neighborhood search.
We train the policy on problem instances drawn from a speciﬁc domain.
This allows automatically learning domain-speciﬁc neighborhood selection policies that can outperform generic policies.
To demonstrate our approach, we focus speciﬁcally on mixed integer programming (MIP), an NPcomplete combinatorial optimization problem.
See Figure 1.
The reward function is deﬁned as the primal integral [14] (section 2.1), which combines both solution quality and search effort into a scalar metric.
We express the neighborhood selection policy as a graph convolutional neural network (GCNN)
[9] that operates on a bipartite graph representation of the MIP.
Related Work Relaxation Induced Neighborhood Search (RINS) [17] is a well-known large neighborhood search meta-heuristic for improving a given feasible MIP solution.
It compares the feasible solution to the solution obtained by relaxing the integer variables, and destroys variables whose values differ between the two.
The resulting sub-MIP is then solved using a MIP solver.
Adaptive
LNS [26] uses an ensemble of LNS algorithms for MIPs with a multi-armed bandit to adaptively switch among them during a MIP solve.
While we do not focus on an ensemble approach, our work can be used as another ensemble member to improve performance.
Two other works apply learning to large neighborhood search.
Hottung et al.
[28] learn Repair using
RL for the capacitated vehicle routing problem.
This is complementary to our approach and can be
[Figure]
C u rr en t c o m p l e t e a ss i gn m en t
Initialize
C u rr en t pa r t
i a l a ss i gn m en t
MIP to be optimized
Destroy K variables
Repair K variables
[Figure]
combined.
We focus on learning Destroy because the sub-problem in Repair can be made small enough for an existing solver to work well.
Concurrent with our work, Song et al.
[45] propose to learn Destroy using imitation learning and RL for MIPs.
There are two key differences from our work.
First, their Destroy policy partitions the variables into disjoint subsets and iteratively selects one subset at a time.
We do not impose such a restriction.
Second, their per-step reward is the change in objective value after each Repair step, which ignores its computational expense.
We use a reward that explicitly takes into account the computational effort for Repair .
Several works apply learning to solving MIPs, such as for branching [25, 31, 5, 19, 51, 50], adding constraints [47], and primal heuristics [32].
These are complementary approaches that aim to improve different components of a MIP solver and can be combined with ours.
More generally, our work is an instance of learning for combinatorial optimization - see Bengio et al.
[10] for a recent survey.
Approach 2.1 Background: Integer Programming
An Integer Program is deﬁned as min x { d T x |
Ax ≤ b, x i ∈
Z ∀ i } , where x ∈ Z n is an n -dimensional integer vector being optimized, A ∈ R m × n and b ∈ R m deﬁne linear constraints on x , and d ∈ R n deﬁnes the linear objective function.
A Mixed Integer Program admits both continuous and integer variables.
We consider an Integer Program here to simplify notation, but our approach applies equally to MIPs as well.
Removing integrality constraints { x i ∈ Z } results in a Linear
Program (LP).
2.2 MDP Formulation
We consider a contextual MDP [3, 23] M c parameterized with respect to a context c .
Here we deﬁne c to be the parameters of an Integer Program, i.e., c = { A, b, d } .
We assume that it is computationally easy to compute a feasible assignment given a MIP.
Although theoretically an NP-hard problem, ﬁnding a feasible solution is often much easier in practice than ﬁnding the optimal solution for many
MIPs ( e.g., Minimum set cover problem [48]).
Figure 1 illustrates how the learned neighborhood selection policy is applied to a MIP.
For a given M c , at each step of an episode the policy selects one of the variables to un-assign ( Destroy ) from its current value.
At every K -th step (where K is a hyper-parameter), an integer program containing only the K un-assigned variables is deﬁned.
The remaining variables are treated as constants with their values ﬁxed to that in the current assignment.
The resulting integer program (called sub-MIP) is then solved using an existing MIP solver to assign values to the K destroyed variables.
This is continued until a stopping criterion is satisﬁed or the maximum episode length is reached.
Reward function: We deﬁne the reward based on the primal integral [14].
It is normalized to be in [0 , 1] and does not depend on the magnitude of the problem parameters.
It also accounts for how quickly an optimal solution is reached.
Computing the primal integral requires knowing the optimal objective value, but since the reward is needed only during training, it is possible to pre-compute the optimal objective value for each training problem ofﬂine.
A lower primal integral value corresponds to a better optimizer, so the reward is deﬁned as its negative.
2.3 Policy Network Architecture
Given a MIP instance, we construct an undirected bipartite graph G =
( V, C, E, G )
using the
Constraint-Variable Incidence Graph (CVIG) model [6].
In the CVIG model, vertices correspond to either variables or constraints, and edges correspond to the occurrence of a variable in a constraint with a non-zero coefﬁcient.
We use Graph Convolutional Neural Networks (GCNNs) to parameterize our policy and value networks [9, 22, 42, 24, 34].
Due to their permutation invariance and generalization capabilities, GCNNs are a natural choice for learning representations over graphs and have been applied in successfully solving a wide range of problems with graph-structured data [24, 34, 41, 8, 21].
Our GCNN network takes the state representation, s t
=
( V, C, E, G ) as input and iteratively computes embeddings for the variable and the constraint nodes.
Details about the policy architecture can be found in the Appendix.
2.4 Learning Method
For training the policy, we use the V-trace algorithm from Espeholt et al.
V-trace is an off-policy actor-critic algorithm designed for multi-task training at a massive scale.
Our training framework consists of actors for policy evaluation and a learner for policy optimization similar to [18].
Actors collect trajectories using randomly sampled MIPs from the training dataset.
The collected trajectories are streamed to an experience replay buffer [36], and are sampled uniformly by the learner for training.
The update rules used for training are detailed in [18] and reproduced for completeness in the Appendix.
Experiments 3.1 Datasets
We evaluate our approach against a wide variety of challenging benchmark MIP datasets from the literature [19, 29, 30, 33, 35, 16, 11, 7].
We use data sets of MIPs of four different kinds labeled cauction , facilities , indset , setcover .
Details about these data sets can be found in the Appendix.
3.2 Baselines
We use the following heuristics as baselines for comparison against our approach.
Random :
A policy that selects K integer variables to Destroy by sampling from a uniform random distribution.
LeastIntegral :
Optimize the relaxation of each integer variable one at a time while setting the remaining integer variables to their corresponding values in the current solution.
Select the top- K integer variables with relaxed solutions furthest away from the current solution.
Any ties for the top- K are broken randomly, similar to randomized rounding heuristics for solving MIPs [12].
Most-Integral :
Identical to the Least-Integral heuristic above, except we select the top- K variables with the relaxed solutions closest to the incumbent solution.
RINS :
We modify the Relaxation Induced Neighborhood
Search (RINS) heuristic proposed by Danna et al.
[17] to a local move policy so that it is applicable outside of the branch-and-bound framework for which it was originally designed.
In our adaptation of RINS, we compare the current solution with the optimal solution of the LP relaxation and ﬁx those integer variables that take the same value in both.
Among the remaining variables, we choose at most
K integer variables to Destroy using the random policy from above.
3.3 Training Details
To evaluate the generalizability of NLNS, we train a separate policy for each of the datasets and use the trained policy for prediction during LNS on unseen samples from the test set.
For each individual training session, we tune the hyper-parameters K and the learning rate to maximize performance on the validation set.
See Appendix for a complete list of hyper-parameters and values used for training.
Results
In this section, we present the results of our evaluation.
We use the primal gap and the primal integral (Section 2.1) as our evaluation metrics.
[Figure] [Figure]
Standalone Evaluation.
Figure 2 shows the average primal integral achieved by NLNS on test problem instances for each dataset.
NLNS is at least on-par and in some cases substantially better than the baselines across all datasets.
On average, NLNS achieves a relative improvement of 2.1 × over the best baseline across all datasets, with a maximum improvement of 4.5 × on the facilities dataset.
Evaluation as a primal heuristic.
LNS is typically used in solving MIPs as a primal improvement heuristic [12, 17, 13].
Primal heuristics run alongside the branch-and-bound algorithm to quickly ﬁnd feasible solutions of high quality.
Figure 3 shows the average primal integral of the branch-and-bound algorithm with NLNS on the test datasets.
NLNS achieves an average relative improvement of 1.9 × over the best baseline across all datasets, and a maximum improvement of 4.2 × on the cauction dataset.
Importantly, the best baseline changes for different datasets, while NLNS consistently provides the strong performance because it is able to tailor its neighborhood selection policy to each dataset.
Conclusion
We presented NLNS, a reinforcement learning approach to automatically learning problem-speciﬁc neighborhood selection polices for LNS.
NLNS selects neighborhoods that guide an existing solver to high-quality assignments efﬁciently.
Future work could enable the policy to dynamically select the size of the neighborhood K .
We will open-source our code, datasets, and the complete list of hyper-parameters used to reproduce our experiments once the paper is published.
Appendix Primal Gap
Let us denote an Integer Program as
[Equation]
, where x ∈ Z n is an n -dimensional integer vector being optimized, A ∈ R m × n and b ∈ R m deﬁne linear constraints on x , and d ∈ R n deﬁnes the linear objective function.
Following the notation from [14], given a solution x and an optimal solution x opt we deﬁne the primal gap γ ( x ) as
[Equation]
Primal Integral
Let γ i denote the primal gap of the best solution after expansion of the i th branch-and-bound node.
If no solution has been found, we use a default value of γ i = 1 .
Let N denote the total number of branch-and-bound nodes expanded so far.
Then primal integral function P ( N ) of a run is deﬁned as follows:
[Equation]
Primal integral also requires measuring the resource usage needed to achieve a given objective function value.
The original deﬁnition uses running time [14], but that is hardware-dependent.
Instead we use the total number of branch-and-bound nodes required across the calls to the Repair , which is hardware-agnostic.
Datasets
We evaluate our approach against a wide variety of challenging benchmark MIP datasets from the literature [19, 29, 30, 33, 35, 16, 11, 7].
We use data sets of MIPs of four different kinds labeled cauction , facilities , indset , setcover .
We provide references to the original sources where details to generate them can be found.
We also provide a brief description below for completeness.
Combinatorial Auction ( cauction ): Following Leyton-Brown et al.
[35], we generate combinatorial auction instances with 100 items and 500 bids.
Capacitated Facility Location ( facilities ): Following
Cornuéjols et al.
[16], we generate capacitated facility location instances with 100 facilities and 100 customers.
Maximum Independent Set ( indset ): Following Bergman et al.
[11] we generate maximum independent set instances on Erdos-Rényi random graphs with 500 nodes, with afﬁnity set to 4 (see [19]).
Set Cover ( setcover ): Following Balas and Ho [7], we generate set cover instances with 500 rows and 1000 columns.
In order to ﬁlter out trivial instances and bolster hardness, we apply several rounds of pre-processing before ﬁnally extracting useful features from the raw MIPs.
Details about the two steps can be found below.
All datasets except for corlat have been taken from [19].
Dataset Pre-processing & Feature Extraction
We generate a large number of sample problem instances and solve them to optimality, recording the primal integral achieved during this process.
We select only those instances with primal integral at or above the 99 .
9 th percentile to include in our datasets.
Finally we randomly split this ﬁltered set into training, validation and test sets with 1000 , 100 and 100 samples respectively.
COR LAT ( corlat ):
About 2000 MIP instances are generated based on real data used in the construction of a wildlife corridor for grizzly bears in the Northern Rockies region [29].
This dataset is part of the ACLIB benchmark library [30], which is a standard for evaluation of different methods for solving hard computational problems.
Each MIP instance in the dataset is passed through the following stages of pre-processing and extraction before it can be used for training or evaluation.
[Figure]
Pre-processing: The input MIP instance is transformed into an equivalent but a more compact formulation by passing it through the presolve routine in SCIP.
Presolving is a relatively quick step run at the beginning of the solving process that usually results in instances with fewer variables and constraints that are easier to solve.
Feature Extraction: We extract several static features corresponding to constraints, edges and variables similar to [19, 33].
Additionally, other dynamic features are incorporated into the variable features.
A complete list of these features is provided below.
Solving MIP: Finally, the MIP and its continuous relaxation are solved to optimality.
The optimal objective and solution are stored alongside the dataset to be used to calculate the reward during training and report the ﬁnal performance during evaluation.
Note that the optimal solution is not used as an input feature or for any other purpose during training or evaluation.
The ﬁrst feasible solution found by SCIP during the solving process is also stored to be used as the starting solution for the LNS during training.
We will release all our datasets and the code required to reproduce our results once the paper is published.
Features
We provide a brief list of features used for training below.
For a comprehensive list, please see the
README ﬁle in our codebase.
Variable Features:
Variable feature vector v i comprises of several static and dynamic features.
Static features assigned to the variables include the solution to the LP relaxation, coefﬁcients in the objective function, ﬂags indicating the variable types (discrete/continuous).
Some of the key dynamic features include the current incumbent solution, if and when the agent has called Destroy on the variables in the current local move etc.
Constraint and Edge Features:
All the constraints are ﬁrst converted to ≤ form.
Feature vector c i of the constraint-nodes is simply the value on the right hand side of the constraint.
For each edge, e = ( e k , v k , c k ) , the feature vector e k is the normalized coefﬁcient of the variable corresponding to v k in the constraint corresponding to c k .
Global Features:
Among others we use the following global features G alongside the bipartite graph features: objective value of the current solution, number of Destroy moves left in the current local move, number of local moves completed so far in this episode etc.
Policy Network Architecture:
Our GCNN network takes the state representation, s t
=
( V, C, E, G ) as input and iteratively computes embeddings for the variable and the constraint nodes.
This proceeds in T rounds as deﬁned below.
Let the variable node embeddings at round t be v ( t ) k and the constraint node embeddings be c ( t ) k .
We initialize these embeddings with features from §3.4 i.e., v (0) k
= v k , c (0) k
= c k .
Each round involves two sequential stages of message passing, from variables to constraints and then from constraints to variables which we refer to as v-to-c and c-to-v respectively.
See Figure 4 for a visual illustration.
In the v-to-c stage, for each edge e ( j,i ) incident to the constraint-node c i from the variable-node v j , a “message” m ( j,i ) is computed as m ( j,i ) =
g ( t ) (cid:16) c ( t ) i , v ( t ) j , e ( j,i ) (cid:17) where g ( t ) is a multi-layer perceptron (MLP).
These messages are pooled using mean aggregation and passed as input along with c i to f ( t ) , another MLP network.
The ﬁnal update for the v-to-c stage takes the following form:
[Equation]
where ξ c i is the set of all variable-nodes with an edge to the constraint-node c i and | .
| denotes the cardinality operator.
Once the constraint-node embeddings c ( t +1)
i are computed using the update rule (2), they are used to compute the updated variable-node embeddings v ( t +1)
i in a similar way using the following rule where p ( t ) , q ( t ) are MLPs:
[Equation]
Finally, after T rounds of updates, we get v ( T ) i , c ( T ) i .
We discard the constraint-node embeddings.
A ﬁnal MLP is applied independently to the variable-node embedding and the global features independently at each node.
A masked softmax function produces the ﬁnal policy action distribution π ( a | s t ) .
All of our MLPs use the ReLU activation function [37].
Full list of all the architectural hyper-parameters can be found in the Appendix.
Hyper Parameters
A full list of the hyper-parameters will be made available along with the code which will be released upon publication.
A brief list of few key hyper-parameters in our approach is provided below:
Hyper Parameter
Value
Learning Rate 10 −
Entropy Regularization Constant 2 ∗ 10 −
GNN Dept
Number of Local Moves
Mini-batch size
Trajectory Length
Node embedding dimension
Edge embedding dimension
Global embedding dimension
Policy Torso Hidden Layer Sizes 64 ,
Value Torso Hidden Layer Sizes 64 ,
Replay memory size
Implementation Details
We use TensorFlow [2] framework with GraphNets library [9] for implementing and training our approach.
Policy optimization in learner and batched inference in actors are accelerated using
NVIDIA V100 GPUs [1].
We use SCIP (version: 6.0.1) [4] as our default MIP solver.