A GNN-G UIDED P REDICT - AND -S EARCH F RAMEWORK FOR M IXED -I NTEGER L INEAR P ROGRAMMING
Anonymous authors
Paper under double-blind review
A BSTRACT
Mixed-integer linear programming (MILP) is widely employed for modeling combinatorial optimization problems.
In practice, similar MILP instances with only coefficient variations are routinely solved, and machine learning (ML) algorithms are capable of capturing common patterns across these MILP instances.
In this work, we combine ML with optimization and propose a novel predict-and-search framework for efficiently identifying high-quality feasible solutions.
Specifically, we first predict the solution distributions, then we search for the best feasible solution within a properly defined ball around the predicted solution.
We show that our framework is both theoretically and computationally superior to fixing strategies.
We conduct extensive experiments on four public datasets and numerical results demonstrate that our proposed framework achieves 51% and 9% better primal gaps than state-of-the-art general-purpose optimization solvers SCIP and Gurobi, respectively.
I NTRODUCTION
Mixed-integer linear programming is one of the most widely used techniques for modeling combinatorial optimization problems, such as production planning Pochet & Wolsey (2006); Chen (2010), resource allocation Liu & Fan (2018); Watson & Woodruff (2011), and transportation management Luathep et al.
; Sch ¨ obel (2001).
In real-world settings, MILP models from the same application share similar patterns and characteristics, and such models are repeatedly solved without making uses of those similarities.
ML algorithms are well-known for its capability of recognizing patterns Khalil et al.
and hence they are helpful for building optimization algorithms.
Recent works have shown the great potential of utilizing learning techniques to address MILP problems.
The work of Bengio et al.
categorized ML efforts for optimization as (i) end-to-end learning Vinyals et al.
; Bello* et al.
; Khalil et al.
(2022), (ii) learning to configuring algorithms Bischl et al.
; Kruber et al.
(2017); Gasse et al.
and (iii) learning alongside optimization Gasse et al.
; Khalil et al.
; Gupta et al.
For the sake of interest, we only focus on the end-to-end approach.
While such an approach learns to quickly identify high-quality solutions, it generally faces the following two challenges:
(I) high sample collection cost .
The supervised learning task for predicting solutions is to map from the instance-wise information to a high-dimensional vector.
Such a learning task becomes computationally expensive since it necessitates collecting a considerable amount of optimal solutions (see, e.g., Kabir et al.
(II) feasibility .
Most of the end-to-end approaches directly predict solutions to MILP problems, ignoring feasibility requirements enforced by model constraints (e
g. Yoon (2022); Nair et al.
As a result, the solutions provided by ML methods could potentially violate constraints.
We propose a novel predict-and-search framework to address aforementioned challenges.
In principle, end-to-end approaches for MILP problems require the collection of abundant optimal solutions.
However, the cost of collecting such training samples is generally impractical since obtaining optimal solutions is excessively time-consuming.
A possible approach is to learn distributions of solutions as proposed by Nair et al.
but it still demands sufficient solutions for calculations.
Hence, we consider to approximate the distribution by weighing near-optimal solutions with their corresponding objective values.
This reduces the cost of sample collections by avoiding gathering optimal solutions as mentioned in challenge (I) .
Regarding challenge (II) , we implement a trust region inspired algorithm that searches for near-optimal solutions within the intersection of the original feasible region and a properly defined ball centered at a prediction point.
The overall framework is outlined in Figure 1.
[Figure]
The commonly used end-to-end approaches usually directly fix variables by the prediction, such as in Nair et al.
(2020) and Yoon (2022).
However, such approaches could lead to sub-optimal or even infeasible sub-problems.
Rather than forcibly fixing variables, our search module looks for highquality solutions in a sub-set of the original feasible region, which allows better feasibility while maintaining optimality.
The distinct contributions of our work can be summarized as follows
• We propose a novel predict-and-search framework that first trains GNNs to predict the weighted conditional marginal probability of each variable and then constructs a trust region to search for high quality feasible solutions.
• We demonstrate the ability of our proposed framework to provide equivalently good or better solutions than fixing-based end-to-end approaches.
• We conduct comprehensive computational studies on several public benchmarks datasets and the computational results show that our proposed framework achieves 51% and 9% smaller primal gaps than state-of-the-art general-purpose optimization solvers SCIP and Gurobi , respectively. We make our code publicly available at https://anonymous.4open.science/r/predict-and-search-0F6F
P RELIMINARIES
Given a vector v ∈ R n and an index set I ⊆ { 1 , 2 , ..., n } , let v I ∈ R | I | denote a subvector that corresponds to I .
2. M IXED - INTEGER LINEAR PROGRAMMING
MILP techniques are used to model combinatorial optimization problems, and an MILP instance can be formulated as: min x ∈ D c ⊤ x , where D ≡ { x ∈ Z q × R n − q : Ax ≤ b, l ≤ x ≤ u } denotes a set of feasible solutions.
There are n variables, with c, l, u ∈ R n being their objective coefficients, lower and upper bounds, respectively.
Without loss of generality, the first q variables are discrete.
A ∈ R m × n denotes the coefficient matrix while b ∈ R
m represents the right hand side vector.
For convenience, let M ≡ ( A, b, c, l, u, q ) denote an MILP instance.
For the sake of interest, we only consider discrete variables to be binary, i.e. x i ∈ { 0 , 1 } for 1 ≤ i ≤ q .
Besides, considering continuous variables will not change either the methodology or the outcome of our approach.
[Equation]
Gasse et al.
(2019) proposed a bipartite graph representation for MILP problems.
Specifically, let
G ≡ ( V , E ) , where V ≡ { v 1 , ..., v n , v n +1 , ..., v n + m } denotes the set of n variable nodes and m constraint nodes, and E represents the set of edges that only connect between nodes of different types.
Variable nodes and constraint nodes are individually associated with the instance information e.g. degrees and coefficients (see appendix).
The adjacency matrix A is a |V| × |V| matrix that represents the connectivity of G , as shown by Equation (1).
[Equation]
where C i,j = 1 A i,j
[Equation]
=0 .
[Equation]
[Equation]
Let N ( v i ) ≡ { v j ∈ V :
A i,j ̸ = 0 } denote the set of neighbors of node v i .
We construct a k -layer
GNN as follows:
[Equation]
where function f k 1 aggregates the feature information over the set of neighboring nodes and function f k 2 combines the nodes’ hidden features from iteration ( k − 1) with the aggregated neighborhood features, and h kv i denotes the hidden state of node v i in the k th layer.
Initially, h 0 v
i is the output of an embedding function g ( · ) .
Note that the bipartite graph associated with an MILP instance does not include edges between variable nodes as well as constraint nodes.
In order to enable information aggregation across all nodes, we use two interleaved half-convolutions Gasse et al.
(2019) to replace the graph convolution.
After that, the GNN we used can be formulated as:
[Equation]
where MLP ( k ) c , MLP ( k ) v and g ( · ) are 2-layer perceptrons with ReLU activation functions.
That is, a half-convolution is performed to promote each constraint node aggregating information from its relevant variable nodes; after that, another one is performed on each variable node to aggregate information from its relevant constraint nodes and variable nodes.
2. T RUST R EGION M ETHOD
The trust region method is designed for solving non-linear optimization problems as follows:
[Equation]
where H ⊆ R n denotes a set of feasible solutions.
The main idea is to convert the originally difficult optimization problem into a series of simple local search problems.
Specifically, it iteratively searches for trial steps within the neighborhood of the current iterate point Yuan (2015).
The trial step d k is obtained by solving the following trust region problem:
[Equation]
where (cid:101) f k ( d ) is an approximation of the objective function f
( x k + d ) , H k denotes a shifted H − x k of the set H , ∥ .
∥ W k is a norm of R n , and ∆ k denotes the radius of the trust region.
After solving the problem in the k th iteration, x k +1 is updated to x k + d k or x k accordingly.
Then, new ∥ .
∥ W k +1 and
P ROPOSED F RAMEWORK
The supervised learning task in end-to-end approaches maps the instance-wise information to a highdimensional vector.
Ding et al.
(2020) predicts optimal solutions based on a GNN that learns the values of variables that stay unchanged across collected solutions.
However, such a group of variables does not necessarily exist for many combinatorial optimization problems.
Another approach is to learn distributions rather than directly learning solution mappings; Li et al.
(2018) predict a set of probability maps for variables, and then utilize such maps to conduct tree search for maximum independent set problems.
This work, while obtained remarkable results, is designed for specific tree search algorithm, while we aim at developing a general method that is suitable for most of the
MILP problems. A more general case is raised by Nair et al. (2020) to learn the conditional probability distribution in the solution space of an MILP instance; guided by a neural network, a subset of variables is fixed to reduce the problem size.
The fixing strategy can potentially accelerate the solving process, however, it assigns values to variables without explicitly considering constraints, such that the resulting sub-problem becomes infeasible.
To alleviate these issues, we adopt the idea of trust region method and design a novel approach that searches for near-optimal solutions within a properly defined region.
Specifically, we propose a predict-and-search framework that: (i) predicts weighted conditional marginal probabilities of all binary variables in a given MILP instance by a trained GNN model; (ii) searches for near-optimal solutions within the trust region constructed from the prediction.
3. P REDICT
In this part, we aim at training a GNN by supervised learning to predict weighted conditional marginal probability distribution for MILP instances.
To this end, we define the conditional probability distribution learning.
On this basis, we present the training label in the form of a vector, i.e. weighted conditional marginal probabilities.
3.1. D ISTRIBUTION LEARNING
A probability distribution learning model outputs the conditional probability distribution on the entire solution space of an MILP instance M .
A higher conditional probability is expected when the corresponding solution is more likely to be optimal.
Nair et al.
(2020) proposed a method to construct the probability distribution with objective values, and for a solution x , the conditional probability p ( x | M ) can be calculated by:
[Equation]
where E ( x ; M ) ≡ (cid:26)
c ⊤ x if x is feasible, + ∞ otherwise.
This implies that an infeasible solution leads to a probability of 0 , while the optimal solution induces the highest probability value.
Note that each instance corresponds to only one distribution, hence, the training task transforms the one-to-many learning into one-to-one learning.
For the collected dataset { ( M i , L i ) } Ni =1 , L i ≡ { x i,j } N i j =1 denotes the set of N i feasible solutions to instance M i .
The probability of each solution in the dataset can be calculated by Equation (5).
In general, the distance between two distributions can be measured by Kullback-Leible divergence.
Thus, the loss function for the supervised learning task is defined as:
[Equation]
where w i,j ≡ exp
[Equation]
c [Equation] P θ ( x i,j ; M i ) is the prediction from the GNN denoted as F θ with learnable parameters θ .
The conditional probability distribution of an MILP problem can be approximated by a part of the entire solution space.
Consequently, The number of samples to be collected for training is remarkably reduced.
3.1. WEIGHT - BASED SAMPLING
To further investigate the learning target and align labels with outputs, we propose a vector form of the label.
With the learning task specified in Equation (6), a new challenge arises that high- dimensional sampling for solution acquiring is computationally prohibitive.
A common technique is to reduce the dimension by decomposing the high-dimensional distribution into low-dimensional distributions.
Given an instance M , let x d denote the d th dimension of a solution x , Nair et al. (2020) assume that variables are independent of each other, i.e.,
[Equation]
With this assumption, the high-dimensional sampling problem is decomposed into n 1dimensional sampling problems for each x i according to their probabilities p θ ( x i | M ) .
Since p θ ( x i = 1; M ) = 1 − p θ ( x i = 0; M ) , we only need p θ ( x i = 1; M ) for i ∈ { 1 , 2 , ..., n } to represent the conditional probability P θ ( x ; M ) .
Then the conditional probability distribution mapping outputs a n − dimension vector ( p θ ( x 1 = 1; M ) , ..., p θ ( x n = 1; M )) .
Hence, the prediction of the
GNN model can be represented as F θ ( M ) ≡ (ˆ p 1 , ˆ p 2 , ..., ˆ p n ) , where ˆ p id ≡ p θ ( x d = 1; M ) for d ∈ { 1 , 2 , ..., n } .
Let S id ⊆ (cid:8) 1 , 2 , ..., N i (cid:9) denote the set of indices in L i with their d th component being 1 .
[Equation]
where p id is normalized by | L i | to a value between 0 and 1.
Given an MILP instance M , we can calculate a corresponding learning target in the form of vector, i.e, P ≡ ( p 1 , p 2 , ..., p q ) , where each component is calculated by Equation (8).
This equation calculates the weighted conditional marginal probability, where the weight is 1 if the variable holds a value of 1 in the corresponding solution and vice versa.
We define such a learning target in the form of a vector as weighted conditional marginal probability distribution .
When the weighting coefficients w ij are the same, each individual solution contributes equally to the loss function; when larger weighting coefficients are given to high-quality feasible solutions, this means they play more important roles in the loss function.
For the loss function shown in Equation (6), based on the assumption in Equation (7) and the calculation of probabilities in Equation (6), we have:
[Equation]
This indicates that the loss function can be derived as approximating the weighted conditional marginal probability of each component in a cross-entropy format.
Thus, with assumption (7), the conditional probability distribution learning is converted to a weighted conditional marginal probabilities learning.
3. S EARCH
With weighted conditional marginal probabilities as inputs, we adopted a trust region like method to carry out a search algorithm.
In this section, we first introduce our observation that the distance between the solution obtained from a rounded learning target and the optimal solution can be very insignificant.
Finally, we adopt a trust region like method to address such an issue and present a proposition to manifest the superiority of our framework.
The complete framework can be found in
Algorithm 1.
3.2. O BSERVATION
A variable’s weighted conditional marginal probability is closer to 1 if it’s more likely to be 1 in the optimal solution, and vice versa.
Given an MILP instance M and its learning target P , we set the partial solution size parameter ( k 0 , k 1 ) to represent the numbers of 0’s and 1’s in a partial solution.
Let I 0 denote the set of indices of the k 0 smallest components of P , and I 1 denote the set of indices of the k 1 largest components of P .
If d ∈ I ≡
I 1 ∪ I 0 , we get a partial solution x I by:
[Equation]
Let x ∗ denote an optimal solution to M , empirically, we found x I is close to x ∗ I as discussed in
Section 5.2.
Explicitly, we define the distance by ℓ 1 norm, and there still exists a small ∆ > 0 , such that ∥ x I − x ∗ I ∥ 1 < ∆ while ( k 0 + k 1 ) is a large number.
We speculate that, since the optimal solution has the largest weight as shown in equation (8), it is closer to the learning target than all other solutions.
As a result, we hypothesize that similar phenomena can be observed with a slightly larger ∆ and the same ( k 0 , k 1 ) when obtaining the set of indices I based on prediction F θ ( M ) .
With this observation, it is reasonable to accelerate the solving process for MILP problems by fixing variables in the partial solution.
Specifically, the sub-problem of an instance M using the fixing strategy with the partial solution x I can be formulated as:
[Equation]
where S ( x I ) ≡ { x ∈ R n : x I = x I } .
However, once x I ̸ = x ∗ I , the fixing strategy may well lead the above sub-problem to sub-optimal and even infeasible, which can also be observed in Figure ?? .
3.2. S EARCH WITHIN A TRUST REGION
Under the above observation and analysis, we design a more practicable method.
Inspired by the trust region method, we use the partial solution as the starting point to establish a trust region and search for a trial step, and then the trial step can be applied to the starting point to generate a new solution.
Specifically, given instance M , we acquire the set I via the prediction F θ ( M ) from a trained GNN, and get a partial solution ˆ x I by equation (9) to construct the trust region problem for a trail step d ∗ similar to problem (3), (4).
At last, output the point updated by trail step.
Such a trust region problem is equivalent to following:
[Equation]
where B (ˆ x, ∆) ≡ { x ∈ R n : ∥ ˆ x I − x I ∥ 1 ≤ ∆ } .
In order to reduce computational costs, we solve this problem only once to find a near-optimal solution.
The pseudo-implementation of such an algorithm can be found in the appendix.
We show that our proposed method always outperforms fixing-based methods under a fixed method of variable selection in Proposition 1.
Proposition 1.
let z Fix and z Search denote optimal values to Equation (10) and (11) respectively.
z Search ≤ z Fix , if they have same partial solution ˆ x I .
Proof.
Note that S (ˆ x I ) =
B (ˆ x I , 0) ⊂
B (ˆ x I , ∆) , it is obvious
[Equation]
i.e. z Search ≤ z Fix
□
This proposition demonstrates the advantage of our proposed search strategy over fixing strategy; that is, when fixing strategies lose optimality or feasibility as a consequence of inappropriate fixing, applying such a trust region search approach can always add flexibility to the sub-problem.
Empirically, results given in computational studies also support this proposition.
Algorithm 1 details the search algorithm of the proposed framework. It takes the prediction F θ ( M ) as input, and acquires a partial solution x d and neighbor constraints. A complete solution x is then attained by solving the modified instance M ′ with neighbor constraints added. The solving process is denoted by SOLV E ( M ′ ) , i.e. using SCIP/Gurobi to solve instance M ′ .
Algorithm Parameter : size { k 0 , k 1 } , Radius of the Area: ∆ Input : Instance M , Probability prediction F θ ( M )
Output : Solution x ∈ R n 1: Sort the {+components in F θ ( M ) from smallest to largest to obtain sets I 
 and I 1 .+}
{+2: for d = 1 : n do 3: if d ∈ I 0 ∪ I 1 then 4: create binary variable δ d 5: if d ∈ I 
+}
then 6: create constraint x d ≤ δ d 7: else 8: create constraint 1 − x d ≤ δ d 9: end if 10: end if 11: end for 12: create constraint (cid:80) d ∈ I 0 ∪ Iδ d ≤ ∆ 13: Let M ′ be M with added constraints and variables 14: Let x = SOLV E ( M ′ ) 15: return x
We conduct extensive experiments on four public datasets with fixed testing environments to ensure the fairness of comparisons.
Detailed model structures are also introduced in this section.
Benchmark Problems Four MILP benchmark datasets are considered in our computational studies.
Two minimizing problems come from the ML4CO 2021 competition Gasse et al.
(2022), including the Balanced Item Placement (denoted by IP ) dataset and the Workload Appointment (denoted by WA )
dataset (Gasse et al., 2022).
Then we generated two maxmization problems: Independent
Set ( IS ) and Combinatorial Auction ( CA ) using Ecole library (Prouvost et al., 2020) referencing instances used by Gasse et al.
Set Covering and Capacitated Facility Location are not chosen since they can be easily solved by Gurobi and SCIP.
Detailed dimensions of selected datasets can be found in the appendix.
Graph neural networks A single layer perceptron embeds feature vectors so they have the same dimension of 64, and the layer normalization Ba et al.
(2016) is applied for a better performance of the network.
Then we adopted 2 half-convolution layers from Gasse et al.
(2019) to conduct information aggregation between nodes.
Finally, weighted conditional marginal probabilities are obtained by feeding the aggreated variable nodes into a 2-layer perceptron followed by a sigmoid activation function.
Training protocol Each dataset contains 400 instances, including 240 instances in the training set, 60 instances in the validation set, and 100 instances in the test set.
All reported numerical results are running in the test set.
Our model is constructed with the PyTorch framework, and the training process runs on GPUs.
The loss function is specified in Equation (6) and (7) with a batch size of 8.
The ADAM optimizer Kingma & Ba (2014) is used for optimizing the loss function with a start learning rate of 0.003.
Evaluation metrics We use the primal objective values obtained by running single-thread default
Gurobi for 3 , 600 seconds as the performance baseline.
To measure the performance of a given approach, we calculate the average objective values (OBJ) obtained by the approach within a certain amount of time across instances of different datasets.
For each instance, we compare objective values obtained by different approaches, and choose the best one as the best known solution (BKS).
With that, the performance gap can be formulated as: gap avg ≡ | OBJ − BKS | / (cid:0) | BKS | + 10 − 10 (cid:1) .
We compare our approach with SCIP and Gurobi by measuring their performance gaps from the BKS, and compute the improvement ratio.
Similarly, we can assess the performance gap gap p between any two approaches by replacing BKS and OBJ with their average primal gaps, i.e. gap p ≡ | OBJ a − OBJ b | / (cid:0) | OBJ a | + 10 − 10 (cid:1) for approach a and b. Evaluation Configurations For detailed configurations, please check the appendix. Data collection Details will be provided in the appendix.
R ESULTS AND D ISCUSSION
In order to investigate the benefits of applying such a predict-and-search framework, we carried out comprehensive computational studies that: (1) compare our framework with SCIP and Gurobi; (2) measure the distance between rounded label and the optimal solution to support Proposition 1; (3) demonstrate the advantages of our approach over fixing based methods.
The rest of the comparison experiments (e.g. against Neural Diving (Nair et al., 2020)), and implementation details can be found in the appendix.
For the sake of simplicity, we denote the predict-and-search framework as PS in Figures and Tables.
[Table]
5. C OMPARING AGAINST STATE - OF - THE - ART SOLVERS
We implement the predict-and-search approach with both SCIP and Gurobi to ensure the fairness.
Figure 2 exhibit in line plots the tendency of average primal gap as the solving process goes on.
In
Figure 2a, we observed a huge performance improvement of our framework (blue) comparing to the default SCIP (green), and such an improvement also exists for WA, CA and IS datasets as shown in 2b, 2c and 2d.
Changing the under-laying solver does not dramatically diminish the performance; in
Figure 2a, 2c and 2d, our approach (red) also outperforms Gurobi (black), and it is remarkable that, in Figure 2d, the predict-and-search framework obtained optimal solutions to IS problems withinseconds.
Our framework exposed a performance drawback in the early solving stage as shown in
Figure 2b, but the performance is quickly restored within 100 seconds.
We present detailed objective values for a time limit of 1 , 000 seconds in Table 1, and the objective value gaps are calculated, along with improvement ratios.
Larger gap means better performance.
Our framework outperforms SCIP and Gurobi with improvements of 50 . 8% and 9 .
3% , respectively.
We noticed that the enhancement for Gurobi is not as significant as for SCIP; it is due to the fact that Gurobi can quickly obtain high-quality solutions.
5. C OMPARING TO FIXING STRATEGIES
In this part, we compare our framework with fixing strategies to show the advantages.
We directly use the learning target to define confidence scores, and we obtain partial solutions by setting different size parameters ( k 0 , k 1 ) as stated in Section 3.2.1.
ℓ 1 norms are computed to measure such distances.
We selected (700 , 0) , (750 , 0) , (800 , 0) , (850 , 0) , and (900 , 0) as size parameters and obtained their corresponding ℓ 1 distances as 0 . 00 , 0 . 04 , 1 . 20 , 35 . 26 , and 85 .
We observed that such distances can be insignificant if only a small portion of variables are taken into consideration.
However, as we enlarge the number of involved variables, the distance increases dramatically.
Hence, the predictand-search approach can involve larger set of variables than fixing strategy does while still retaining the optimality.
To testify this argument, we directly compare the performance of the fixing strategy and our approach.
Figure 3 shows the average primal gap achieved by different approaches, where the numbers [Figure] of selected variables are fixed for both approaches.
One can spot the dominance of our approach over fixing strategy from Figure 3a; this implies that fixing strategy only leads to mediocre solutions, while the search method (red) achieves much better qualities.
In Figure 3d, both approaches are capable of finding optimal solutions instantly, but the ones obtained by fixing method is far from optimal solutions to original problems.
We notice that in Figure 3b and 3c
[Figure] our framework struggles to find good solution in the early solving phases [Figure] but produces equivalent or better solutions within 1 , 000 seconds comparing to the fixing method.
A possible reason is that our solution results in larger feasible regions comparing to the fixing strategy.
Conclusively, our framework always outperforms the fixing strategy
[Figure]
e r a g e p r i m a l g a p
[Figure]
C ONCLUSIONS
We propose a predict and search framework for tackling difficult MILP problems that are routinely solved.
A GNNs model was trained under a supervised learning setting to map from bipartite graph representations of MILP problems to weighted conditional marginal probabilities.
The key contribution of this work is that we design a trust region based algorithm to search for high-quality feasible solutions with the guidance of such a mapping.
Both theoretical and empirical supports are provided to illustrate the superiority of this framework over fixing-based strategies.
With an extensive set of publicly available MILP datasets, we demonstrate the effectiveness of our proposed framework in quickly recognizing high-quality feasible solutions.
Overall, our proposed framework achieved 51% and 9% better primal gaps comparing to SCIP and Gurobi, respectively.