A Solver-Free Framework for Scalable Learning in Neural ILP Architectures s
AffiliationAddress email
Abstract
There is a recent focus on designing architectures that have an Integer Linear
Programming (ILP) layer within a neural model (referred to as Neural ILP in thispaper).
Neural ILP architectures are suitable for pure reasoning tasks that requiredata-driven constraint learning or for tasks requiring both perception (neural) andreasoning (ILP).
A recent SOTA approach for end-to-end training of Neural ILPexplicitly defines gradients through the ILP black box [Paulus et al., 2021] – thistrains extremely slowly, owing to a call to the underlying ILP solver for everytraining data point in a minibatch.
In response, we present an alternative trainingstrategy that is solver-free , i.e., does not call the ILP solver at all at training time.
Neural ILP has a set of trainable hyperplanes (for cost and constraints in ILP),together representing a polyhedron.
Our key idea is that the training loss shouldimpose that the final polyhedron separates the positives (all constraints satisfied)from the negatives (at least one violated constraint or a suboptimal cost value),via a soft-margin formulation.
While positive example(s) are provided as partof the training data, we devise novel techniques for generating negative samples.
Our solution is flexible enough to handle equality as well as inequality constraints.
Experiments on several problems, both perceptual as well as symbolic, whichrequire learning the constraints of an ILP, show that our approach has superiorperformance and scales much better compared to purely neural baselines and otherstate-of-the-art models that require solver-based training.
In particular, we are ableto obtain excellent performance in 9 × 9 Visual Sudoku, to which the other Neural
ILP solver is not able to scale.
Introduction
There has been a growing interest in the community which focuses on developing neural modelsfor solving combinatorial optimization problems.
These problems often require complex reasoningover discrete symbols.
Many of these problems can be expressed in the form an underlying Integer
Linear Program (ILP).
Two different kinds of problem (input) settings have been considered in theliterature: (a) purely symbolic and (b) combination of perceptual and symbolic.
Solving an n-Queensproblem given the partial assignment of queens on the board as input would be an example of theformer, and solving a sudoku puzzle given the image of a partially filled board as input would bean example of latter.
While the first setting corresponds to a pure reasoning task, second involves acombination of perception and reasoning tasks which need to be solved in a joint fashion.
Existingliterature proposes various approaches to handle one or both these settings.
One line of work proposespurely neural models [Palm et al., 2018, Nandwani et al., 2021, Dong et al., 2019] for solvingthese tasks representing the underlying constraints and costs implicitly.
While standard CNNs areused to solve the perceptual task, neural models such as Graph Neural Networks take care of thereasoning component.
In an alternate view, one may want to solve these tasks by explicitly learning the constraints and cost of the underlying ILP.
While the perceptual reasoning would still be handledby using modules such as CNN, reasoning is taken care of by having an explicit representation in theform of an ILP layer representing constraints and costs.
Such an approach would have the potentialadvantage of being more interpretable, and also being more accurate if the underlying constraintscould be learned in a precise manner.
Some recent approaches take this route, and include worksby [Paulus et al., 2021, Pogancic et al., 2020, Berthet et al., 2020].
We refer to the latter set ofapproaches as neural ILP architectures.
Learning in neural ILP architectures is complicated by the fact that there is a discrete optimization (inthe form of an ILP layer) at the end of the network, which is typically non-differentiable, making theend-to-end learning of the system difficult.
One of the possible ways is to instead use an iterative ILPsolving algorithm such as a cutting-plane method [Gomory, 2010] that uses a continuous relaxationin each iteration which is shown to be differentiable due to the introduction of continuous variables[Ferber et al., 2020, Wilder et al., 2019].
Most of these works are concerned with learning only thecost function and assume that the constraints are given.
Recent work by Paulus et al.
[2021] hasproposed an approach to directly pass the gradients through a black-box ILP solver.
Specifically, theyrely on Euclidean distances between the constraint hyperplanes and the current solution obtained bythe ILP solver to approximate the gradients.
Though this approach improves the quality of resultscompared to earlier works involving continuous relaxations, scalability gets severely affected sincean ILP has to be solved at every learning iteration, making the training extremely slow.
We are interested in answering the following question: is there a way to train a neural ILP architecturein an end-to-end manner, which does not require an access to an underlying solver?
Such an approach,if exists, could presumably result in significantly faster training times, resulting in scalability.
Inresponse, we propose a novel technique to differentiate through the learnable constraints as well as thelearnable cost function of an unknown Integer Linear Program (ILP).
During training, our techniquedoesn’t solve the ILP to compute the gradients.
Instead, we cast the learning of ILP constraints(and cost) as learning of a polyhedron, consisting of a set of hyperplanes, such that points inside thepolyhedron are treated as positive, and points outside as negative.
While a positive point needs to beclassified as positive by each of the hyperplanes, a negative point needs to be classified as negative byat least one of the hyperplanes.
Our formulation incorporates the learning of ILP cost also as learningof one of the hyperplanes in the system.
We formulate a novel margin-based loss to learn thesehyperplanes in a joint fashion.
A covariance based regularizer that minimizes the cosine similaritybetween all pairs of hyperplanes ensures that learned constraints (hyperplanes) are not redundant.
Since the training data comes only with positive examples, i.e., solutions to respective optimizationproblems, we develop several techniques for sampling the negatives, each of which is central to theeffective learning of our hyperplanes in our formulation.
We present several experiments on problems which require learning of ILP constraints and cost, withboth symbolic as well as perceptual input.
These include solving a sudoku puzzle given the image ofa partially filled board [Wang et al., 2019] (perceptual) and ILPs with random constraints (symbolic),
Knapsack from sentence description (perceptual), and key-point matching (perceptual) from [Pauluset al., 2021].
Our closest competitor, CombOptNet
[Paulus et al., 2021], can not solve even thesmallest of the sudoku boards sized 4 × 4 , whereas we can easily scale to 9 × 9 , getting close to100% accuracy.
We are comparable on key-point matching, and obtain significantly better accuracyon random ILPs.
We also outperform purely neural baselines (wherever applicable).
Related Works
In the first line of work, a neural model such as a Graph Neural Network (GNN) replaces a combinatorial solver altogether and implicitly encodes the rules or constraints in their weights, and learns themfrom data.
Nandwani et al.
[2022, 2021], Palm et al.
[2018] train a Recurrent Relational Networkfor learning to solve puzzles like sudoku, graph coloring, futoshiki etc; Dong et al.
[2019] propose
Neural Logic Machines (NLMs) that learn lifted first order rules and experiment with blocks world,reasoning on family trees etc; Selsam et al.
[2019], Amizadeh et al.
[2019a,b] train a GNN for solvingany CSP when constraints are explicitly given in a standard form such as CNF, DNF or Boolean
Circuits.
This is clearly different from us since we are interested in explicit learning the constraintsof the underlying optimization problem.
Inverse Optimization [Chan et al., 2021] aims to learn the constraints (cost) of a linear program (LP)from the observed optimal decisions.
Recently [Tan et al., 2019, 2020] use the notion of Parameterized
Linear Programs (PLP) in which both the cost and the constraints of an LP are parameterized byunknown weights and a feature vector.
[Gould et al., 2019] show how to differentiate throughcontinuous constrained optimization problem using the notion of a ‘ declarative node ’ that usesimplicit function theorem [Mingari Scarpello and Ritelli, 2002].
Similar to this are other works[Amos and Kolter, 2017, Agrawal et al., 2019] that define methods for differentiating through othercontinuous problems like convex quadratic programs (QPs) or cone programs.
While all thesetechniques are concerned with learning the constraints (cost) for optimization problems definedover continuous variables, our focus is on learning constraints (cost) for an ILP which involvesoptimization over discrete variables and can be a significantly harder problem.
In the space of learning cost for an ILP, several approaches have been proposed recently.
[Pogancicet al., 2020] define gradient w.r.t.
linear cost function that is optimized over a given discrete space.[Rolínek et al., 2020a,b] exploit it for the task of Deep Graph Matching, and for differentiatingthrough rank based metrics such as Average Precision/Recall respectively.
[Berthet et al., 2020]replace the black-box discrete solver with its smooth perturbed version during training and exploit‘perturb-and-MAP’
[Papandreou and Yuille, 2011] to compute the gradients.
On similar lines, [Niepertet al., 2021] also exploit perturb-and-MAP but propose a different noise model for perturbation andalso extends to the case when the output of the solver may feed into a downstream neural model.
These methods assume the constraints to be given, and only learn the cost for an ILP.
Other methods use relaxations of specific algorithms for the task of learning the constraints andcost for an ILP.
Ferber et al.
[2020] backprop through the KKT conditions of the LP relaxationcreated iteratively while using cutting-plane method [Gomory, 2010] for solving a MILP; Wilder et al.[2019] add a quadratic penalty term to the continuous relaxation and use implicit function theorem tobackprop through the KKT conditions, as done in Amos and Kolter [2017] for smooth programs;
Mandi and Guns [2020] instead add a log-barrier term to get the LP relaxation and differentiatethrough its homogeneous self-dual formulation linking it to the iterative steps in the interior pointmethod.
Wang et al.
[2019] uses a low rank SDP relaxation of MAXSAT and defines how thegradients flow during backpropagation.
Presumably, these approaches are limited in their applicationsince they rely on specific algorithmic relaxations.
Instead of working with a specific algorithm, some recent works differentiate through a black-boxcombinatorial solver for the task of learning constraints [Paulus et al., 2021].
The solver is called atevery learning step to compute the gradient.
This becomes a bottleneck during training, since theconstraints are being learned on the go, and the problem could become ill-defined resulting in a largersolver time, severely limiting scalability of such approaches.
In contrast, we would like to performthis task in a solver-free manner.
[Pan et al., 2020] propose learning of constraints for the task ofstructured prediction.
They represent the linear constraints compactly using a specific two layer
Rectifier Network [Pan and Srikumar, 2016].
A significant limitation is that their method can only beused for learning constraints, and not costs.
Further, their approach does not do well experimentallyon the benchmark domains that we compare with.
[Meng and Chang, 2021] propose a non-learningbased approach for mining constraints in which for each training data ( c , y ) , a new constraint isadded which essentially imply that no other target can have better cost than the given target y for thecorresponding cost c .
We are specifically interested in learning not only the constraints, but also thecost of an ILP from data.
Convex Polytope Machines (CPM)
[Kantchelian et al., 2014] learns a nonlinear binary classifier in the form of a polytope from a dataset of positive and negative examples.
In contrast, our goal is to learn the cost as well as constraints for an ILP where both the constraintsand the cost could be parameterized by another neural network.
Also, we do not have access to anynegative samples.
Differentiable ILP Loss
3.1 Background and Task Description
We are interested in learning how to solve combinatorial optimization problems that can be expressedas an Integer Linear Program (ILP) with equality as well as inequality constraints:
[Equation]
Here c ∈ R n represents an n dimensional cost vector, the matrix U ∈ R m 1 × n and v ∈ R m 1represent m 1 linear equality constraints, and the matrix G ∈
R m 2 × n and h ∈ R m 2 represent m 2linear inequality constraints, together defining the feasible region.
Without loss of generality, one can replace an equality constraint u Ti z
= v i by two inequalityconstraints: u Ti z ≥ v i and − u Ti z ≥ − v i .
Here u i and v i represent the i th row of the matrix
U andi th element of the vector v respectively.
Using this, one can reduce ILP in eq.
(1) to an equivalentform with just inequality constraints:
[Equation]
Here matrix A ∈ R m × n is row-wise concatenation of the matrices G , U and − U .
Vector b ∈ R mis a row-wise concatenation of vectors h , − v and v , and m = 2 m 1 + m 2 is the total numberof inequality constraints.
We represent the i th constraint
a Ti z
+ b i ≥ 0 , as [ a i | b i ] .
The integerconstraints z ∈ Z n make the problem NP hard.
Neural ILP Architecture: In a neural ILP architecture, the constraint matrix A , vector b , andthe cost vector c are neural functions f A , f b , and f c (of input x ) parameterized by learnable Θ =( θ
A , θ b , θ c ) , i.e. , A = f A ( x ; θ A ) ; b = f b ( x ; θ b ) ; c = f c ( x ; θ c ) .
This results in a different ILPfor each input x .
For an input x , the solution given by a neural ILP model, M Θ ( x ) , is nothing butthe optima of the following paramterized ILP where A , b , c are replaced by corresponding neuralfunctions f A , f b , f c evaluated at input x :
[Equation]
Example of visual-sudoku: In k × k visual-sudoku, the input x is an image of a sudoku puzzle andy ∗ ∈ { 0 , 1 } n is the corresponding solution, represented as a n = k 3 dimensional binary vector: theinteger in each of the k 2 cells is represented by a k dimensional one-hot binary vector.
Functionf c ( x ; θ c ) parameterizing the cost is nothing but a neural digit classifier that classifies the content ofeach of the k 2 cells into one of the k classes.
The neural functions f A and f b are independent of theinput x
as the constraints are the same for every k × k sudoku puzzle.
Therefore, f A ( x ; θ A ) =
θ A ,and f b ( x ; θ b ) =
θ b , where θ A is just a learnable matrix of dimension m × n and θ b is a learnablevector of dimension m .
See Bartlett et al.
[2008] for an ILP formulation of k × k sudoku with 4 k 2equality constraints in a n = k 3 dimensional binary space { 0 , 1 } n .
Learning Task:
Given a training dataset D = { ( x s , y ∗ s ) | s ∈ { 1 . . .
S }} with S training samples,the task is to learn the parameters Θ
=
( θ A , θ b , θ c ) , such that y ∗ s = M Θ ( x s ) for each s .
To do so,one needs to define derivatives ∂L∂ A , ∂L∂ b , and ∂L∂ c w.r.t.
A , b , and c respectively, of an appropriate lossfunction L .
Once such a derivative is defined, one can easily compute derivatives w.r.t.
θ A , θ b , andθ c using the chain rule: ∂L∂θ A =
∂L∂
A ∂f A ( x ; θ A )
∂θ A ; ∂L∂θ b = ∂L∂ b ∂f b ( x ; θ b ) ∂θ b ; and ∂L∂θ c = ∂L∂ c
∂f A ( x ; θ c )
∂θ c .
Hence,in the formulation below, we only worry about computing gradients w.r.t.
the constraint matrix A ,vector b , and the cost vector c .
The existing approaches, e.g. , [Paulus et al., 2021], explicitly need access to current model prediction
M Θ ( x ) This requires solving the ILP in eq.
(3), making the learning process extremely slow.
In contrast, we present a ‘solver-free’ framework for computation of an appropriate loss and itsderivatives w.r.t.
A , b , and c .
Our framework does not require solving any ILP while training, therebymaking it extremely scalable as compared to existing approaches.
3.2 A Solver-free Framework
Conversion to a constraint satisfaction problem: As a first step, we convert the constraint optimization problem in eq.
(2) to an equivalent constraint satisfaction problem by introducing an additional‘ cost–constraint ’:
c T z ≤ c T y ∗ , equivalent to a m +1 = − c and b m +
=
c T y ∗ .
Note that the above cost–constraint separates the solution y ∗ of the original ILP in eq.
(2) fromthe other feasible integral points.
This is because y ∗ must achieve the minimum objective valuec T y ∗ amongst all the feasible integral points and hence no other feasible integral point
z can obtainobjective value less than c T y ∗ .
The new constraint [ a m +1 | b m +1 ] together with the original mconstraints guarantee that y ∗ is the only solution satisfying all of the ( m + 1) constraints.
This results in the following equivalent linear constraint satisfaction problem:
[Equation]
Constructing such an equivalent satisfaction problem requires access to the solution y ∗ of the original
ILP in eq.
(2) and that is already available to us for the training data samples.
By rolling up the costvector c into an additional constraint with the help of y ∗ , we have converted our original objective oflearning both the cost and constraints to just learning of constraints in eq.
The main intuition behind our framework comes from the observation that each of the m linearconstraints defining the feasible region are essentially m linear binary classifiers (hyperplanes) in 196 R n separating the ground truth y ∗ from the infeasible region.
The additional cost–constraint ineq.
(4) separates y ∗ from other integral samples feasible w.r.t.
the original m constraints.
Learningconstraints of an ILP is akin to simultaneously learning m linear binary classifiers separating y ∗from other infeasible points of the original ILP along with learning a classifier for cost–constraint,separating y ∗ for other feasible integral points.
For a vector y ∗ to lie inside the feasible region, all the classifiers need to classify it positively, andhence it acts as a positive data point for all the m + 1 binary classifiers.
In the absence of explicitlyprovided negative samples, we propose a couple of strategies for sampling them from Z n for eachground truth y ∗ s .
We discuss them in detail in section 3.3.
For now, let N y ∗ be the set of all thenegative samples generated by all our sampling techniques for a ground truth positive sample y ∗ .
Incontrast to a positive point which needs to satisfy all the m + 1 constraints, a negative point becomesinfeasible even if it violates any one of the m + 1 constraints.
As a result, it would suffice if any oneof the m + 1 classifiers correctly classify it as a negative sample (refer to appendix for an illustration).
While learning, one cannot assign a negative sample to any specific classifier as their parametersare being updated continuously.
Instead, we make a soft assignment depending upon the Euclideandistance of the negative sample from the hyperplane.
With this intuition, we now formally define our
ILP–Loss.
Formulating solver-free ILP–Loss: Let d ( z ; [ a i | b i ])
=
a Ti z + b i | a i | represent the signed Euclideandistance of a point
z ∈ R n from the hyperplane corresponding to the constraint [ a i | b i ] .
We want thesigned distance from all the hyperplanes to be positive for the ground truth samples and negativefrom at least one hyperplane for all the sampled negative points.
We operationalize this via a marginbased loss function:
[Equation] [Equation]
We call L + , L − , and L o as the positive, negative and covariance loss respectively.
The average in L +ensures that a ground truth sample is positively classified by all the classifiers.
µ + and µ − are thehyperparameters representing the margins for the positive and the negative points respectively.
w i y − in
L − represents the soft assignment of y − to the i th constraint [ a i | b i ] and is computed by temperatureannealed softmax over the negative distances in eq.
Softmax ensures that the hyperplane which ismost confident to classify it as negative gets the maximum weight.
When y − lies inside the feasibleregion, then the most confident classifier is the one closest to y − .
To avoid the pathological behaviourof decreasing the loss by changing the weights, we ensure that gradients do not flow through w i y − ineq.
The temperature parameter τ needs to be annealed as the training progresses.
A high temperatureinitially can be seen as ‘ exploration ’ for the right constraint that will be violated by y − .
Thisis important as the constraints are also being learnt and are almost random initially, so a givennegative y − should not commit to a particular hyperplane.
Additionally, this encourages multipleconstraints to be violated for each negative, which leads to a robust set of constraints.
As thetraining progresses, we reduce the temperature τ , ensuring that the most confident classifier with leastsigned distance gets almost all the weight.
If y − is correctly classified as a negative with a marginµ − by any classifier i.e. , d ( y − ; [ a i | b i ]) ≤ − µ − for some i , then the corresponding negative loss,max { 0 , µ − + d ( y − ; [ a i | b i ]) } , becomes zero, and a low value of τ ensures that it gets all the weight.
The last term, L o acts as a regularizer and tries to ensure that no two learnt constraints are similar.
This is done by minimizing the cosine similarity between all the pairs of constraints.
The weightsλ pos , λ neg , and λ cov are computed automatically with a multi-loss weighing technique using coefficient of variations as described in Groenendijk et al.
Intuitively, the loss term with maximumvariance over the learning iterations, gets most of the weight.
Other details: See appendix for specific parameterization and initialization of A , b ; handling knownboundaries; and encoding equality constraints.
3.3 Negative Sampling
A meaningful computation of ILP–Loss in eq.
(5) depends crucially on the negative samples N ∗ y .
Randomly sampling negatives from Z n is one plausible strategy, but it may not be efficient from thelearning perspective: any point which is far away from any of the classifiers will easily be classifiedas negative and will not contribute much to the loss function.
In response, we propose multiplealternatives for sampling the negative points:
Integral k-hop neighbours: To generate negatives, we sample the integral neighbours that are at aHamming Distance of k from y ∗ .
For small k , these form the hardest negatives as they are closestto the positive point.
Note that it is possible for a few integral neighbours to be feasible w.r.t.
the mconstraints, but they must have worse cost than the given ground truth y ∗ .
Such samples contributetowards learning the cost parameters θ c .2.
Project and sample: In this method, we project the ground truth y ∗ on each of the m hyperplanesand then randomly sample a neighbour of the projection, generating a total of m negatives.
Samplingprobability in the j th dimension depends on the j th coordinate of the projection: if value of thej th coordinate is r / ∈ Z , then we sample floor ( r ) and ceil ( r ) with probability ceil ( r ) − r andr − floor ( r ) respectively.
If r ∈ Z , then we sample r with probability 1 .
Projection samples areclose to the boundary of the currently learnt polyhedron, thus taking the training progress into account.
Further, each hyperplane is likely to be assigned to a close-by negative due to projection sampling.3.
Batch negatives: In this strategy, we consider every other ground truth y ∗ s ′ s ′ ̸
= s in the minibatchas a potential negative sample for y ∗ s .
We call it as a ‘batch–negative’.
This is particularly useful forlearning the cost parameters θ c when the learnable constraints are the same for all the ground truthsamples, such as in sudoku.
In such cases, a batch–negative y ∗ s ′ being a feasible point of the original
ILP formulation, must always satisfy all of the m learnable constraints of the original ILP in eq.
Hence, the only way for y ∗ s ′ to be correctly classified as a negative for y ∗ s is by violating the costconstraint in eq.
(4), Learning cost parameters θ c that result in violation of the cost constraint forevery batch–negative helps in ensuring that the ground truth y ∗ s indeed has the minimum cost.
Experiments
The goal of our experiments is to evaluate the effectiveness and scalability of our proposed approachcompared to the SOTA black-box solver based approach, i.e., CombOptNet
[Paulus et al., 2021].
We experiment on 4 datasets: visual sudoku [Wang et al., 2019], and three datasets from [Pauluset al., 2021]: random constraints, knapsack from sentence description and key-point matching.
Wealso compare with an appropriately designed neural baseline for each of our datasets.
To measurescalability, in each of the domains, we compare performance of each of the algorithms in terms oftime taken, as well as accuracy obtained, for varying problem sizes.
[-Gold 
 CPU.-] For each of the algorithms, wekept a maximum time limit of 12 hours for training.
We next describe details of each of these datasets,appropriate baselines, and our results.
See the appendix for the details of the ILP solver used in ourexperiments, the hardware specifications, the hyper-parameters, and various other design choices.
4.1 Visual Sudoku
This task involves learning the rules of the sudoku puzzle.
The input x is an image of a sudoku puzzlewhere a digit is replaced by a random MNIST image of the same class.
A k × k sudoku puzzle canbe viewed as an ILP with 4 k 2 equality constraints over k 3 binary variables [Bartlett et al., 2008].
Each of the digit images in the k 2 cells are decoded into a k dimensional real vector, resulting ina k 3 dimensional cost c ∈ R n where n = k 3 .
The k 3 dimensional binary output vector (solution)y ∗ is created by representing each of the k 2 cells of the puzzle by a k dimensional binary vectorwhich takes a value of 1 at i th dimension if and only if the cell is filled by digit i .
Here, the costvector needs to be inferred from the input image.
Our objective is to learn the constraint matrix
A = f A ( x ; θ A ) =
θ A and vector b = f b ( x ; θ b ) = θ b , representing the linear constraints of sudoku.
The cost vector c = f c ( x ; θ c ) is a function of the input image x , parameterized by θ c which needs tobe learned.
Dataset: We experiment with 3 different datasets for k = 4 , 6 , and 9 .
We first build symbolic sudokupuzzles with k digits, and use MNIST to convert a puzzle into an image.
For 9 × 9 sudokus, we use astandard dataset from Kaggle [Park, 2017], for k =
we use publically available data from [Arcotand Kalluraya, 2019], and for k =
we use the data generation process described in [Nandwaniet al., 2022].
We randomly select 10 , 000 samples for training, 1000 samples for validation, and 500samples for testing for each k .
To generate the input images for visual-sudoku, we use the officialtrain and test split of MNIST[Deng, 2012].
Digits in our train and test splits are randomly replacedby images in the MNIST train and test sets respectively, avoiding any leakage.
Baselines: Our neural baseline is Recurrent Relational Networks (RRN)
[Palm et al., 2018]: a purelyneural approach for reasoning based on Graph Neural Networks.
We use the default set of parametersprovided in their paper for training RRNs.
We note that the RRN baseline uses additional informationin the form of graph structure which is not available to our solver.
We make this comparison to seehow well we perform compared to one of the SOTA techniques for this problem.
We also compareagainst another neuro-symbolic architecture that uses SATNet [Wang et al., 2019] as the reasoninglayer. The neural component in all the neuro-symbolic methods is a CNN which decodes images intodigits.
[Table]
Results: Table 1 compares the performance and training time of ourmethod against the different baselinesdescribed above.
CompOptNet failsmiserably on this problem, not beingable to complete training for any ofthe sizes in the stipulated time.
Thepurely neural model gives a decentperformance and is competitive withours for smaller board sizes 2 .
But forlarger board size of 9 × 9 , we beat RRN based model by a significant margin of about 25 points.
While SATNet performs comparableon smaller 4 × 4 sudoku puzzles, its performance degrades drastically to 17.8% for 9 × 9 board size.
See appendix for a comparison on the dataset used in Wang et al.
4.2 Random constraints
This is the synthetic dataset borrowed from Paulus et al..
The training data D = { ( c s , y ∗ s ) | s ∈{ 1 · · · S }} is created by first generating a random polyhedron P ⊆ Y by sampling m ′ hyperplanes { [ a ′ i |
b ′ i ] | a ′ i , b ′ i ∈ R n , i ∈ { 1 · · · m ′ }} in an n dimensional bounded continuous space
Y .
A cost vector c s ∈ R n is then randomly sampled and corresponding y ∗ s is obtained asy ∗ s = arg min
z ∈P c Ts z , z ∈ Z n .
Objective is to learn a constraint matrix A and vector b suchthat for a ground truth ( c , y ∗ ) pair, y ∗ =
arg min z ∈Y c T z , z ∈ Z n , and Az + b ≥ 0 .
Dataset: Restricting Y to [0 , 1] n and [ − 5 , 5] n results in two variations, referred as ‘binary’ and‘dense’ settings, respectively.
For both of the output spaces, we experiment with four different settingsin n = 16 dimensional space by varying the number of ground truth constraints as m ′
= 1 , 2 , 4 , and inference
[Table] For each m , we experiment with all the 10 datasets provided by Paulus et al overthe 10 models.
The training data in each case consists of 1600 ( c , y pairs model performance istested for 1000 cost vectors.
Baseline:
Here we compare only against CombOptNet.
A neural baseline (an MLP) is shown toperform badly for this symbolic problem in Paulus et al.. Hence we exclude this in our experiments.
Results: Table 2 presents the comparison of the two algorithms in terms of accuracy as well as timetaken for training.
While we perform marginally better than CombOptNet in terms of training time,our performance is significantly better, on almost all problems in the dense setting, and the largerproblems in the binary setting.
We are more than 35 accuracy points better compared to CombOptNeton the largest problem instance with 8 constraints in both settings.
This result demonstrates that ourapproach is not only faster in terms of training time, but also results in better solutions even when thebaseline converges to a solution.
This validates the effectiveness of our framework.
4.3 Knapsack from Sentence Descriptions
[Table] [Table]
This task, also borrowed
[Table]
from Paulus et al., is basedon the classical NP-hard
Knapsack problem. Eachinput consists of N sentences describing the priceand weight of N items andthe objective is to select asubset of items that maximizes their total price whilekeeping their total weight less than the knapsack’s capacity C max p T z s w z ≤ C } N , where p , w ∈ R N are the price and weight vectors, respectively. sentence has beenconverted into a d = 4096 dimensional dense embedding
Conneau et al
[2017], so that eachinput is N × d dimensional vector x . corresponding output is y ∗ ∈ {
} N
The knapsackcapacity C is fixed for all instances in a dataset.
task is to learn the parameters θ c θ A and θ b ofa neural network that extracts the cost and the constraints from x .
Note that here both the cost andthe constraints need to be inferred from the input.
Dataset and Methodology:
The original dataset in Paulus et al. has 4500 train and 500 test instanceswith fixed N = 10 items, extracted from a corpus containing 50 , 000 sentences and their embeddings.
In addition to experimenting with the original dataset, to demonstrate the scalability of our method,we also bootstrap new datasets with N = 15 , 20 , 25 and 30 , by randomly selecting sentences fromthe original corpus.
Each bootstrapped dataset also has 4500 train and 500 test instances.
Following
Paulus et al.
, we use an MLP to extract c and A from N d -dimensional sentence embeddings, andkeep b = 1 .
We keep number of constraints as m = 4 , the setting which achieves best performancein Paulus et al..
Results: Table 3 presents a comparison between the training time and accuracy of our method against
CombOptNet.
We significantly outperform the baseline, both in terms of accuracy and train-timesacross all the datasets.
Note that our relative gain increases significantly with increasing problem size.
CombOptNet fails to complete even a single epoch in the stipulated time and results in 0% accuracyon the largest problem.
4.4 Keypoint Matching
Here we experiment on a real world task of matching key points between two images of differentobjects of same type Rolínek et al.
[2020b].
The input x consists of a pair of images ( I 1 , I 2 ) of sametype along with a shuffled list of coordinates (pixels) of k keypoints in both images.
The task is tomatch the same keypoints in the two images with each other.
The ground truth constraint enforces abijection between the keypoints in the two images.
The output y is represented as a k × k binarypermutation matrix, i.e. , entries in a row or column should sum to 1 .
The cost c is a k × k sizedvector parameterized by θ c .
The goal is to learn A , b and θ c .
Dataset: We experiment with image pairs in SPair-17k [Min et al., 2019] dataset used for the taskof keypoint matching in [Rolínek et al., 2020b].
Since the neural ILP models can deal with a fixedsize output space, we artificially create 4 datasets for 4,5,6 and 7 keypoints from the original dataset.
While generating samples for k keypoints dataset, we randomly sample any k annotated pairs frominput image pairs that have more than k keypoints annotated.
See appendix for details on dataset size.
Baselines: We compare against a strong neural baseline which is same as the backbone modelparameterizing the k 2 dimensional cost vector in [Rolínek et al., 2020b].
It is trained by minimizing
BCE loss between negative learnt cost ( − c ) and target y ∗ .
We create an additional baseline by doingconstrained inference with ground truth constraints and learnt cost (‘Neural + CI’) in table 4.
datasets with varying number of keypoints.
Neural+CI is the ILP inference with the known constraints over the cost learnt by neural model.
[Table]
Results: Table 4 presents the percentage of keypoints matched correctly by different models.
Both CombOptNet and our method achieve similar performance.
Note that we obtain a gain of2 points over the purely neural baseline without the knowledge of constraints.
This is almost same as the gain obtained by constrainedinference with known constraints on top of neural model.
For this task, we did not observeany gain in the training time over CombOptNet.
This is expected as the bottleneck here is thebackbone neural model parameterizing the costfunction.
Even though our method spends lesstime in each iteration, it takes more number of iterations to train, balancing out the gain obtainedby speeding up each iteration.
Conclusion and Future Work
We have presented a solver–free framework for scalable training of a neural ILP architecture.
Ourmethod learns the linear constraints by viewing them as linear binary classifiers, separating thepositive points (inside the polyhedron) from the negative points (outside the polyhedron).
Whilegiven ground truth acts as positives, we propose multiple strategies for sampling negatives.
A simpletrick using the available ground truth outputs in the training data, converts the cost vector into aconstraint, enabling us to learn the cost vector and constraints in a similar fashion.
Future work involves extending our method to neural-ILP-neural architectures, i.e., where the outputof ILP is an input to a downstream neural model (see appendix for a detailed discussion).
Second, aneural ILP model works with a fixed dimensional output space, even though the constraints for thesame underlying problem are the same in first order logic, e.g. , constraints for k × k sudoku puzzlesremain the same in first order irrespective of k .
Creating neural ILP models that can parameterizethe constraints on the basis of the size of the input (or learnt) cost vector is a potential direction forfuture work.
Lastly, the constraints learnt in a neural ILP architecture might not be interpretable evenif the ground truth constraints are.
In future, we would like to develop methods that distill the learntconstraints into a human-interpretable form.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] .
In Conclusion and future works
section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We will makethe code and data available on acceptance
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Details are in experiments section 4 under ‘Training Methodology’and in the appendix.(c)
Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes]
In one of the experiments on random constraints.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In the experiments section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Datasets are cited inthe respective experiment sections. (b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
Appendix 3 Differentiable ILP Loss [Figure] 3.2 A Solver-free Framework
[Figure] In fig. 1, we consider a 2-dimensional ILP with 4 ground truth constraints ˜ a i T z + ˜ b i ≥ 0 , i = 1 , 2 , 3 , 4 ,a cost vector ˜ c , and target solution y ∗ .
The dots represent points in Z 2 .
The solid lines representhyperplanes (here lines) corresponding to the constraints, and the dashed line represents our costconstraint.
The shaded area in fig.
1a is the feasible region of the constraint satisfaction problem:arg min z 0 T z subject to ˜ Az + ˜ b ≥ 0 ; ˜ c T
[Equation]
Figure 1a shows the ground truth ILP.
The signs of ˜ a i ’s and ˜ b
i ’s are such that the closed polyhedron(here polygon) containing the blue points forms the feasible region ˜ Az + ˜ b ≥ 0 .
The green point isoptimal w.r.t.
the cost ˜ c .
The red points are infeasible, i.e. , ˜ Az + ˜ b < 0 for red points.
Note that thetarget solution y ∗ (point with green border) is the only integral point in the shaded region.
The otherblue points violate the cost constraint ˜ c T z ≤ ˜ c T y ∗ , and the red points violate at least one of the fourfeasibility constraints ˜
a i T z + ˜ b
i ≥ 0 , i ∈ { 1 · · · 4 } .
Figure 1b shows a possible situation during learning.
For simplicity, consider temperature close tozero ( τ ≈ 0 ), so that only the closest hyperplane contributes to the negative loss for a negative sample.
Also consider margins close to zero ( µ + ≈ 0 , µ − ≈ 0 ).
In fig.
1b, the point with the green borderis outside the shaded region, whereas one red point is inside.
The ground truth y ∗ is on the wrongside of only the fourth hyperplane, i.e. , a T 4 y ∗ + b 4 < 0 , and hence it is the only contributor to thepositive loss, i.e. , L + = − 14 d ( y ∗ ; [ a 4 | b 4 ]) , where d ( y ∗ ; [ a 4 | b 4 ])
=
a T 4 y ∗ + b 4 | a 4 | < 0 .
The red negativepoint denoted as y − (inside the shaded region) being closest to the third hyperplane contributesd ( y − ; [ a 3 | b 3 ]) =
a T 3 y − + b 3 | a 3 | > 0 to the negative loss L − .
The green and red dotted arrows indicatethe directions of movement of the constraint hyperplanes on weight update.
3.2 A Solver-free Framework
Other details:
Parameterization of equality constraints: Recall that we replace an equality u Ti z = v i by twoinequality constraints: v i ≤ u Ti z ≤ v i .
In practice, to enhance learnability, we add a small margin ofϵ on both sides: v i − ϵ ≤ u Ti z ≤ v i + ϵ .
We pick an ϵ small enough so that the probability of thenew feasible region to include an infeasible integral point is negligible.
Known boundaries: In many cases, the boundary conditions on the output variables are known, i.e. ,z ∈ Y =
[ l, u ] n , where l and u are the lower and upper bounds on each dimension of z .
We handlethis by adding known boundary constraints: l ≤ z i ≤ u, ∀ i ∈ { 1 · · · n } .
Over-parameterization of constraints: As done in Paulus et al.
, we also over-parameterize eachconstraint hyperplane by an additional learnable offset vector o i which can be viewed as its own localorigin.
Radius r i represents its distance from its own origin
o i , resulting in the following hyperplanein the base coordinate system: a Ti z
+ r i − o
Ti a i / | a i | ≥ 0 .
Initialization of A : The way we initialize the constraints may have an impact on the learnability.
While [Paulus et al., 2021] propose to sample each entry of a i uniformly (and independently) between[ − 0 .
5] , we also experiment with a standard Gaussian initialization.
The latter results in initialhyperplanes with their normal directions ( a i ’s) uniformly sampled from a unit hyper-sphere.
Inexpectation, such initialization achieves minima of L o that measures total pairwise covariance.
4 Experiments
Details of the ILP Solver and the hardware used for experiments: To solve the learnt ILPs, weuse Gurobi ILP solver [Gurobi Optimization, LLC, 2022] available under ‘Named-user academiclicense’.
All our experiments were run on 11 GB ‘GeForce GTX 1080 Ti’ GPUs installed on amachine with 2.60GHz Intel(R)
Xeon(R)
Gold 6142 CPU.
For each of the algorithms, we kept amaximum time limit of 12 hours for training.
4.1 Visual Sudoku
Hyperparameters and other design choices: (a) # of learnable constraints:
For k = 4 and 6 , wekeep m = ( n + 1) as the number of learnable constraints where n = k 3 is the number of binaryvariables.
For k = 9 , we learn m = ( n + 1) / 2 equality constraints, which is equivalent to learning( n + 1) inequality constraints with parameter tying.
(b) Margin: We find that a margin of 0 .
01 workswell across domains and problem sizes.
(c) Temperature: we start with a temperature τ = 1 andanneal it by a factor of 0 .
1 whenever the performance on a their use report it
Hence we trained both SATNet and our model on the dataset available on SATNet’s Github repo.
[Table]
Interestingly, on their dataset which we believe to be easier (as shown by performance numbers), ourrun of SATNet achieves 71.0% board accuracy whereas our method achieves 98.3%.
4.2 Random Constraints
[Table]
Hyperparameters and other design choices: we keep number of learnable constraints as twice thenumber of ground truth constraints, i.e. , m = 2 m ′ , as Paulus et al.
report best performance in most ofthe settings with it.
Here we initialize a i uniformly between [ − 0 .
Rest of the hyperparamtersare set as in the case of sudoku.
4.4 Keypoint Matching
[Table]
Hyperparameters:
For each k , the number of learnable constraints is set to 2 k : same as the numberof ground truth constraints.
In our method, for keypoints 5,6 and 7, we initialize the backbone costparameters θ c with the one obtained by training it on 4 keypoints.
We start with a temperature τ = 0 . 5and anneal it by a factor of 0 . 2 at 10 th and 30 th epoch. Here we initialize a i uniformly between[ − 0 .
Rest of the hyperparmeters are same as those used for sudoku.
Dataset details: See table 5 for the number of train and test samples in the four datasets created for4 , 5 , 6 , and 7 keypoints.
5 Future Work
Discussion on training Neural-ILP-Neural architectures:
In the current formulation, availabilityof the solution to the ground truth ILP is important for our solver-free approach to work.
Specifically,it is required to: 1.) convert the constrained optimization problem to a constraint satisfaction problemby including the cost-constraint eq.
(4), and 2.) to calculate the positive loss eq.
However, ina Neural-ILP-Neural architecture, the intermediate supervision for only the Neural-ILP part ( i.e. ,solution of the ground truth ILP) is not available.
On the other hand, solver based methods such as CombOptNet, do not require access to the solutionof the ground-truth ILP.
Instead, they rely on the solution of the current intermediate ILP (duringlearning) to compute the gradients of the loss and
thus their approach is not solver free.
We note thateven though in princple they can train Neural-ILP-Neural architectures, their experiments are only inthe Neural-ILP settings.
Extending our current work for Neural-ILP-Neural architectures is an important direction of futurework.
One plausible approach could be to train an auxiliary inverse network that converts a givenoutput of Neural-ILP-Neural architecture to a predicted symbolic target of Neural-ILP component.
This predicted target can be used as a proxy to the ground truth solution of the Neural-ILP part.
Similar ideas of using an inverse network have been explored in [Agarwal et al., 2021], albeit underthe setting where ILP is known and only the neural encoder and decoder needs to be learnt.