A Solver-free Framework for Scalable Learning in Neural ILP Architectures
AffiliationAddress email
Abstract
There is a recent focus on designing architectures that have an Integer Linear
Programming (ILP) layer within a neural model (referred to as Neural ILP in thispaper).
Neural ILP architectures are suitable for pure reasoning tasks that requiredata-driven constraint learning or for tasks requiring both perception (neural) andreasoning (ILP).
A recent SOTA approach for end-to-end training of Neural ILPexplicitly defines gradients through the ILP black box [Paulus et al., 2021] – thistrains extremely slowly, owing to a call to the underlying ILP solver for everytraining data point in a minibatch.
In response, we present an alternative trainingstrategy that is solver-free , i.e., does not call the ILP solver at all at training time.
Neural ILP has a set of trainable hyperplanes (for cost and constraints in ILP),together representing a polyhedron.
Our key idea is that the training loss shouldimpose that the final polyhedron separates the positives (all constraints satisfied)from the negatives (at least one violated constraint or a suboptimal cost value),via a soft-margin formulation.
While positive example(s) are provided as partof the training data, we devise novel techniques for generating negative samples.
Our solution is flexible enough to handle equality as well as inequality constraints.
Experiments on several problems, both perceptual as well as symbolic, whichrequire learning the constraints of an ILP, show that our approach has superiorperformance and scales much better compared to purely neural baselines and otherstate-of-the-art models that require solver-based training.
In particular, we are ableto obtain excellent performance in 9 × 9 Visual Sudoku, to which the other Neural
ILP solver is not able to scale.
Introduction
There has been a growing interest in the community which focuses on developing neural modelsfor solving combinatorial optimization problems.
These problems often require complex reasoningover discrete symbols.
Many of these problems can be expressed in the form an underlying Integer
Linear Program (ILP).
Two different kinds of problem (input) settings have been considered in theliterature: (a) purely symbolic and (b) combination of perceptual and symbolic.
Solving an n-Queensproblem given the partial assignment of queens on the board as input would be an example of theformer, and solving a sudoku puzzle given the image of a partially filled board as input would bean example of latter.
While the first setting corresponds to a pure reasoning task, second involves acombination of perception and reasoning tasks which need to be solved in a joint fashion.
Existingliterature proposes various approaches to handle one or both these settings.
One line of work proposespurely neural models [Palm et al., 2018, Nandwani et al., 2021, Dong et al., 2019] for solvingthese tasks representing the underlying constraints and costs implicitly.
While standard CNNs areused to solve the perceptual task, neural models such as Graph Neural Networks take care of thereasoning component.
In an alternate view, one may want to solve these tasks by explicitly learning the constraints and cost of the underlying ILP.
While the perceptual reasoning would still be handledby using modules such as CNN, reasoning is taken care of by having an explicit representation in theform of an ILP layer representing constraints and costs.
Such an approach would have the potentialadvantage of being more interpretable, and also being more accurate if the underlying constraintscould be learned in a precise manner.
Some recent approaches take this route, and include worksby [Paulus et al., 2021, Pogancic et al., 2020, Berthet et al., 2020].
We refer to the latter set ofapproaches as neural ILP based architectures.
Learning in neural ILP architectures is complicated by the fact that there is a discrete optimization (inthe form of an ILP layer) at the end of the network, which is typically non-differentiable, making theend-to-end learning of the system difficult.
One of the possible ways is to instead use an iterative ILPsolving algorithm such as a cutting-plane method [Gomory, 2010] that uses a continuous relaxationin each iteration which is shown to be differentiable due to the introduction of continuous variables[Ferber et al., 2020, Wilder et al., 2019].
Most of these works are concerned with learning only thecost function and assume that the constraints are given.
Recent work by Paulus et al.
[2021] hasproposed an approach to directly pass the gradients through a black-box ILP solver.
Specifically, theyrely on Euclidean distances between the constraint hyperplanes and the current solution obtained bythe ILP solver to approximate the gradients.
Though this approach improves the quality of resultscompared to earlier works involving continuous relaxations, scalability gets severely affected sincean ILP has to be solved at every learning iteration, making the training extremely slow.
We are interested in answering the following question: is there a way to train a neural ILP architecturein an end-to-end manner, which does not require an access to an underlying solver?
Such an approach,if exists, could presumably result in significantly faster training times, resulting in scalability.
Inresponse, we propose a novel technique to differentiate through the learnable constraints as well as thelearnable cost function of an unknown Integer Linear Program (ILP).
During training, our techniquedoesn’t solve the ILP to compute the gradients.
Instead, we cast the learning of ILP constraints(and cost) as learning of a polyhedron, consisting of a set of hyperplanes, such that points insidethe polyhedron are treated as positive, and points outside as negative.
While a positive point needsto be classified as positive by each of the hyperplanes, a negative point needs to be classified asnegative by at least one of the hyperplanes.
Our formulation incorporates the learning of ILP costalso as learning of one of the hyperplanes in the system.
We formulate a novel margin-based loss tolearn these hyperplanes in a joint fashion.
A covariance based regularizer that minimizes the cosinesimilarity between all pairs of hyperplanes ensures that learned constraints (hyperplanes) are notredundant.
Since the training data comes only with positive examples, i.e., solutions to respectiveoptimization problems, we develop several techniques for sampling the negatives: (a) k-hop distanceintegral points (b) closest integral point after projecting on a hyperplane (c) integral points which aresolutions to other optimization problems in the batch. Each of these classes of negatives is central tothe effective learning of our hyperplanes in our formulation.
We present several experiments on problems which require learning of ILP constraints and cost,with both symbolic as well as perceptual input.
These include solving a sudoku puzzle given theimage of a partially filled board [Wang et al., 2019] (perceptual), ILPs with random constraints(symbolic)
[Paulus et al., 2021], and key-point matching [Paulus et al., 2021] (perceptual).
Ourclosest competitor, CombOptNet
[Paulus et al., 2021], can not solve even the smallest of the sudokuboards sized 4 × 4 , whereas we can easily scale to 9 × 9 , getting close to 100% accuracy.
We arecomparable on key-point matching, and obtain significantly better accuracy on random ILPs.
We alsooutperform purely neural baselines (wherever applicable).
Related Works
In the first line of work, a neural model such as a Graph Neural Network (GNN) replaces a combinatorial solver altogether and implicitly encodes the rules or constraints in their weights, and learns themfrom data.
Nandwani et al.
[2022, 2021], Palm et al.
[2018] train a Recurrent Relational Networkfor learning to solve puzzles like sudoku, graph coloring, futoshiki etc; Dong et al.
[2019] propose
Neural Logic Machines (NLMs) that learn lifted first order rules and experiment with blocks world,reasoning on family trees etc; Selsam et al.
[2019], Amizadeh et al.
[2019a,b] train a GNN for solvingany CSP when constraints are explicitly given in a standard form such as CNF, DNF or Boolean
Circuits.
This is clearly different from us since we are interested in explicit learning the constraintsof the underlying optimization problem.
Inverse Optimization [Chan et al., 2021] aims to learn the constraints (cost) of a linear program (LP)from the observed optimal decisions.
Recently [Tan et al., 2019, 2020] use the notion of Parameterized
Linear Programs (PLP) in which both the cost and the constraints of an LP are parameterized byunknown weights and a feature vector.
[Gould et al., 2019] show how to differentiate throughcontinuous constrained optimization problem using the notion of a ‘ declarative node ’ that usesimplicit function theorem [Mingari Scarpello and Ritelli, 2002].
Similar to this are other works[Amos and Kolter, 2017, Agrawal et al., 2019] that define methods for differentiating through othercontinuous problems like convex quadratic programs (QPs) or cone programs.
While all thesetechniques are concerned with learning the constraints (cost) for optimization problems definedover continuous variables, our focus is on learning constraints (cost) for an ILP which involvesoptimization over discrete variables and can be a significantly harder problem.
In the space of learning cost for an ILP, several approaches have been proposed recently.
[Pogancicet al., 2020] define gradient w.r.t.
linear cost function that is optimized over a given discrete space.[Rolínek et al., 2020a,b] exploit it for the task of Deep Graph Matching, and for differentiatingthrough rank based metrics such as Average Precision/Recall respectively.
[Berthet et al., 2020]replace the black-box discrete solver with its smooth perturbed version during training and exploit‘perturb-and-MAP’
[Papandreou and Yuille, 2011] to compute the gradients.
On similar lines, [Niepertet al., 2021] also exploit perturb-and-MAP but propose a different noise model for perturbation andalso extends to the case when the output of the solver may feed into a downstream neural model.
These methods assume the constraints to be given, and only learn the cost for an ILP.
Other methods use relaxations of specific algorithms for the task of learning the constraints andcost for an ILP.
Ferber et al.
[2020] backprop through the KKT conditions of the LP relaxationcreated iteratively while using cutting-plane method [Gomory, 2010] for solving a MILP; Wilder et al.[2019] add a quadratic penalty term to the continuous relaxation and use implicit function theorem tobackprop through the KKT conditions, as done in Amos and Kolter [2017] for smooth programs;
Mandi and Guns [2020] instead add a log-barrier term to get the LP relaxation and differentiatethrough its homogeneous self-dual formulation linking it to the iterative steps in the interior pointmethod.
Wang et al.
[2019] uses a low rank SDP relaxation of MAXSAT and defines how thegradients flow during backpropagation.
Presumably, these approaches are limited in their applicationsince they rely on specific algorithmic relaxations.
Instead of working with a specific algorithm, some recent works differentiate through a black-boxcombinatorial solver for the task of learning constraints [Paulus et al., 2021].
The solver is called atevery learning step to compute the gradient.
This becomes a bottleneck during training, since theconstraints are being learned on the go, and the problem could become ill-defined resulting in a largersolver time, severely limiting scalability of such approaches.
In contrast, we would like to performthis task in a solver-free manner.
[Pan et al., 2020] propose learning of constraints for the task ofstructured prediction.
They represent the linear constraints compactly using a specific two layer
Rectifier Network [Pan and Srikumar, 2016].
A significant limitation is that their method can only beused for learning constraints, and not costs.
Further, their approach does not do well experimentallyon the benchmark domains that we compare with.
[Meng and Chang, 2021] propose a non-learningbased approach for mining constraints in which for each training data ( c , y ) , a new constraint isadded which essentially imply that no other target can have better cost than the given target y for thecorresponding cost c .
We are specifically interested in learning not only the constraints, but also thecost of an ILP from data.
Convex Polytope Machines (CPM)
[Kantchelian et al., 2014] learns a nonlinear binary classifier in the form of a polytope from a dataset of positive and negative examples.
In contrast, our goal is to learn the cost as well as constraints for an ILP where both the constraintsand the cost could be parameterized by another neural network.
Also, we do not have access to anynegative samples.
Differentiable ILP Loss
3.1 Background and Task Description
We are interested in learning how to solve combinatorial optimization problems that can be expressedas an Integer Linear Program (ILP) with equality as well as inequality constraints:
[Equation]
Here c ∈ R n represents an n dimensional cost vector, the matrix U ∈ R m 1 × n and v ∈ R m 1represent m 1 linear equality constraints, and the matrix G ∈
R m 2 × n and h ∈ R m 2 represent m 2linear inequality constraints, together defining the feasible region.
Without loss of generality, one can replace an equality constraint u Ti z
= v i by two inequalityconstraints: u Ti z ≥ v i and − u Ti z ≥ − v i .
Here u i and v i represent the i th row of the matrix
U andi th element of the vector v respectively.
Using this, one can reduce ILP in eq.
(1) to an equivalentform with just inequality constraints:
[Equation]
Here matrix A ∈ R m × n is row-wise concatenation of the matrices G , U and − U .
Vector b ∈ R mis a row-wise concatenation of vectors h , − v and v , and m = 2 m 1 + m 2 is the total numberof inequality constraints.
We represent the i th constraint
a Ti z
+ b i ≥ 0 , as [ a i | b i ] .
The integerconstraints z ∈ Z n make the problem NP hard.
Neural ILP Architecture: In a neural ILP architecture, the constraint matrix A , vector b , andthe cost vector c are neural functions f A , f b , and f c (of input x ) parameterized by learnable Θ =( θ
A , θ b , θ c ) , i.e. , A = f A ( x ; θ A ) ; b = f b ( x ; θ b ) ; c = f c ( x ; θ c ) .
This results in a different ILPfor each input x .
For an input x , the solution given by a neural ILP model, M Θ ( x ) , is nothing butthe optima of the following paramterized ILP where A , b , c are replaced by corresponding neuralfunctions f A , f b , f c evaluated at input x :
[Equation]
Example of visual-sudoku: In k × k visual-sudoku, the input x is an image of a sudoku puzzle andy ∗ ∈ { 0 , 1 } n is the corresponding solution, represented as a n = k 3 dimensional binary vector: theinteger in each of the k 2 cells is represented by a k dimensional one-hot binary vector.
Functionf c ( x ; θ c ) parameterizing the cost is nothing but a neural digit classifier that classifies the content ofeach of the k 2 cells into one of the k classes.
The neural functions f A and f b are independent of theinput x
as the constraints are the same for every k × k sudoku puzzle.
Therefore, f A ( x ; θ A ) =
θ A ,and f b ( x ; θ b ) =
θ b , where θ A is just a learnable matrix of dimension m × n and θ b is a learnablevector of dimension m .
See Bartlett et al.
[2008] for an ILP formulation of k × k sudoku with 4 k 2equality constraints in a n = k 3 dimensional binary space { 0 , 1 } n .
Learning Task:
Given a training dataset D = { ( x s , y ∗ s ) | s ∈ { 1 . . .
S }} with S training samples,the task is to learn the parameters Θ
=
( θ A , θ b , θ c ) , such that y ∗ s = M Θ ( x s ) for each s .
To do so,one needs to define derivatives ∂L∂ A , ∂L∂ b , and ∂L∂ c w.r.t.
A , b , and c respectively, of an appropriate lossfunction L .
Once such a derivative is defined, one can easily compute derivatives w.r.t.
θ A , θ b , andθ c using the chain rule: ∂L∂θ A =
∂L∂
A ∂f A ( x ; θ A )
∂θ A ; ∂L∂θ b = ∂L∂ b ∂f b ( x ; θ b ) ∂θ b ; and ∂L∂θ c = ∂L∂ c
∂f A ( x ; θ c )
∂θ c .
Hence,in the formulation below, we only worry about computing gradients w.r.t.
the constraint matrix A ,vector b , and the cost vector c .
The existing approaches, e.g. , [Paulus et al., 2021], explicitly need access to current model prediction
M Θ ( x ) This requires solving the ILP in eq.
(3), making the learning process extremely slow.
In contrast, we present a ‘solver-free’ framework for computation of an appropriate loss and itsderivatives w.r.t.
A , b , and c .
Our framework does not require solving any ILP while training, therebymaking it extremely scalable as compared to existing approaches.
3.2 A Solver-free Framework
Conversion to a constraint satisfaction problem: As a first step, we convert the constraint optimization problem in eq.
(2) to an equivalent constraint satisfaction problem by introducing an additional‘ cost–constraint ’:
c T z ≤ c T y ∗ , equivalent to a m +1 = − c and b m +
=
c T y ∗ .
Note that the above cost–constraint separates the solution y ∗ of the original ILP in eq.
(2) fromthe other feasible integral points.
This is because y ∗ must achieve the minimum objective valuec T y ∗ amongst all the feasible integral points and hence no other feasible integral point
z can obtainobjective value less than c T y ∗ .
The new constraint [ a m +1 | b m +1 ] together with the original mconstraints guarantee that y ∗ is the only solution satisfying all of the ( m + 1) constraints.
This results in the following equivalent linear constraint satisfaction problem:
[Equation]
Constructing such an equivalent satisfaction problem requires access to the solution y ∗ of the original
ILP in eq.
(2) and that is already available to us for the training data samples.
By rolling up the costvector c into an additional constraint with the help of y ∗ , we have converted our original objective oflearning both the cost and constraints to just learning of constraints in eq.
The main intuition behind our framework comes from the observation that each of the m linearconstraints defining the feasible region are essentially m linear binary classifiers (hyperplanes) in 198 R n separating the ground truth y ∗ from the infeasible region.
The additional cost–constraint ineq.
(4) separates y ∗ from other integral samples feasible w.r.t.
the original m constraints.
Learningconstraints of an ILP is akin to simultaneously learning m linear binary classifiers separating y ∗from other infeasible points of the original ILP along with learning a classifier for cost–constraint,separating y ∗ for other feasible integral points.
For a vector y ∗ to lie inside the feasible region, all the classifiers need to classify it positively, andhence it acts as a positive data point for all the m + 1 binary classifiers.
In the absence of explicitlyprovided negative samples, we propose a couple of strategies for sampling them from Z n for eachground truth y ∗ s .
We discuss them in detail in section 3.3.
For now, let N y ∗ be the set of all thenegative samples generated by all our sampling techniques for a ground truth positive sample y ∗ .
Incontrast to a positive point which needs to satisfy all the m + 1 constraints, a negative point becomesinfeasible even if it violates any one of the m + 1 constraints.
As a result, it would suffice if any oneof the m + 1 classifiers correctly classify it as a negative sample (refer to appendix for an illustration).
While learning, one cannot assign a negative sample to any specific classifier as their parametersare being updated continuously.
Instead, we make a soft assignment depending upon the Euclideandistance of the negative sample from the hyperplane.
With this intuition, we now formally define our
ILP–Loss.
Formulating solver-free ILP–Loss: Let d ( z ; [ a i | b i ])
=
a Ti z + b i | a i | represent the signed Euclideandistance of a point
z ∈ R n from the hyperplane corresponding to the constraint [ a i | b i ] .
We want thesigned distance from all the hyperplanes to be positive for the ground truth samples and negativefrom at least one hyperplane for all the sampled negative points.
We operationalize this via a marginbased loss function:
[Equation] [Equation]
We call L + , L − , and L o as the positive, negative and covariance loss respectively.
The average in L +ensures that a ground truth sample is positively classified by all the classifiers.
µ + and µ − are thehyperparameters representing the margins for the positive and the negative points respectively.
w i y − in
L − represents the soft assignment of y − to the i th constraint [ a i | b i ] and is computed by temperatureannealed softmax over the negative distances in eq.
Softmax ensures that the hyperplane which ismost confident to classify it as negative gets the maximum weight.
When y − lies inside the feasibleregion, then the most confident classifier is the one closest to y − .
The temperature parameter τ needs to be annealed as the training progresses.
A high temperatureinitially can be seen as ‘ exploration ’ for the right constraint that will be violated by y − .
This is important as the constraints are also being learnt and are almost random initially, so a givennegative y − should not commit to a particular hyperplane.
Additionally, this encourages multipleconstraints to be violated for each negative, which leads to a robust set of constraints.
As thetraining progresses, we reduce the temperature τ , ensuring that the most confident classifier with leastsigned distance gets almost all the weight.
If y − is correctly classified as a negative with a marginµ − by any classifier i.e. , d ( y − ; [ a i | b i ]) ≤ − µ − for some i , then the corresponding negative loss,max { 0 , µ − + d ( y − ; [ a i | b i ]) } , becomes zero, and a low value of τ ensures that it gets all the weight.
The last term, L o acts as a regularizer and tries to ensure that no two learnt constraints are similar.
This is done by minimizing the cosine similarity between all the pairs of constraints.
The weightsλ pos , λ neg , and λ cov are computed automatically with a multi-loss weighing technique using coefficient of variations as described in Groenendijk et al.
Intuitively, the loss term with maximumvariance over the learning iterations, gets most of the weight.
Next, we discuss different ways ofsampling the negative points for computing the ILP–Loss.
3.3 Negative Sampling
A meaningful computation of ILP–Loss in eq.
(5) depends crucially on the negative samples N ∗ y .
Randomly sampling negatives from Z n is one plausible strategy, but it may not be efficient from thelearning perspective: any point which is far away from any of the classifiers will easily be classifiedas negative and will not contribute much to the loss function.
In response, we propose multiplealternatives for sampling the negative points:
Integral k-hop neighbours: To generate negatives, we sample the integral neighbours that are at aHamming Distance of k from y ∗ .
For small k , these form the hardest negatives as they are closestto the positive point.
Note that it is possible for a few integral neighbours to be feasible w.r.t.
the mconstraints, but they must have worse cost than the given ground truth y ∗ .
Such samples contributetowards learning the cost parameters θ c .2.
Project and sample: In this method, we project the ground truth y ∗ on each of the m hyperplanesand then randomly sample a neighbour of the projection, generating a total of m negatives.
Samplingprobability in the j th dimension depends on the j th coordinate of the projection: if value of thej th coordinate is r / ∈ Z , then we sample floor ( r ) and ceil ( r ) with probability ceil ( r ) − r andr − floor ( r ) respectively.
If r ∈ Z , then we sample r with probability 1 .
Projection samples areclose to the boundary of the currently learnt polyhedron, thus taking the training progress into account.
Further, each hyperplane is likely to be assigned to a close-by negative due to projection sampling.3.
Batch negatives: In this strategy, we consider every other ground truth y ∗ s ′ s ′ ̸
= s in the minibatchas a potential negative sample for y ∗ s .
We call it as a ‘batch–negative’.
This is particularly useful forlearning the cost parameters θ c when the learnable constraints are the same for all the ground truthsamples, such as in sudoku.
In such cases, a batch–negative y ∗ s ′ being a feasible point of the original
ILP formulation, must always satisfy all of the m learnable constraints of the original ILP in eq.
Hence, the only way for y ∗ s ′ to be correctly classified as a negative for y ∗ s is by violating the costconstraint in eq.
(4), i.e. , c T y ∗ s ′ > c T y ∗ s . Learning cost parameters θ c that result in violation of thecost constraint for every batch–negative helps in ensuring that the ground truth y ∗ s indeed has theminimum cost. 3.4 Implementation Details
Parameterization of equality constraints: Recall that we replace an equality u Ti z = v i by twoinequality constraints: v i ≤ u Ti z ≤ v i .
In practice, to enhance learnability, we add a small margin ofϵ on both sides: v i − ϵ ≤ u Ti z ≤ v i + ϵ .
We pick an ϵ small enough so that the probability of thenew feasible region to include an infeasible integral point is negligible.
Soft assignment of negatives to hyperplanes: While computing the weights in eq.
(9), we ensurethat the gradients flow only through the margin loss term in eq. (7), and not through the weights w i y − . This is done to avoid the pathological behaviour of decreasing the loss by only changing the weightsbut not the individual hyperplane distances. Known boundaries: In many cases, the boundary conditions on the output variables are known, i.e. ,z ∈ Y =
[ l, u ] n , where l and u are the lower and upper bounds on each dimension of z .
We handlethis by adding known boundary constraints: l ≤ z i ≤ u, ∀ i ∈ { 1 · · · n } .
Over-parameterization of constraints: As done in Paulus et al.
, we also over-parameterize eachconstraint hyperplane by an additional learnable offset vector o i which can be viewed as its own local origin.
Radius r i represents its distance from its own origin
o i , resulting in the following hyperplanein the base coordinate system: a Ti z
+ r i − o
Ti a i / | a i | ≥ 0 .
Initialization of A : The way we initialize the constraints may have an impact on the learnability.
While [Paulus et al., 2021] propose to sample each entry of a i uniformly (and independently) between[ − 0 .
5] , we also experiment with a standard Gaussian initialization.
The latter results in initialhyperplanes with their normal directions ( a i ’s) uniformly sampled from a unit hyper-sphere.
Inexpectation, such initialization achieves minima of L o that measures total pairwise covariance.
The goal of our experiments is to evaluate the effectiveness and scalability of our proposed approachcompared to the SOTA black-box solver based approach, i.e., CombOptNet
[Paulus et al., 2021].
We experiment on 3 datasets: (a) visual sudoku [Wang et al., 2019] (b) random constraints [Pauluset al., 2021] (c) key-point matching [Paulus et al., 2021].
We also compare with an appropriatelydesigned neural baseline for each of our datasets.
To measure scalability, in each of the domains, wecompare performance of each of the algorithms in terms of time taken, as well as accuracy obtained,for varying problem sizes.
To solve the learnt ILPs, we use Gurobi ILP solver [Gurobi Optimization,
LLC, 2022] available under ‘Named-user academic license’.
All our experiments were run on 11 GB‘GeForce GTX 1080 Ti’ GPUs installed on a machine with 2.60GHz Intel(R)
Xeon(R)
{+Gold 
 CPU.+} For each of the algorithms, we kept a maximum time limit of 12 hours for training.
We nextdescribe details of each of these datasets, experimental methodology and our results.
4.1 Visual Sudoku
This task involves learning the rules of the sudoku puzzle.
The input x is an image of a sudoku puzzlewhere a digit is replaced by a random MNIST image of the same class.
A k × k sudoku puzzle canbe viewed as an ILP with 4 k 2 equality constraints over k 3 binary variables [Bartlett et al., 2008].
Each of the digit images in the k 2 cells are decoded into a k dimensional real vector, resulting ina k 3 dimensional cost c ∈ R n where n = k 3 .
The k 3 dimensional binary output vector (solution)y ∗ is created by representing each of the k 2 cells of the puzzle by a k dimensional binary vectorwhich takes a value of 1 at i th dimension if and only if the cell is filled by digit i .
Here, the costvector needs to be inferred from the input image.
Our objective is to learn the constraint matrix
A = f A ( x ; θ A ) =
θ A and vector b = f b ( x ; θ b ) = θ b , representing the linear constraints of sudoku.
The cost vector c = f c ( x ; θ c ) is a function of the input image x , parameterized by θ c which needs tobe learned.
Dataset: We experiment with 3 different datasets for k = 4 , 6 , and 9 .
We first build symbolic sudokupuzzles with k digits, and use MNIST to convert a puzzle into an image.
For 9 × 9 sudokus, we use astandard dataset from a Kaggle [Park, 2017], for k =
we use publically available data from [Arcotand Kalluraya, 2019], and for k =
we use the data generation process described in [Nandwaniet al., 2022].
We randomly select 10 , 000 samples for training, 1000 samples for validation, and 500samples for testing for each k .
To generate the input images for visual-sudoku, we use the officialtrain and test split of MNIST[Deng, 2012].
Digits in our train and test splits are randomly replacedby images in the MNIST train and test sets respectively, avoiding any leakage.
Methodology: (a) # of learnable constraints: For k = 4 and 6 , we keep m =
( n + 1) as the numberof learnable constraints where n = k 3 is the number of binary variables. For k = 9 , we learnm = ( n + 1) / 2 equality constraints, which is equivalent to learning ( n + 1) inequality constraintswith parameter tying. (b) Margin: We find that a margin of 0 .
01 works well across domains andproblem sizes.
(c) Temperature: we start with a temperature τ = 1 and anneal it by a factor of 0 .
1whenever the performance on a small held out set plateaus.
(d) Early Stopping: we early stop thetraining based on validation set performance, with a timeout of 12 hours for each experiment.
Webypass the validation bottleneck of solving the ILPs from scratch by providing the gold solutions ashints when invoking Gurobi.
(e) Negative Sampling:
For sampling neighbors, we select all the n onehop neighbors, and an equal number of randomly selected 2,3 and 4 hop neighbors, resulting in atotal of 4 n neighbors.
(f) Initialization: we initialize a i from a standard Gaussian distribution for
CombOptNet and our method.
We use a CNN based neural network, LeNet [Lecun et al., 1998], to decode the images into digits.
Our neural baseline is Recurrent Relational Networks (RRN)
[Palm et al., 2018]: a purely neural approach for reasoning based on Graph Neural Networks.
We use the default set of parametersprovided in their paper for training RRNs.
We note that the RRN baseline uses additional informationin the form of graph structure which is not available to our solver.
We make this comparison to seehow well we perform compared to one of the SOTA techniques for this problem.
Results:
[Table]
Table 1 compares the performanceand training time of our methodagainst the different baselines described above.
CompOptNet fails miserably on this problem, not being ableto complete training for any of thesizes in the stipulated time.
The purelyneural model gives a decent performance and is competitive with oursfor smaller board sizes 1 .
But for largerboard size of 9 × 9 , we beat RRN based model by a significant margin of about 25 points.
4.2 Random constraints
This is the synthetic dataset borrowed from Paulus et al..
The training data D = { ( c s , y ∗ s ) | s ∈{ 1 · · · S }} is created by first generating a random polyhedron P ⊆ Y by sampling m ′ hyperplanes { [ a ′ i |
b ′ i ] | a ′ i , b ′ i ∈ R n , i ∈ { 1 · · · m ′ }} in an n dimensional bounded continuous space
Y .
A cost vector c s ∈ R n is then randomly sampled and corresponding y ∗ s is obtained asy ∗ s = arg min
z ∈P c Ts z , z ∈ Z n .
Objective is to learn a constraint matrix A and vector b suchthat for a ground truth ( c , y ∗ ) pair, y ∗ =
arg min z ∈Y c T z , z ∈ Z n , and Az + b ≥ 0 .
Dataset: Restricting Y to [0 , 1] n and [ − 5 , 5] n results in two variations, referred to ‘binary’ and‘dense’ settings, respectively.
For both of the output spaces, we experiment with four different settingsin n = 16 dimensional space by varying the number of ground truth constraints as m ′
= 1 , 2 , 4 , and8 .
For each m ′ , we experiment with all the 10 datasets provided by Paulus et al. and report mean overthe 10 models.
The training data in each case consists of 1600 ( c , y ) pairs and model performance istested for 1000 cost vectors.
Methodology: we keep number of learnable constraints as twice the number of ground truthconstraints, i.e. , m = 2 m ′ , as Paulus et al. report best performance in most of the settings with it.
Here we initialize a i uniformly between [ − 0 . 5 , − 0 . Rest of the hyperparamters are set as in thecase of sudoku.
A neural baseline (an MLP) is shown to perform very badly for this purely symbolicproblem [Paulus et al., 2021], and hence we do not include this in our experiments.
Results: Table 2 presents the comparison of the two algorithms in terms of accuracy as well as timetaken for training.
While we perform marginally better than CombOptNet in terms of training time,our performance is significantly better, on almost all problems in the dense setting, and the largerproblems in the binary setting.
We are more than 35 accuracy points better compared to CombOptNeton the largest problem instance with 8 constraints in both settings.
This result demonstrates that ourapproach is not only faster in terms of training time, but also results in better solutions even when thebaseline converges to a solution.
This validates the effectiveness of our framework.
[Table]
[Table]
4.3 Keypoint Matching
Here we experiment on a real world task of matching key points between two images of differentobjects of same type Rolínek et al.
[2020b].
The input x consists of a pair of images ( I 1 , I 2 ) of sametype along with a shuffled list of coordinates (pixels) of k keypoints in both images.
The task is tomatch the same keypoints in the two images with each other.
The ground truth constraint enforces abijection between the keypoints in the two images.
The output y is represented as a k × k binarypermutation matrix, i.e. , entries in a row or column should sum to 1 .
The cost c is a k × k sizedvector parameterized by θ c .
The goal is to learn A , b and θ c .
Dataset: We experiment with image pairs in SPair-17k [Min et al., 2019] dataset used for the taskof keypoint matching in [Rolínek et al., 2020b].
Since the neural ILP models can deal with a fixedsize output space, we artificially create 4 datasets for 4,5,6 and 7 keypoints from the original dataset.
While generating samples for k keypoints dataset, we randomly sample any k annotated pairs frominput image pairs that have more than k keypoints annotated.
See appendix for details of the samplesin each dataset.
Methodology: For each k , the number of learnable constraints is set to 2 k : same as the number ofground truth constraints.
In our method, for keypoints 5,6 and 7, we initialize the backbone costparameters θ c with the one obtained by training it on 4 keypoints. We start with a temperature τ = 0 . 5and anneal it by a factor of 0 .
2 at 10 th and 30 th epoch. Here we initialize a i uniformly between[ − 0 .
5 , − 0 .
Rest of the hyperparmeters are same as those used for sudoku. We compare againsta strong neural baseline which is same as the backbone model parameterizing the k 2 dimensionalcost vector in [Rolínek et al., 2020b]. It is trained by minimizing Binary Cross Entropy loss betweennegative learnt cost ( − c ) and target y ∗ .
We create an additional baseline by doing constrainedinference with ground truth constraints and learnt cost (‘Neural + CI’) in table 3.
[Table]
Results: Table 3 presents the percentage of keypoints matched correctly by different models.
Both CombOptNet and our method achieve similar performance.
Note that we obtain a gain of2 points over the purely neural baseline withoutthe knowledge of constraints.
This is almostsame as the gain obtained by constrained inference with known constraints on top of neuralmodel.
For this task, we did not observe anygain in the training time over CombOptNet.
Thisis expected as the bottleneck here is the backbone neural model parameterizing the cost function.
Even though our method spends less time in each iteration, it takes more number of iterations to train,balancing out the gain obtained by speeding up each iteration.
Conclusion and Future Work
We have presented a solver free framework for scalable training of a neural ILP architecture.
Ourmethod learns the linear constraints by viewing them as linear binary classifiers, separating thepositive points (inside the polyhedron) from the negative points (outside the polyhedron).
Whilegiven ground truth acts as positives, we propose multiple strategies for sampling negatives.
A simpletrick using the available ground truth outputs in the training data, converts the cost vector into aconstraint, enabling us to learn the cost vector and constraints in a similar fashion.
Future work involves extending our method to neural-ILP-neural architectures, i.e., where the outputof ILP is an input to a downstream neural model.
Second, a neural ILP model works with a fixeddimensional output space, even though the constraints for the same underlying problem are the samein first order logic, e.g. , constraints for k × k sudoku puzzles remain the same in first order irrespectiveof k .
Creating neural ILP models that can parameterize the constraints on the basis of the size of theinput (or learnt) cost vector is a potential direction for future work.
Lastly, the constraints learnt in aneural ILP architecture might not be interpretable even if the ground truth constraints are.
In future,we would like to develop methods that distill the learnt constraints into a human-interpretable form.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] .
In Conclusion and future works
section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] We will make the code and data available on acceptance
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Details are in experiments section 4 under ‘Training Methodology’ and in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
In one of the experiments on random constraints.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In the experiments section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Datasets are cited inthe respective experiment sections.(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
N/A]