L EARNING TO S OLVE C ONSTRAINT S ATISFACTION P ROBLEMS WITH R ECURRENT T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Constraint satisfaction problems (CSPs) are about ﬁnding values of variables that satisfy the given constraints.
We show that the Transformer model extended with recurrence is a viable approach to learning to solve CSPs in an end-to-end manner, having clear advantages over the state-of-the-art methods such as Graph Neural Networks, SATNet and some neuro-symbolic models.
With the ability of Transformers to handle visual input, the proposed Recurrent Transformer can straightforwardly be applied to visual constraint reasoning problems while successfully addressing the symbol grounding problem.
We also show how to leverage deductive knowledge of discrete constraints in the Transformer’s inductive learning to achieve sample-efﬁcient learning and semi-supervised learning for CSPs
I NTRODUCTION
Constraint Satisfaction Problems (CSPs) are about ﬁnding values for variables that satisfy the given constraints.
They have been much studied in symbolic AI with the emphasis on designing efﬁcient algorithms for deductively ﬁnding solutions for explicitly stated constraints.
On the other hand, there are rising interests in CSPs with the deep learning approach, where the focus is on inductively learning the constraints and solving them in an end-to-end manner.
One of the main methods is graph neural networks (GNNs).
For example, Recurrent Relational Network (RRN) (Palm et al., 2018) uses message passing over graph structures to learn logical constraints, achieving high accuracy on Sudoku.
On the other hand, it uses hand-coded information about Sudoku constraints, namely, which variables are allowed to interact.
Moreover, it is limited to textual input.
SATNet (Wang et al., 2019) is a differentiable MAXSAT solver that can infer logical rules and be integrated into DNNs.
SATNet was shown to solve even Visual Sudoku, where the input is a hand-written Sudoku board.
The problem is harder because a model has to learn how to map visual inputs to symbolic digits without explicit supervision.
However, Chang et al.
(2020) noted that the experiment had a label leakage issue.
With the proper evaluation, SATNet’s performance on Visual Sudoku dropped to 0%.
Moreover, SATNet’s evaluation is limited to easy puzzles, and it does not perform well on hard puzzles that RRN could solve.
On another aspect, although these models could learn complicated constraints purely from data, in many cases, (part of) constraints are already known, and exploiting such deductive knowledge in inductive learning could be helpful for sample-efﬁcient and robust learning.
The problem is challenging, especially if the knowledge is in the form of discrete constraints, whereas standard deep learning is mainly about continuous and differentiable parameter optimization.
This paper provides a viable solution to the limitations of the above models based on the Transformer architecture.
Transformer-based models have not been shown effective for CSPs despite their widespread applications in language (Vaswani et al., 2017; Zhang et al., 2020;
Helwe et al., 2021;
Li et al., 2020) and vision (Dosovitskiy et al., 2020; Gabeur et al., 2020).
Creswell et al.
(2022) asserted that Transformer-based large language models (LLMs) tend to perform poorly on multi-step logical reasoning problems.
In the case of Sudoku, the typical solving requires about 20 to 60 steps of reasoning.
Despite the various ideas on prompting GPT-3, GPT-3 is not able to solve Sudoku.
Nye et al.
(2021) note that LLMs work well for system 1 intuitive thinking but not for system 2 logical thinking.
Given the superiority of other models on CSPs, one might conclude that Transformers are unsuitable for CSPs.
We ﬁnd that Transformers needs to incorporate recurrence to successfully apply to CSPs.
The added recurrence encourages the Transformer model to apply multi-step reasoning similar to RRNs.
Interestingly, this simple change already yields better results than the other models above and gives several other advantages.
The learning is more robust than SATNet’s.
By looking at the learned attention matrices, we could interpret what the Transformer has learned.
Intuitively, multi-head attention extracts distinct information about the problem structure.
Adding more attention blocks and recurrences tends to make the model learn better.
Analogous to the extension Vision Transformer (Dosovitskiy et al., 2020) made, our model can be easily extended to process the visual input.
Moreover, the model avoids the symbol grounding problem that SATNet encountered.
In addition, we present a way to inject discrete constraints into Recurrent Transformer training, borrowing the idea from (Yang et al., 2022).
That paper shows a way to encode logical constraints as a loss function and use Straight-Through Estimators (STE)
(Courbariaux et al., 2015) to make discrete constraints meaningfully differentiable for gradient descent.
We apply this idea to Recurrent
Transformers with some modiﬁcations.
We note that adding explicit constraint loss to all recurrent layers helps Transformers learn more effectively.
We also add a constraint loss to the attention matrix so that constraints can help learn better attention values.
Including these constraint losses in training improves accuracy and lets the Transformer learn with fewer labeled data (semi-supervised learning).
In summary, the paper makes the following contributions.
Recurrent Transformer for Constraint Reasoning.
We show that the Recurrent Transformer is a viable approach to learning to solve CSPs, with clear advantages over the state-of-the-art methods, such as RRN and SATNet.
Symbol Grounding with Recurrent Transformers.
With the ability of Transformers to handle vision problems well, we demonstrate that our model can straightforwardly be applied to visual constraint reasoning problems while successfully addressing the symbol grounding problem.
It achieves 93.5% test accuracy on the SATNet’s Visual Sudoku test set, for which even the enhanced SATNet from (Topan et al., 2021) could achieve only 64.8% accuracy.
Injecting Logical Constraints into Transformers.
We show how to inject discrete logical constraints into Recurrent Transformers training to achieve sample-efﬁcient learning and semi-supervised learning for CSPs.
B ACKGROUND 2. C ONSTRAINT S ATISFACTION P ROBLEMS
A constraint satisfaction problem 1 is deﬁned as (cid:104) X , D , C (cid:105) where
X = {X 1 , . . .
, X t } is a set of t logical variables; D = { D 1 , . . .
, D t } and each D i is a ﬁnite set of domain values for logical variable
X i ; and C is a set of constraints.
An atom (i.e., value assignment) is of the form
X i = v where v ∈ D i .
A constraint on a sequence (cid:104)X i , . . .
, X j (cid:105) of variables is a mapping: D i × · · · × D j
→ { TRUE , FALSE } that speciﬁes the set of atoms that can or cannot hold at the same time.
A (complete) evaluation is a set of t atoms {X i = v | i ∈ { 1 , . . .
, t } , v ∈ D i } .
An evaluation is a solution if it does not violate any constraint in C , i.e., it maps all constraints to TRUE .
One of the commonly used constraints is the cardinality constraint :
[Equation]
where l and u are nonnegative integers denoting bounds, and for k ∈ { i, . .
. , j } , X k ∈ X and v k ∈ D k .
Cardinality constraint (1) is TRUE iff the number of atoms that are true in it is between l and u .
If l = u , constraint (1) can be simpliﬁed to
[Equation]
which is TRUE iff the number of atoms in the given set is exactly l .
If i = j and l = 1 , constraint (2) can be further simpliﬁed to X i
= v i .
Example 1 (CSP for Sudoku)
A CSP for Sudoku puzzle is such that X
= { cell 1 , . .
. , cell 81 } denotes all 81 cells in a Sudoku board; D = { D 1 , . . .
, D 81 } and D i
= { 1 , . .
. , 9 } denotes all possible values in each cell; and C consists of constraints
cell i
=
d for each given digit d at cell i , and constraints
[Equation]
for d ∈ { 1 , . . .
, 9 } and any set { i, . . .
, j } of 9 cell indices that belong to the same row/column/box, saying that “each digit d should appear exactly once in each row/column/box.”
The solution to the
CSP corresponds to the solution to the Sudoku puzzle.
2. R ELATED M ODELS
Graph Neural Networks.
Graph neural networks (Gori et al., 2005; Veliˇckovi´c et al., 2018; Kipf &
Welling, 2017) are closely related to Transformers.
GNNs encode graph structures where the adjacent nodes affect each other by recurrent message passing.
Vanilla Transformer does not have the explicit encoding of the graph structure.
Or in other words, it assumes fully connected graphs and does not exploit the sparsity of graphs.
There have been many recent works to bring the complementary nature of Transformers and GNNs, such as (Dehghani et al., 2018; Dai et al., 2019; Veliˇckovi´c et al., 2018;
Yun et al., 2019;
Rong et al., 2020; Cai & Lam, 2020;
Hu et al., 2020;
Ying et al., 2021; Dwivedi &
Bresson, 2021).
SATNet.
SATNet (Wang et al., 2019) explores semideﬁnite program relaxations as a tool for solving
MAXSAT, which can be employed as a layer in deep neural networks to solve composite learning problems, such as solving Visual Sudoku puzzles, that require both visual perception and logical reasoning.
SATNet learns to solve Visual Sudoku puzzles without any hand-coded knowledge, but its training relied on the “leakage" of labels, as discovered by Chang et al.
(2020) and remedied by Topan et al.
The following ﬁgure is from (Topan et al., 2021) to illustrate the symbol grounding problem in the context of a 3 × 3 portion of Sudoku.
The task is to identify a in given only a invisual and a out as the labels (called Ungrounded Dataset).
However, the original SATNet training was performed on Grounded Dataset, where the labels included a in as well so that a digit classiﬁer was trained with the label in a supervised way.
Obviously, learning from the Ungrounded Dataset is harder because learning cannot be broken into two stages, classifying and solving.
Neuro-Symbolic Models for Constraint Reasoning.
CLR-DRNet (Bai et al., 2021) is a curriculumlearning-with-restarts model which leverages a Deep Reasoning Network (DRNet)
(Chen et al., 2019)
and is applied to Visual Sudoku with a grounded dataset using rules given as a loss function but cannot be applied to Ungrounded Datasets.
Neuro-symbolic models like DeepProbLog (Manhaeve et al., 2018), NeurASP (Yang et al., 2020), and NeuroLog (Tsamoura et al., 2021) integrate neural networks with logic programming languages.
They perform perception in neural networks and logical reasoning in hand-written logic programs.
These models have shown that neural network training can beneﬁt from constraints in logic programs.
R ECURRENT T RANSFORMERS FOR C ONSTRAINT S ATISFACTION P ROBLEMS 3. R ECURRENT T RANSFORMERS
Given a constraint satisfaction problem (cid:104) X , D , C (cid:105) such that, for i ∈ { 1 , . . .
, t } , 1 ≤ | D i | ≤ c for a constant c , the Recurrent Transformer takes as an input the sequence (cid:104)X 1 , . . .
, X t (cid:105) of logical variables, and outputs the probability distribution over the values in the domain D i of each X i .
Let c i be the domain size of X i .
Without loss of generality, we assume values in D i are represented by their indices, i.e., D i = { 1 , . . .
, c i } .
The probability of X i =
j is given by the j -th value of the output for X i .
A logical variable X i is treated as a token whose token embedding is a vector of length d h encoding the given information about this logical variable (e.g., some numbers, a textual description, an image, etc).
The position embedding of X i is a randomly initialized vector of length d h and is to be learned to record data-invariant information for logical variable X i .
Let E tok , E
pos ∈ R t × d h denote the token and position embeddings of t logical variables.
The r -th recurrent step in a Recurrent Transformer with L self-attention blocks and R recurrences can be formulated as follows ( r ∈ { 1 , . . .
, R } ):
H ( r, 0) = H ( r − 1 ,L )
H ( r,l ) =
block l ( H ( r,l − 1) )
∀ l ∈ { 1 , . . .
, L }
X ( r,l ) =
softmax(layer _ norm( H ( r,l ) ) · W out ) ∀ l ∈ { 1 , . . .
, L } where the initial hidden embedding H (0 ,L ) is E tok + E pos and + denotes element-wise addition;
H ( r,l ) ∈
R t × d h denotes the hidden embedding of t logical variables after the l -th (self-attention) block in the r -th recurrent step; block l denotes the l -th Transformer block in the model; layer _ norm denotes layer normalization; · denotes matrix multiplication, W out ∈ R d h × c is the weight of the output layer; and X ( r,l ) ∈ [0 , 1] t × c denotes the NN output with the hidden embedding H ( r,l ) .
The ﬁgure of the model architecture and the formal deﬁnition of block l are given in Appendix A.
For logical variable X i and its domain D i = { 1 , . . .
, c i } where c i ≤ c , the scalar X ( r,l ) i,j (i.e., element i, j of matrix X ( r,l ) ) is interpreted as the probability of atom X i = j for j ∈ { 1 , . . .
, c i } .
Example 2 (Recurrent Transformer for Visual Sudoku) Figure 1 shows how a Recurrent Transformer is used to solve the Visual Sudoku problem from (Wang et al., 2019).
Here, a Sudoku board is represented by 9 × 9 = 81 MNIST digit images where empty cells are represented by images of digit 0.
The Recurrent Transformer takes as an input the sequence (cid:104) cell 1 , . .
, cell 81 (cid:105) of logical variables, and outputs the probability distribution over atoms cell i = v for i ∈ { 1 , . .
. , 81 } , v ∈ { 1 , . .
The given information for each logical variable cell i is the MNIST digit image in the i -th cell.
Within the Recurrent Transformer, the token, position, and hidden embeddings E tok , E pos , H ( r,l ) are in
R 81 × 128 , and the output X ( r,l ) is in R [Table]
Recurrence in Transformers.
Adding recurrence to standard Transformer is not a new idea, but its application to CSP is novel.
Other implementations of recurrence in Transformers (Dehghani et al., 2018; Hao et al., 2019) largely follow the traditional training and inference procedure, where causal attention encourages the model to focus on the next token to be generated.
In contrast, we use an encoder-only model and force the Transformer to update all unknown variables at every recurrence, which encourages it to learn a convergent reasoning algorithm while considering the global picture to generate a solution incrementally in any order.
We ﬁnd that with this framework, the
Transformer solves the CSP problems incrementally, gradually predicting more unknown variables after it is conﬁdent in others.
Since Universal Transformers (Dehghani et al., 2018) produce outputs sequentially, they are forced to predict output variables in an arbitrary ﬁxed order, only attending to previous sequence elements, which may not be suitable for reasoning about many CSPs.
We remove masked attention during training and inference, and instead allow for attention to every other input variable.
With our design, during inference time, an arbitrary number of recurrences can be used (different number from the training time), which can further boost performance.
Furthermore, instead of computing a single loss on the ﬁnal output as in Universal Transformer, we accumulate loss for each output from every attention block at every recurrent step, which yields better performance, as shown in Appendix B.2.
3. T RAINING O BJECTIVE
Consider a labeled data instance (cid:104) t , l (cid:105) where t is t input tokens (that will be turned into the token and position embeddings E tok , E pos ) and l ∈ { na, 1 , . .
. , c } t is a label for t , where na denotes unknown label.
Let X ( r,l ) ∈
R t × c be the NN output with input t at recurrent step r ∈ { 1 , . . .
, R }
and block l ∈ { 1 , . .
, L } .
The cross-entropy loss L cross is deﬁned as follows, where l i denotes element i in l .
[Equation]
For example, in ungrounded Visual Sudoku, t is a list of t = 81 MNIST images and l ∈ { na, 1 , . .
, 9 } 81 is the “ungrounded” solution for the Sudoku puzzle where the label for all given digits is na .
The cross-entropy loss on NN output X ( r,l ) ∈
R 81 × 9 depends only on empty cell predictions.
In other words, no supervision for given digits is provided during training.
The baseline loss L base is the sum of L cross over NN output X ( r,l ) from all recurrent steps and blocks.
[Equation]
Note that we apply cross-entropy loss to the NN outputs from all recurrent steps and all layers instead of from the very last one.
We ﬁnd that this makes the Recurrent Transformer converge faster.
E XPERIMENTS WITH R ECURRENT T RANSFORMER
We use LxRyHz to denote our Recurrent Transformer with L = x self-attention blocks, R =
y recurrent steps, and z self-attention heads.
If omitted, the number of heads z is 4 and the embedding size d h is 128.
4. S UDOKU
In this section, we apply Recurrent Transformers to solve Sudoku problem, where a board can be either textual or visual.
Dataset.
For textual Sudoku, we use the SATNet dataset from (Wang et al., 2019) and the RRN dataset from (Palm et al., 2018).
The difference is that RRN dataset is much harder and bigger (with 17-34 given digits in each puzzle and 180k/18k training/test data) than the SATNet dataset (with 31given digits and 9k/1k training/test data).
Each labeled data instance in textual Sudoku is (cid:104) t , l (cid:105) where t ∈ { 0 , . . .
, 9 } 81 denotes a Sudoku puzzle ( 0 represents an empty cell) and l ∈ { 1 , . .
, 9 } 81 is the solution to the puzzle.
For Visual Sudoku, we use the ungrounded SATNet-V dataset from (Topan et al., 2021).
SATNet-V was created based on the SATNet dataset where (i) each textual input in { 0 , . .
, 9 } in the training (or testing resp.)
split is replaced with a randomly-selected MNIST image in the MNIST training (or testing resp.)
dataset, and (ii) the label for each given digit is na , i.e., unknown label that cannot be used to help training.
In addition to SATNet-V, we created a new ungrounded dataset, RRN-V, following the same procedure based on RRN dataset.
For faster evaluation on RRN-V dataset, we randomly sampled 9k/1k training/test data and denoted it by “RRN-V (9k/1k)”.
Baselines and our Model.
We take RRN and SATNet as the baselines for textual Sudoku, and take
RRN, SATNet, and SATNet ∗ (Topan et al., 2021) (which resolves the symbol grounding issue of
SATNet by clustering the input images) as the baselines for Visual Sudoku.
As RRN was not designed for Visual Sudoku, we applied the same convolutional neural network (CNN) from (Wang et al., 2019) to turn each MNIST image into the initial number embedding of that cell in RRN.
For our method on both textual and Visual Sudoku, we apply Recurrent Transformer as in Figure 1 where the only difference is that the token embedding layer is a linear embedding layer for textual Sudoku and is the same CNN for Visual Sudoku.
All evaluations of our model use 32 recurrence steps for training and 64 for evaluation, as is done in (Palm et al., 2018).
[Table]
Table 1 shows that our method outperforms state-of-the-art neural network models for both textual and Visual Sudoku in different difﬁculties.
Note that among all methods, only RRN requires prior knowledge about Sudoku rules (i
[Table]
e., there is an edge in the graph between every 2 nodes if their related cells are in the same row/column/box).
Both RRN and SATNet fail on the (ungrounded) SATNet-V dataset due to the symbol grounding issue.
While SATNet ∗ could learn to solve Visual Sudoku with the ungrounded dataset, it requires training an InfoGAN with 13M parameters to cluster the inputs.
Unlike SATNet ∗ , our model works out-of-the-box on Visual Sudoku without carefully adjusting the structure and outperforms SATNet ∗ by a large margin.
Although the L1R32H4 model has already achieved the new state-of-the-art results, we may further improve the accuracy by increasing the number L of attention blocks, the number H of heads, or the hidden embedding size d h , with a trade-off of bigger model size.
We will analyze the effects of these decision choices on smaller datasets in the following sections.
4.1. A BLATION S TUDY ON M ODEL D ESIGN (L X R Y H Z ) WITH TEXTUAL S UDOKU
Effects of Blocks and Recurrences.
To analyze the effects of blocks and recurrences, we trained six LxRy(H4) models with different numbers of self-attention blocks L and recurrences R on 8k/2k (training/test)
RRN dataset with L base .
Figure 2 ( left ) compares the whole board accuracy of these models, showing that more self-attention steps (equal to L × R ) lead to higher accuracy.
With the same number of self-attention steps, when L is small (e.g., L ≤ 4 ), more parameters introduced by a bigger L slightly increase the accuracy.
On the other hand, adding recurrences is essential, and the non-recurrent model L64R1 performs poorly compared to the Recurrent Transformers.
The number of recurrences T during testing can be higher than R during training.
Indeed, Figure( right ) shows that when T ≤ 64 , the more recurrent steps T is, the higher accuracy is achieved with the same L1R32 model trained on 32 steps, and the improvement is bigger for harder puzzles.
What Multi-Head Attentions Look at.
Without prior knowledge of the Sudoku game, the Recurrent
Transformer learns purely from (cid:104) puzzle, solution (cid:105) pairs that each cell should pay attention to all cells in the same row, column, and 3 × 3 box through the attention mechanism.
We trained an L1R model with 1 to 4 self-attention heads on the SATNet dataset with L base .
We found that the attentions on row, column, and box are clearly separated in different heads if the number of heads is greater or equal to 3 and would merge otherwise.
Besides, more attention heads help faster convergence, and the accuracy may decrease if the number of attention heads is too small to capture different semantic meanings.
More details and the visualization of the attention matrices are given in Appendix B.1.
Effect of Positional Embedding.
To evaluate the effect of positional embedding, we trained the
L1R32 model without positional embedding on the SATNet dataset with L base , ﬁnding that removing positional embedding decreases the test accuracy from 100% to 0%.
This is because positional embedding is essential for a CSP as it is the only source to differentiate logical variables (e.g., cell 1 , . .
, cell 81 ) with the same given information (e.g., digit 2 in both cell 4 and cell 10 in Figure 1).
4.1. A NALYSES ON S YMBOL G ROUNDING WITH V ISUAL S UDOKU
In Visual Sudoku, we observed similar effects of different model designs as in textual Sudoku.
We refer the reader to Appendix B for more details.
In this section, we analyze how the symbol grounding issue is resolved in Recurrent Transformer by applying the same L1R32 model on both RRN-V dataset and its grounded version (i.e., the label for every given digit is provided instead of na ).
For each of the two trained models, we evaluate their (i) whole board accuracy, (ii) solution accuracy where a board is counted correct if the prediction on all non-given cells are correct (even when the given digits are incorrectly classiﬁed), and (iii) givens cell accuracy, i.e., per-cell classiﬁcation accuracy of the given cells.
When trained on the grounded dataset, the L1R32 model quickly learns to classify the givens inepoch, as shown in Figure 3 ( right ).
On the other hand, with the ungrounded dataset, the L1R model starts to classify the givens correctly at around epoch 85.
In Figure 3 ( left ) and ( right ), the solution accuracy and givens cell accuracy (with the ungrounded dataset) increase around the same time, indicating that digit classiﬁcation is being jointly learned with solving.
Interestingly, our model achieves 99.36% classiﬁcation (givens cell) accuracy without explicitly training for it.
Furthermore, the solution accuracy (75.5%) is consistently higher than the whole board accuracy (74.8%) as shown in Figure 3 ( left ), meaning that even when givens are not correctly classiﬁed, the solution can still be attained.
We attribute this to the fact that reasoning is on the latent space instead of classifying and solving in two steps, as SATNet does.
4. O THER E XPERIMENTS
Due to lack of space, we summarize other experiments with the Recurrent Transformer below and refer the reader to Appendix D for the details.
16x
Sudoku.
We train our L1R32H8 model ( d h = 256 ) on 16x16 textual Sudoku.
We generate two 10k (9k/1k training/test split) datasets of difﬁculty “simple” with an average of 111 givens and “medium” with an average of 95 givens.
With 64 recurrent steps during inference, we achieved 99.9% accuracy on both test sets, meaning there is only one wrongly predicted board solution.
Due to the absence of a 16x16 Sudoku generator that can produce fewer givens, we could not test on harder boards.
MNIST Mapping.
The MNIST Mapping problem was proposed in (Chang et al., 2020) as a simple test for the symbol grounding problem.
It requires learning a bijection that maps an image of MNIST digit to one of the 10 symbolic digits.
(Chang et al., 2020) shows that SATNet is sensitive to this task and often fails without delicate tuning.
Our Recurrent Transformer achieves 99% accuracy.
Nonograms.
Nonogram ( https://en.wikipedia.org/wiki/Nonogram ) is a game that consists of an initially empty N × N grid representing a binary image, where each cell must take on a value of 0 or 1.
Each row and column have constraints that must be satisﬁed to complete the image successfully.
A constraint for a row/column is a list of numbers, where each number corresponds to contiguous blocks of cells with value 1 for a row/column.
The Recurrent Transformer L1R16H achieved 97.5% test accuracy on 7x7 grids and 78.3% on 15x15 grids.
I NJECTING L OGICAL C ONSTRAINTS IN R ECURRENT T RANSFORMERS 5. I NJECTING G ENERAL C ARDINALITY C ONSTRAINTS VIA STE
Although Recurrent Transformers can learn to solve CSPs purely from labeled data, we could inject the known constraints to help Transformers learn with fewer labeled data.
In this section, we follow the idea from CL-STE
(Yang et al., 2022)
and propose a lightweight constraint loss method for a special family of constraints in CSP, namely the cardinality constraint that restricts the number of atoms in a set that can hold at the same time.
The main idea of CL-STE is to use a binarization function B to turn continuous values x in NN output X ( r,l ) into discrete values B ( x ) where B ( x ) is 1 (denoting TRUE ) if x ≥ 0 . 5 and 0 (denoting FALSE ) otherwise.
A constraint loss is then deﬁned on these Boolean values and a set of propositional formulas in Conjunctive Normal Form (CNF).
Since ∂ B ( x ) ∂x is zero almost everywhere, the idea of STE is to replace ∂ B ( x ) ∂x with a straight-through estimator ∂s ( x ) ∂x for some (sub)differentiable function s ( x ) so that the constraint loss has a non-zero gradient on X ( r,l ) .
While CL-STE has successfully injected discrete constraints into NN training, representing cardinality constraints in CNF is tedious.
On the other hand, we notice that the binarization function B ( x ) enables direct counting on discrete values.
Under this observation, for the cardinality constraint
[Equation]
we construct a vector x ∈ R k of probabilities of the atoms in the given set such that x j (i.e., element j in x ) is the probability of X ij = v j for j ∈ { 1 , . . . , k } , and design a constraint loss as follows
[Equation]
where scalar c ( x ) = (cid:80) B ( x ) = (cid:80) j B ( x j ) , and 1 condition is 1 if condition is true, 0 otherwise.
Similarly, constraint |{X i 1 = v 1 , . . .
, X ik = v k }| = n can be encoded in the following loss.
[Equation]
In constraint losses (4) and (5), c ( x ) is the number of 1s in the binarized vector B ( x ) , which corresponds to counting the number of true atoms in constraints (1) and (2).
Note that binarization function B ( x ) enables the counting, but its gradient ∂ B ( x ) ∂x is always 0 whenever differentiable, so minimizing (4) and (5) won’t work on updating NN parameters.
As with CL-STE, we use the identity STE to replace the gradient ∂ B ( x ) ∂x with 1 so that the gradient of each constraint loss to B ( x ) becomes the “straight-though estimator” of the gradient to x .
In this way, we can do counting on the NN output with meaningful gradients.
While CL-STE could also represent the constraint loss (5) for n = 1 (uniqueness and existence of values), the size of the CNF representation could be huge.
Example 3 (Constraint Loss on Output) Cardinality constraint loss (5) can be used to deﬁne the constraints in Sudoku problem
[Equation]
where X ( r,l ) ∈ R 81 × 9 is the NN output; X row , X col , X box ∈ R 81 × 9 are reshaped copies of X ( r,l ) such that each row in them contains the predictions in the same row/column/box; and X ki, : denotes row i of matrix X k .
Intuitively, L Sudoku says that “exactly one digit in { 1 , . .
, 9 } can be predicted in the same row/column/box”.
Note that, in CL-STE, the same Sudoku constraints are represented by a CNF with 729 atoms and 8991 clauses, which requires computation on a big matrix in {− 1 , 0 , 1 } 8991 × 729 .
Furthermore, since the values in vector x are not limited to probabilities in NN outputs, we can apply these cardinality constraint losses to an attention matrix, representing additional constraints (not in the original CSP) that should be satisﬁed by the attention.
Example 4 (Constraint Loss on Attention)
In Sudoku problem, an attention matrix A ( r,l ) ∈
R 81 × 81 is computed in the l -th block at the r -th recurrence where A ( r,l ) i,j is a normalized attention weight that can be interpreted as the percentage of attention from cell i to cell j .
The cardinality constraint loss (5) can also be used to deﬁne the following constraint loss
[Equation]
where x = (cid:80) j ( A ( r,l ) : ,j (cid:12) M : ,j ) ; M is the adjacency matrix in { 0 , 1 } 81 × 81 such that M i,j is 1 iff cells i and j are in the same row, column, or box; (cid:12) denotes element-wise multiplication.
Intuitively, the i -th element in x ∈ R 81 denotes the probability of the i -th cell paying attention to its adjacent cells.
Minimizing L attention ( A ( r,l ) ) makes all 81 cells pay attention to their adjacent cells.
Similar to the baseline loss L base , which is the sum of L cross over NN output X ( r,l ) from all recurrent steps and blocks, the total constraint loss L constraint is also accumulated over all NN outputs.
The total loss with constraint loss is L total =
L base + L constraint .
The constraint loss for Sudoku problem is
[Equation]
where α, β are reals in [0 , 1] that are hyper-parameters speciﬁed in Appendix F.
[Table]
5. E XPERIMENTS ON I NJECTING L OGICAL C ONSTRAINTS IN R ECURRENT T RANSFORMER T RAINING
[Table]
[Table]
To evaluate the effects of different logical constraint losses, we trained the L1R32 model on 9k/1k (training/test) RRN dataset and 9k/1k RRN-V dataset for 300 epochs till convergence with and without constraint losses.
Table 2 shows that the same Recurrent Transformer model can further be improved if, in the total loss, we include L attention and/or L Sudoku on each neural network output, where the accuracy is evaluated with 32 or 64 recurrent steps T during testing.
We also observe a better performance gain with L Sudoku than with L attention because the baseline model (trained with L base only) already learns the attention matrices well, as shown in Figure 5 in Appendix B.1.
Besides, when we use 64 recurrences during testing (whereas trained with 32 recurrences), the same Recurrent
Transformer model has bigger improvements on the test accuracy when it is also trained with constraint losses.
Since constraint loss L constraint (accumulated by L Sudoku and L attention ) doesn’t require labels, we could use it for semi-supervised learning tasks.
Table 3 shows that, with only 4k labeled data, adding
L constraint increases the whole board accuracy of 1k test data, which can further be improved by adding additional 4k and 8k unlabeled data along with L constraint .
The shortest path problem is from (Xu et al., 2018) and is about ﬁnding shortest path given a graph and the two endpoints.
The example was used in (Xu et al., 2018; Yang et al., 2020) to demonstrate the effectiveness of semantic/constraint loss on neural network learning.
Our experiment tells that the
Recurrent Transformer achieves higher constraint accuracy, conﬁrming the effects of the injected path constraints on the Recurrent Transformer (Table 7 in Appendix D.1).
C ONCLUSION
With the widespread success of Transformers on system 1 perception tasks, it is intriguing that they could also perform well on system 2 logical reasoning problems.
Adding recurrences to the baseline model already outperforms the existing methods, especially on Visual Sudoku puzzles with large margins (93.5% over enhanced SATNet’s 64.8%), successfully addressing the issue of symbol grounding.
We improve the results further by injecting underlying constraints into Transformer training so that the model can learn with less data, converge faster, and even improve accuracy.
Our experiments show that more recurrences during training tend to yield higher test accuracy and additional recurrences during testing could also help.
The number of attention blocks affects the size and modeling power of the Recurrent Transformer.
More attention heads lead to faster convergence and the accuracy may decrease if the heads are too few to capture different semantic meanings.
A R ECURRENT T RANSFORMER D ETAILS
Figure 4 shows a multi-layer Transformer encoder architecture ( a ) and the Recurrent Transformer architecture in our work ( b ), where every dotted box denotes a self-attention block.
An output layer consists of a layer normalization, a linear layer, and a softmax activation function.
In ( b ), all output layers share the same parameters, while every self-attention block has its own parameters.
The Recurrent Transformer with L self-attention blocks and R recurrences can be formulated as follows:
[Equation]
where H ( r,l ) ∈
R t × d h denotes the hidden embeddings of t input tokens after the l -th (self-attention) block in the r -th recurrent step, block l denotes the l -th Transformer block in the model (i.e., the l -th dotted box in Figure 4 ( b )), layer _ norm denotes layer normalization, · denotes matrix multiplication,
W out ∈ R d h × c is the weight of the output layer for c classes, and X ( r,l ) ∈ [0 , 1] t × c denotes the NN output with the hidden embedding H ( r,l ) .
Each block l is deﬁned on weights W ( l ) K , W ( l ) Q , W ( l ) V , W ( l ) P ∈ R d
h × d h (we describe a single head case for simplicity) and a Multi-Layer Perceptron MLP l with output size d h .
K ( r,l ) = layer _ norm( H ( r,l ) ) · W ( l ) K
[Equation]
,l ) [Equation] layer _ norm [Equation]
[Equation]
V ( r,l ) = layer
_ norm( H ( r,l ) ) · W ( l ) V
[Equation]
block l ( H (
[Equation]
[Equation]
[Equation]
= MLP
[Equation]
layer _ norm( V ∗ ))
+ V ∗
Here, H ( r,l ) , K ( r,l ) , Q ( r,l ) , V ( r,l ) , V ∗ ∈ R t × d h and A ( r,l ) ∈ [0 , 1] t × t .
The parameters are in terms of input vocabulary size ( v ), context size ( t ), number of classes ( c ), hidden embedding size ( d h ), and the hidden layer size ( d MLP ) of MLP l , which is of shape ( d h , d MLP , d h ).
[Table]
As shown in Table 4, the parameters and their counts are shown.
There are a total of 211, parameters.
For SATNet(Wang et al., 2019), the number of parameters for Sudoku are 618,000 total.
This is ( n + 1 + aux ) × m , where n is the number of input variables, aux is the number of auxiliary variables, and m is the rank of the clause matrix.
The (Palm et al., 2018) work has a total of 201, trainable parameters, which come from the row, column, and number embeddings, and the three MLPs used for node updates, message passing, and producing output probabilities.
B M ORE A BLATION S TUDIES ON S UDOKU E XPERIMENTS B.
[Figure]
Without prior knowledge of the Sudoku game, the Recurrent Transformer learns purely from (cid:104) puzzle, solution (cid:105) pairs so that each cell should pay attention to all cells in the same row, column, and 3 × 3 box through attention mechanism.
We trained an L1R32 model with 1 to 4 self-attention heads on SATNet dataset with L base .
Figure 5 ( left ) visualizes the learned attention matrices – they correctly pay attention to each row, column, and box, respectively.
For example, the ﬁrst row of the top-left attention matrix in Figure 5 ( left ) learns purely from data about the 9 atoms to pay attention in constraint (3) where { i, . . .
, j } = { 1 , . . .
, 9 } and d = 1 .
These attentions are clearly separated in different heads if the number of heads is greater or equal to 3 and would otherwise merge.
Figure( right ) compares the whole board accuracy of these models, showing that more attention heads help faster convergence.
The accuracy may decrease if the number of attention heads is too small to capture different semantic meanings.
B. R ECURRENT T RANSFORMER VS . V ANILLA T RANSFORMER
There are two main decision choices in Recurrent Transformer: adding recurrence and applying losses to all blocks at all recurrent steps.
To justify our decision choices, we compared 3 Transformer designs on the textual Sudoku problem under 3 settings.
Figures 6, 7, and 8 show the experimental results on textual Sudoku on SATNet (9k/1k for training/testing), Palm (9k/1k), and Palm (3k/1k) datasets where
• the black line denotes the vanilla Transformer L32R1 with 32 blocks and with the crossentropy loss applied to the ﬁnal output; • the yellow line denotes the Recurrent Transformer L1R32 with a single block, 32 recurrences, and with the cross-entropy loss applied to the last output; • the red line denotes the Recurrent Transformer L1R32 with a single block, 32 recurrences, and with the cross-entropy loss applied to 32 outputs.
We can see that
• (comparing black and yellow lines) adding recurrences allows the model to achieves higher accuracy (especially for harder problems in Palm dataset) with a much fewer parameters in the model (1 block vs. 32 blocks); • (comparing yellow and red lines) applying losses to all blocks makes the Recurrent Transformer model more stable and achieve higher accuracy than the Recurrent Transformer with a single loss; • (comparing Figure 8 with the other 2 ﬁgures) the beneﬁt of recurrence and losses on all blocks is more when the number of data is less.
Figure 8 compares the cell accuracy under the above 3 settings when trained on only 3k Palm data.
In this ﬁgure, using recurrent blocks increases the converged cell accuracy from 17.7% to 46.3%, and applying losses to all blocks further improves the cell accuracy to 76.5%, and it has not converged.
B. S EMI - SUPERVISED LEARNING WITH CONSTRAINT LOSS
In Table 3, we showed how constraint loss helps in semi-supervised setting for textual and visual
Sudoku.
To analyze the effect of constraint loss on more unlabeled data instances, we continued the experiments for both textual and visual Sudoku and recorded the running average of the test accuracy for every 10 epochs in Figures 9 and 10.
We set the batch size to around 64 in experiments in Figure 9 and to around 128 in experiments in
Figure 10.
The batch sizes are slightly adjusted to have integer number split on labeled and unlabeled data in each batch.
To reduce the effect of hyper-parameter tuning on the weights of constraint losses, in all experiments, the weights (cid:104) α, β (cid:105) of the constraint losses L sudoku and L attention are set to (cid:104) 1 , 0 (cid:105) , i.e., only the constraint loss L sudoku is used with ﬁxed weight 1.
We can see that
• adding constraint loss improves the baseline accuracy by a large margin when trained with limited labeled data; • with the help of constraint loss, adding more unlabeled data improves the accuracy in the beginning but the improvement is getting smaller; • if we don’t lower the weight for the constraint loss and keep increasing the number of unlabeled data, adding unlabeled data may lower the accuracy at some point as the signals (i
e., gradients) from constraint loss may overwrite the signals from the labels.
B. C ARDINALITY C ONSTRAINT L OSS VS . CL-STE L OSS
The proposed cadinality constraint loss is different from the constraint loss in CL-STE Yang et al.
We tried the original design of constraint loss in CL-STE but it computes too slow due to the exponential size of CNF used to represent a cardinality constraint.
Thus, we invented a new constraint loss for a general cardinality constraint based on counting discrete values in either NN output or attention matrix, which, to the best of our knowledge, is new to the ﬁeld.
In Table 5, we applied the cross-entropy loss, the CL-STE loss, and the cardinality constraint loss to train the same RRN Palm et al.
(2018) on SATNet
textual
Sudoku dataset Wang et al.
The cross-entropy loss serves as the baseline loss and is used in all four rows in Table 5 during training.
[Table]
Here, R is the number of recurrent steps and is 32 in the RRN model; NumAtom is the number of
Boolean atoms in Sudoku and is 81 × 9 = 729 ; and NumClause is 8991 which is the number of clauses in the CNF for Sudoku.
As we can see, the proposed cardinality constraint loss has the same computation size as the cross-entropy loss, thus almost doesn’t affect the training time.
On the other hand, the constraint loss in CL-STE computes much slower since the computation size is propositional to the number of clauses in a CNF, whose size is exponential to represent a cardinality constraint.
In addition to the cross-entropy loss that is applied to the output from all recurrent steps, if we only apply the CL-STE loss to the output from the ﬁrst recurrent step as done in the CL-STE paper, the training time per epoch is 211s.
Note that batch size is not included in the computation size for simplicity.
All experiments are using a batch size of 16 except for the third row (applying CL-STE loss to the outputs from all recurrent steps).
If we apply the CL-STE loss to all recurrent steps, we have to decrease the batch size by 8 times to ﬁt the GPU memory and the training time per epoch is increased to 3796s.
C D ETAILED C OMPARISON TO SOTA M ODELS FOR V ISUAL S UDOKU
Topan et al.
and Chang et al.
(2020) showed that the original SATNet implementation failed on symbol grounding, which was not apparent due to a data leakage issue where labels for the input symbols were exposed during training time.
Introduced in (Topan et al., 2021), the ungrounded Visual
Sudoku dataset doesn’t include the symbolic label for the given numbers, while the grounded one does.
SATNet fails on ungrounded Visual Sudoku (SATNet-V), achieving 0% accuracy.
To address this, (Topan et al., 2021) alters SATNet by using InfoGAN to cluster the inputs and then jointly trains on
MAXSAT to ground the symbols and solve Sudoku.
They also slightly boost performance by using an additional linear “proofreading” layer, with a ﬁnal ungrounded performance of 64.8% ± 3.0%.
We train improved SATNet from (Topan et al., 2021) using the harder Visual Sudoku dataset RRN-V (9k/1k), and it performs poorly, getting 0% whole board accuracy.
Unlike the SATNet improved in (Topan et al., 2021), our model works out-of-the-box on Visual Sudoku without carefully adjusting the structure.
Yet, it achieves 93.5% on the same dataset, as shown in Table 6.
We also extend (Palm et al., 2018) to include an image embedding layer for Visual Sudoku, denoted as RRN*.
Though it performs well on the textual Sudoku, it fails at symbol grounding for the same boards as images, achieving no better than random performance.
As described in (Topan et al., 2021), the SATNet paper (Wang et al., 2019) erroneously argued a performance bound of 74.8% ( 0 . 992 36 . 2 ) whole board accuracy based on the input classiﬁcation accuracy of LeNet (99.2%) and the average number of givens (36.2).
Our experiments conﬁrm that this is wrong.
We use the same CNN architecture that (Wang et al., 2019) uses, except we change the output size to be the same size as our embedding.
Note that we achieved higher givens cell accuracy with the same CNN (99.77% over 99.2%).
Also, to the best of our knowledge, no simple CNN like this achieves 99.77% accuracy on MNIST if trained for each image separately.
Following the argument above, 92.0% ( = 0 . 9977 36 . 2 ) would have been the theoretical maximum.
However, our model achieves 93.5% whole board accuracy on the SATNet’s Visual Sudoku dataset, which indicates that digit classiﬁcation learning considers the relational information of Sudoku.
Interestingly, our model’s solution board accuracies are consistently higher than whole board accuracies, meaning that even when the givens are misclassiﬁed, the empty cells are ﬁlled with the right values.
This is because our model does not wait to solve after the classiﬁcation of givens is completed; rather, the classiﬁcation and solving are jointly done, which also explains why the accuracy of the given is on par with more advanced vision models which use additional techniques like data augmentation, ensembling, etc.
[Table] Figure 3 compares our models trained with the grounded and the ungrounded versions of the RRN-V dataset.
With the grounded dataset, the input labels are quickly learned, with the model achieving 99.46% givens cell accuracy just after the ﬁrst epoch, while the ungrounded version takes signiﬁcantly longer, achieving 99.36% after 143 epochs.
The grounded dataset appears to help improve the whole board accuracy at ﬁrst but converges at a lower solution board accuracy and whole board accuracy than the model trained with the ungrounded dataset.
With the ungrounded dataset, the model starts to learn the givens and solutions at the same time, at around 85 epochs.
This indicates that digit classiﬁcation is being jointly learned during the solving process.
However, as shown in Figure 3 ( left ), the model has higher solution board accuracy than the whole board accuracy.
This behavior is not surprising since it is enough to solve the non-given cells by just having a sufﬁcient embedding of the given cells without classifying them.
What is more interesting is that even though the model trained on the ungrounded dataset has no access to classiﬁcation labels, it still learns to classify over 99% of the givens.
The large oscillations of whole board accuracy for the model trained on the unground dataset is due to classiﬁcation errors, mostly due to a single given digit being misrecognized.
The graph also shows that for the ungrounded version, the solution board accuracy is consistently higher than the whole board accuracy, meaning that the unground version could still solve correctly even when the classiﬁcation of input digits is wrong.
On the other hand, for the model learned with the grounded dataset, the solution board accuracy overlaps with the whole board accuracy, implying a dependency between the solution board accuracy and givens/classiﬁcation cell accuracy.
The difference in the ﬁnal whole board accuracy between grounded and ungrounded is because the grounded version must optimize the classiﬁcation of the input symbols while the ungrounded version does not.
D M ORE D ETAILS ABOUT O THER E XPERIMENTS D. S HORTEST P ATH Shortest Path in CSP.
A shortest path problem can be viewed as a CSP where
X = { node 1 , . .
, node m , edge 1 , . .
, edge n } denotes all m nodes and n edges in a graph; D = { D 1 , . . .
, D m + n } and D i = { FALSE , TRUE } ; and C is the set of constraints specifying the two end nodes in the graph and that “the selected edges form a path between the end nodes with minimum length”.
The goal is to ﬁnd the solution of this CSP, which represents the solution of the shortest path problem.
Here, node i = TRUE (or edge i = TRUE resp.) represents that node i (or edge i resp.)
is in the shortest path.
Let n 1 , n 2 ∈ { 1 , . . .
, m } denote the indices of the 2 end nodes.
C contains constraints node n 1 = TRUE and node n 2 = TRUE , the following constraint for end nodes i ∈ { n 1 , n 2 } , [Equation]
[Equation]
(6) and the following constraint for non-end nodes i ∈ { 1 , . . .
, m } \ { n 1 , n 2 } in the shortest path (given from label), |{
[Equation]
(7) where { edge i 1 , . .
, edge ik } are the edges connected to node i in the graph.
The ﬁrst constraint says that “each end node should connect to exactly 1 edge in the path” and the second constraint says that “each non-end node in the path should connect to exactly 2 edges in the path”.
Dataset.
We use the shortest path dataset SP4 from (Xu et al., 2018) to illustrate our method where each graph is a 4 × 4 grid with m = 16 nodes and n = 24 edges.
SP4 has 1610 data instances and, as in (Xu et al., 2018), we split the dataset into 60%/20%/20% training/test/validation examples.
In addition, we created a more challenging dataset SP12 where each graph is a 12 × 12 grid with m =nodes and n = 264 edges.
SP12 has 22k data instances, split into 20k/1k/1k training/test/validation examples.
In each problem, two end nodes are randomly picked up, as well as n 3 edges are randomly removed to increase the difﬁculty.
A labeled data instance is (cid:104) t , l (cid:105) where t ∈ { 0 , 1 } m + n such that t i = 1 denotes “node i is a terminal node” when i ≤ m , and denotes “edge ‘ i − m ’ is not removed” when i > m ; and l ∈ { 0 , 1 } n such that l i = 1 denotes “edge i is in the shortest path.”
Figure 11 shows how a Recurrent Transformer is used to solve the shortest path problem in a 4 ×grid with m = 16 nodes and n = 24 edges.
The given information for each logical variable node i ( i ∈ { 1 , . . .
, 16 } ) is a single digit 1 or 0 denoting that node i is an end node or not.
The given information for edge i ( i ∈ { 1 , . . .
, 24 } ) is a single digit 2 or 3 denoting that the edge i is removed or not.
Given a NN output X ( r,l ) ∈ R 40 × 2 for 40 logical variables, we construct the vector v ∈ R 24 such that v i =
X ( r,l ) i +16 , 2 , denoting the probabilities of edge i = TRUE
for i ∈ { 1 , . . .
Let l ∈ { 0 , 1 }denote the label, i.e., l i = 1 iff edge i is in the shortest path.
The cross-entropy loss is deﬁned on v and l .
For the optional constraint loss, let M be the matrix in { 0 , 1 } 16 × 24 such that M i,j =
iff node i is connected with edge j in the graph.
Let c ∈ { 0 , 1 , 2 , 3 , 4 } 16 be M · l .
Intuitively, c i denotes the number of edges in the shortest path containing node i , and c i > 0 means that node i is in the shortest path.
Then, constraints (6) and (7) can be encoded as follows
[Equation]
where the non-zero values in M i, : (cid:12) v ∈ R 24 are the probabilities of edge j = TRUE for all edge j that contains node i .
[Table]
Table 7 compares the constraint accuracy achieved by (i) the baseline Multi-Layer Perceptron (MLP) introduced in (Xu et al., 2018) for the shortest path problem, (ii) the NeurASP method that encodes a path constraint to help train the MLP, (iii) our Recurrent Transformer (L1R32), and (iv) the same
Recurrent Transformer enhanced by the constraint loss L path .
The constraint accuracy are the percentage of the predictions that (i) form a valid path between end nodes, or (ii) do not include removed edges, or (iii) form a shortest path between end nodes.
Table 7 shows that the Recurrent
Transformer signiﬁcantly outperforms the baseline MLP.
Besides, the constraint loss L path further improves the accuracy of the same Recurrent Transformer for predicting a valid path (or a shortest path resp.)
from 84.5% (or 83.5%) to 91.9% (or 91.0%).
Furthermore, we applied the same L1R32 model to the more challenging SP12 dataset.
After 2, epochs of training, we achieved 72.3% accuracy when trained with cross-entropy loss only and 76.0% when trained with both cross-entropy loss and constraint loss L path .
D. N ONOGRAMS
Nonogram ( https://en.wikipedia.org/wiki/Nonogram ) is a game that consists of an initially empty N × N grid representing a binary image, where each cell must take on a value of 0 or 1.
Each row and column have constraints that must be satisﬁed to complete the image successfully.
A constraint for a row/column is a list of numbers, where each number corresponds to contiguous blocks of cells for a row/column.
We created two datasets for 7x7 and 15x15 grids, each has a 9k/1k training/test split.
We use the same Recurrent Transformer model as in previous experiments.
A sample
N × N grid is input as a N 2 long sequence, where each element is a concatenated representation of the row and column constraints associated with the element.
For example, for 15x15 grids, a given cell with the column constraint [1,7,4] and row constraint [2,2,4,1] would have a corresponding sequence element of the concatenation of the two constraint vectors [0,0,1,7,4] and [0,2,2,4,1] (assuming the maximum constraint length is 5 ).
With this simple input encoding only, our Recurrent Transformer
L1R16H4 achieved 97.5% test accuracy on 7x7 grids and 78.3% test accuracy on 15x15 grids.
E C ODE & D ATASETS
The code and the datasets, as well as instructions to reproduce the experiments, are submitted as supplemental material and will be publicly released if the submission gets accepted.
Our Recurrent Transformer implementation is based on Andrej Karpathy’s minGPT repository (https://github.com/karpathy/minGPT) under MIT license.
We extend minGPT (a smaller version of
GPT-3) to allow for full attention, add recurrences, apply additional losses, and alter embeddings.
E. D ATASET C REATED BY O THERS
SATNet and SATNet-V Datasets.
The SATNet and SATNet-V datasets are from SATNet (Wang et al., 2019)
repository ( https://github.com/locuslab/SATNet ) under MIT license.
RRN Dataset.
The RRN dataset is from Recurrent Relational Networks (Palm et al., 2018)
repository ( https://github.com/rasmusbergpalm/recurrent-relational-networks ) where no license information is provided.
MNIST.
We use MNIST images (LeCun et al., 1998) ( http://yann.lecun.com/exdb/ mnist/ ), which are available under the Creative Commons Attribution-Share Alike 3.0 license.
SP4.
We use 4 × 4 shortest path grids from (Xu et al., 2018)
( https://github.com/
UCLA-StarAI/Semantic-Loss ), where no license information is provided.
E. D ATASETS C REATED BY U S
RRN-V. The RRN-V dataset is created using the RRN dataset and MNIST images, similar to the way how SATNet-V dataset was created in SATNet repository.
We construct the visual versions of RRN training/test datasets in which each board cell is represented by a randomly selected MNIST image.
16x
Sudoku.
We generate 16x
, where no license is found.
Though “medium” boards are actually labeled “hard” in their program, we label them “medium” since baseline performance is very high, and the percentage of givens is not low relative to the hardest 9x
Sudoku hardest problems (17-givens, 21% of board).
Nonograms.
For Nonograms dataset, we generate grids using the program "pattern.exe", available under the MIT license available here: https://www.chiark.greenend.org.uk/ ~sgtatham/puzzles/ .
SP12.
We generate 12 × 12 shortest path grids using the answer set solver CLINGO .
F E XPERIMENTAL D ETAILS F. C OMPUTING
All of our experiments were done on Ubuntu 18.04.2 LTS with two 10-cores CPU Intel(R)
Xeon(R)
CPU E5-2640 v4 @ 2.40GHz and four GP
[GeForce GTX 1080].
F. T RAINING D ETAILS
We used a ﬁxed random seed 0 for all experiments.
The values of the weights α and β of the constraint losses L sudoku and L attention are selected from { 0 , 0 . 1 , 0 . 5 , 1 } to achieve the highest training accuracy.
Textual Sudoku.
For the weights (cid:104) α, β (cid:105) of the constraint losses L sudoku and L attention , we used (cid:104) 0 , 0 (cid:105) , (cid:104) 0 , 1 (cid:105) , (cid:104) 1 , 0 (cid:105) , and (cid:104) 0 .
5 (cid:105) in the 4 textual Sudoku experiments in Table 2.
The model structure and hyperparameters are shown in Table 8.
[Table]
Visual Sudoku.
For the weights (cid:104) α, β (cid:105) of the constraint losses L sudoku and L attention , we used (cid:104) 0 , 0 (cid:105) , (cid:104) 0 , 0 .
1 (cid:105) , (cid:104) 1 , 0 (cid:105) , and (cid:104) 1 , 0 .
1 (cid:105) in the 4 Visual Sudoku experiments in Table 2.
The model structure and hyperparameters are shown in Table 9.
16x
Sudoku.
The model structure and hyperparameters used for 16x16 Sudoku experiments are shown in Table 10.
Shortest Path.
The model structure and hyperparameters used for shortest path experiments are shown in Table 11.
MNIST Mapping.
The model structure and hyperparameters used for MNIST Mapping experiments are shown in Table 12.
Nonograms.
A sample 7x7 Nonogram grid and solution is shown in Figure 12.
The model structure and hyperparameters used for Nonograms experiments are shown in Table 13.
[Table] [Table]
[Table]
[Table]
[Table]