From: ICLR.cc/2023/Conference/Paper2445/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper2445/Authors
title: Response to Anonymous Reviewer DaLz
comment: We thank the reviewer for the constructive feedback.

> One should try to compete with at least old-school GRASP and Chaff and see if there is any promise in what they propose, from a practical perspective.

Indeed, neither AsymSAT nor NeuroSAT/DG-DAGRNN is currently competitive compared to these CDCL solvers. However, there are hybrid solvers that incorporate ML in SAT solving, some of which have demonstrated competitive performance, for example, NLocalSAT. However, similar tie-breaking problems exist in the ML part of NLocalSAT and we hope our paper can draw attention to such issues and therefore, also benefit the design of future hybrid SAT solvers.

From: ICLR.cc/2023/Conference/Paper2445/Authors
title: Response to Anonymous Reviewer BQTE
comment: We thank the reviewer for the detailed feedback. Please see below our answers to the questions.

> I don't get the intuition on why in the proposed approach a different GRU has to be used and the initial state has to be the one-hot node encoding.

Using one-hot encoding to represent the node type is a common practice in prior works that use GNNs to process logic circuits, for example [1][2]. We think the GRUs that process the node types should be different from those that take in the latent vectors from the previous iteration of message-passing, therefore, we use two GRUs.

> Why is inferring the node type from the embedding a concern? Isn't it both constant and known a priori?

Yes, node type is already known and is the input of GNN. Here, we are not trying to infer the node types. Prior work [1] maps the embeddings back to the dimension of the node types to use the same GRU that processes the node type, which we think is not necessary. We also update the paper to clarify this in the text.

>  Why using a bi-directional RNN?

Although classic (non-ML) SAT solvers decide variable assignments based on past decisions (therefore, using information from a single direction), we believe supplying more information from both ends do no more harm. After all, the final MLP layers could adjust weights and rely only on the information from a single direction if it has to be that way.

> It is not clear why you expect the latent vectors to contain the necessary information on relationships between variables. Can you elaborate on that?

We expect the latent vectors to contain information about how variables are used in the formula, which is not necessarily the relationship among variables. Such variable usage would be helpful in determining the variable assignments. The relationship among variables are taken care of by the RNN.

> Why validating the sequential decoding on a small set of manually constructed formulas with 3 variables only?

It is already clear from Table 1, without sequential decoding, symmetric problems requiring asymmetric assignments are unsolvable. We think this is a Yes/No problem, the size of the symmetric circuit will not alter the conclusion. Regarding other problems, in Appendix A, we also compare the effectiveness of sequential decoding on SR(10) problems (Table 6). 

> I think that, in principle, the sequential RNN-based decoding layer be applied to NeuroSAT or DG-DAGRNN architectures too, is that correct?

As NeuroSAT uses clustering on the latent vectors to get 0-1 assignments (rather than the output of a network), I'm afraid it won't be as simple as inserting an additional layer. DG-DAGRNN could have such RNN-based decoding layer, but it also needs some kind of a tie-breaker in its unsupervised learning method.

> Can the proposed approach, trained on instances drawn from a certain distribution F(n), generalize to instances with a much larger number of variables from the same distribution (e.g. F(100n))?

We add an appendix in the new version to show results on V(11) to V(15). As the size (#. of logic gates) of V(n) problems doubles as n increases, V(15) is about 128x the size of V(8).

> Can the proposed approach trained of F(n) generalize to instances drawn from a different distribution G(n)? Mixing together training samples from SR(n) and V(n), as done in the experiments reported in Table 5, is not ideal in this sense.

Thanks for the feedback. In our updated version, we add a table (Table 8) to compare the performance of either training on SR+V or solely on SR. There isn’t a significant degradation in the overall performance if trained only on SR problems.

> Why not reporting average and standard deviation of the solution rates?

As larger problems are generally harder, statistics across different sizes may not be a useful indicator. Therefore, we list the solution rates per problem sizes.

> Why not showing the simpler CNF resulting from applying De Morgan rules?

Although applying De Morgan rules seems to produce a simpler result for the XOR circuit we show, in general, the size of CNF will blow up exponentially and therefore, it is not the conventional way of converting a circuit to CNF. Whereas, for Tseitin transformation, the size of CNF is guaranteed to be linear w.r.t to the number of logic gates, and therefore, it becomes the common practice.

> What are 'sequential logic circuits' and 'combinatorial logic networks'?

Sequential logic circuits are those with storage elements. Combinational ones are those without. We also update our paper with an explanation in the text. Thanks!

[1] Amizadeh, Saeed, Sergiy Matusevych, and Markus Weimer. "Learning to solve circuit-SAT: An unsupervised differentiable approach." In ICLR. 2018.

[2] Li, Min, Sadaf Khan, Zhengyuan Shi, Naixing Wang, Huang Yu, and Qiang Xu. "DeepGate: learning neural representations of logic gates." In Proceedings of the 59th ACM/IEEE Design Automation Conference, pp. 667-672. 2022.

From: ICLR.cc/2023/Conference/Paper2445/Authors
title: Response to Anonymous Reviewer zUbq
comment: We thank the reviewer for the valuable feedback.

> Results on generalization to larger problems: Previous works (including NeuroSAT and DG-DAGRNN) have presented results on generalization to problems larger than the ones in the training set. It would be useful to see such results for the proposed method.

In our paper, we demonstrated generalizability in the following way: we trained from SR(3) to SR(10), V(3) to V(8) and tested on V(9) and V(10). The size of V(n) problems doubles as n increases. So V(10) is about 4 times larger than V(8), and it is also significantly larger than SR(40). Therefore, we think this could serve as an indicator of generalizability. Furthermore, we update our paper with new experiment results on  V(11) to V(15) problems, which also show relatively good accuracy.

> The results seem somewhat inconsistent with previous work (e.g., NeuroSAT)

The experiments of the NeuroSAT paper were based on > 10 million samples whose total size is greater than 100 GB as explained by one of the authors [in this Gihub issue]( https://github.com/dselsam/neurosat/issues/2#issuecomment-379064957). Though the number of instances is much smaller in our case, we are comparing relative performances among different models on the same random training and test sets.

> It is also not clear why are the SR problem sizes considered in this work are much smaller than the ones considered by original NeuroSAT paper?

In our case, we are more interested in solving Circuit-SAT problems, therefore, we focus more on experiments with V(n) problems. We update our paper with additional experimental results on SR(20) and SR(40) for your reference. Thanks!

> the link does not work

It turned out that Zenodo misclassified our anonymous upload as spam. We have contacted Zenodo and the access has now been restored. Thanks!

From: ICLR.cc/2023/Conference/Paper2445/Authors
title: Response to Anonymous Reviewer BE2Z
comment: We thank the reviewer for the constructive feedback.

> the targeted category of instances is somewhat hypothetical; in practice, when the number of clauses becomes large, it is rare that two variables could be exactly symmetric

Let's consider a circuit which contains adders. Adders are one of the examples that have plenty of XOR gates, and it is very common to see such symmetry around XOR gates. Even when there are no obvious symmetric variables in the circuits, we can still improve the overall accuracy with our RNN decoding layer. In our paper, Table 2 shows that if we remove the RNN decoding layer, the solution rate significantly reduces.

> the experimental results of NeuroSAT are likely to be wrong. NeuroSAT was originally reported to achieve nearly 100% accuracy for much larger SR(40) instances (see Figure 5 of NeuroSAT paper)

Indeed, Figure 5 in NeuroSAT paper showed nearly 100% accuracy on SR(40), however, that does not invalidate our numbers because the settings are different. The experiments of the NeuroSAT paper were based on > 10 million samples whose total size is greater than 100 GB as explained by one of the authors [in this reference]( https://github.com/dselsam/neurosat/issues/2#issuecomment-379064957). Though the number of instances is much smaller in our case, we are comparing relative performances among different models on the same random training and test sets.

> the synthetic dataset is too small, for instance, only SR(3) - SR(10) are used; in contrast, in the original NeuroSAT paper, SR(10) - SR(40) are used for training, and SR(40) - SR(200) are used in testing.

As explained in the paper, the size of a V(n) problem is significantly larger than that of an SR problem. Also, we update our paper with additional experimental results on SR(20) and SR(40) for your reference.

> even if the exact symmetry sometimes exists, many factors could break it, e.g., random initialization, asynchronous message passing, and numerical errors.

Indeed, there are other factors that can (sometimes “accidentally”) break symmetry. However, we argue that our targeted method for symmetry breaking can be more helpful. For example, experimental results in Appendix B show random initialization may not be as good as our method on the SR(10) problems. 

> on the other hand, the proposed approach does not always break the symmetry issue, given that bi-directional RNN is used. The architecture shown in Figure 2c may be still symmetric.

Although we use bi-directional RNN, the initial states of the RNN are trainable. Training on symmetric problems with asymmetric solutions will break the tie on the two ends of the bi-directional RNN.

From: ICLR.cc/2023/Conference/Paper2445/Authors
title: Revision #1
comment: We would like to thank all the reviewers for their valuable feedback. We have uploaded a revised version. The changes are:

1) A new appendix with supplementary results on larger SR(n) problems and V(n) problems.
2) A few clarifying statements and typo fixes.

We will address the reviewers' questions and concerns in more detail using the individual replies below.

From: ICLR.cc/2023/Conference/Paper2445/Reviewer_BE2Z
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper improves graph neural network (GNN) based SAT solving approach by using a recurrent neural network (RNN) to decode variable assignments. This improvement may help to solve SAT instances with symmetric variables. The key observation is that if two variables are symmetric, their neural embeddings are likely to be identical and thus would be decoded into the same truth value. This can be problematic for certain symmetric constraints like $x \oplus y$. The experimental evaluation on _three synthetic symmetric SAT instances_ show that the proposed approach AsymSAT outperforms NeuroSAT and DG-DAGRNN.

strength_and_weaknesses: Strengths: 
- this paper presents a good motivating example, which helps to understand the main idea quickly
- background knowledge and related work are well-addressed
- multiple datasets have been used in the experimental evaluation

Weaknesses:
- the targeted category of instances is somewhat hypothetical; in practice, when the number of clauses becomes large, it is rare that two variables could be exactly symmetric; in the case one does know the instance under consideration fall into this special category, one may simply set one variable to true and the other to false
- the synthetic dataset is too small, for instance, only SR(3) - SR(10) are used; in contrast, in the original NeuroSAT paper, SR(10) - SR(40) are used for training, and SR(40) - SR(200) are used in testing.
- the experimental results of NeuroSAT are likely to be wrong. NeuroSAT was originally reported to achieve nearly 100% accuracy for much larger SR(40) instances (see Figure 5 of NeuroSAT paper)
- even if the exact symmetry sometimes exists, many factors could break it, e.g., random initialization, asynchronous message passing, and numerical errors (in floating number manipulations); on the other hand, the proposed approach does not always break the symmetry issue, given that bi-directional RNN is used. The architecture shown in Figure 2c may be still symmetric.

clarity,_quality,_novelty_and_reproducibility: This paper presents a simple idea for addressing SAT instances with symmetric variables. The targeted situation is fairly narrow and the solution is also somewhat incremental. The evaluation dataset is synthetic and small, and the evaluation results of NeuroSAT are either wrong or not using a proper configuration. 
summary_of_the_review: The targeted problem is hypothetical, for which there exists a trivial solution. The evaluation is not convincing, and the results of the baseline seem incorrect (i.e., contradicting the original paper). 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper2445/Reviewer_zUbq
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper focuses on addressing the issue of variable dependency within existing GNN-based end-to-end neural SAT solving approaches. Specifically, the paper augments existing GNN-based architecture with an RNN to make the prediction sequential (i.e., where the prediction of variable assignment depends on predictions for previous variables). Experimental evaluation over three datasets shows improved performance compared to the baselines
strength_and_weaknesses: Strengths:
* Interesting new architecture for GNN-based SAT solving that is motivated by the dependency between variables in SAT problems
* Experimental results on three datasets show significant improvement over baselines

Weaknesses:
* My main concerns are about the experimental evaluation:
    * Results on generalization to larger problems: Previous works (including NeuroSAT and DG-DAGRNN) have presented results on generalization to problems larger than the ones in the training set. It would be useful to see such results for the proposed method.
    * The results seem somewhat inconsistent with previous work (e.g., NeuroSAT). While the original NeuroSAT paper does not have the exact same experiment, their results seem much better than the results reported for NeuroSAT in this paper. Specifically, in the original paper, NeuroSAT trained on problems in different sizes SR(U(10,40)) and tested on SR(40) had 70% solved SAT problems and NeuroSAT trained on SR(40) yielded nearly 100% problems solved on SR(40) and even generalized well for problems with SR(80). How is this aligned with the presented results on SR datasets? It is also not clear why are the SR problem sizes considered in this work are much smaller than the ones considered by original NeuroSAT paper?
* Limited technical novelty: it seems the main improvement is the introduction of standard bi-directional decoding later to the architecture. Given the relatively straightforward change, it would be great to at least see analysis with multiple settings, e.g., using attention, using transformers, using dedicated graph-to-sequence approaches.


clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is clear.

- Novelty: the paper has limited technical novelty as noted above.

- Reproducibility: there are missing details in the paper that may be important for reproducibility (e.g., batch sizes, optimizer configuration). The authors claim to provide the source code and datasets, as well as training and testing script, via an anonymized link on zendoo.org however the link does not work ("The record you are trying to access was removed from Zenodo").

summary_of_the_review: The paper presents a simple enhancements to existing GNN-based neural SAT solving approaches by adding an RNN-based prediction of variable assignment. The experimental results show significant improvements. The weaknesses include several concerns on the experimental results and their consistency with previous published results as well as the limited technical novelty of the work.
 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2445/Reviewer_BQTE
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper identifies a major issue with the existing GNN-based approaches to SAT solving, namely the concurrent decoding of truth assignments to the atoms. To circumvent the issue, the authors propose a sequential approach based on recurrent neural networks.
Empirical results validate the effectiveness of the approach.
strength_and_weaknesses: Strenghts:
- The paper is generally well-written
- The proposed technique seems to effectively address a limitation of the existing approaches

Weaknesses (details below):
- I couldn't understand the rationale behind certain architectural choices.
- I think that the evaluation could be improved. 
clarity,_quality,_novelty_and_reproducibility: 
**Presentation**


    "We use two GRUs for forward passes, because the size of a hidden state vector is different from that of the node type vector, whereas Amizadeh et al. (2018) introduced a projection function to project hidden state vectors into the space of node type vectors after each iteration."

I don't get the intuition on why in the proposed approach a different GRU has to be used and the initial state has to be the one-hot node encoding.

    "We argue that the projection could potentially introduce a loss of information"

Why is inferring the node type from the embedding  a concern? Isn't it both constant and known a priori?


---

    "In our AsymSAT, we use a bi-directional recurrent neural network (RNN) to generate sequential predictions on variable assignments, [...]. Sequential prediction mimics classic (non-machine-learning-based) SAT solvers. [...], we expect this RNN layer will be able to learn to predict different variable assignments for the two symmetric variables after training with such examples."

1- Why using a *bi-directional* RNN?

2- The similarities with DPLL-based SAT solvers are limited to the sequential nature of the decisions. In AsymSAT, predictions are made on latent, sub-symbolic encodings of the network structure. It is not clear why you expect the latent vectors to contain the necessary information on relationships between variables. Can you elaborate on that?

---

**Evaluation**

Why validating the sequential decoding on a small set of manually
constructed formulas with 3 variables only? Being the major
contribution I would expect a thorough evaluation on generated
formulas with increasing numbers of variables and dependencies.

---

I think that, in principle, the sequential RNN-based decoding layer be applied to NeuroSAT or DG-DAGRNN architectures too, is that correct?
If yes, why not reporting those results?

---

The generalization capabilities could have been better evaluated in my opinion.
 
1- Can the proposed approach, trained on instances drawn from
a certain distribution F(n), generalize to instances with a much
larger number of variables from the same distribution (e.g. F(100n))? This is shown, to some extent, in Table 4, although a single class is considered (SR) and the test set contains instances that have at most 10 variables, i.e. 2 more than the train instances. The whole point of using GNN-based approaches is to overcome the limits of exact SAT solving, which nowadays easily scales to thousands of variables.


2- Can the proposed approach trained of F(n) generalize to instances
drawn from a different distribution G(n)? Mixing together training
samples from SR(n) and V(n), as done in the experiments reported in Table 5, is not ideal in this sense.

---

Why not reporting average and standard deviation of the solution rates?

---

**Minors**

---

    "with each node representing either a circuit input, a logic gate or a circuit output"

*the* circuit output.

---

How is Figure 1.a different than 1.b? Figure 1.a is already a DAG.
Also, I don't think that applying the Tseitin transformation in this
simple case is doing any good. Why not showing the simpler CNF resulting from
applying De Morgan rules?

A xor B = (A or B) and (~A or ~B)

---

Typo?

    "+Their hidden state vectors"

---

Redundant

    "a projection function to project"

---

    "By default, AIGEN generates sequential logic circuits. We extract the combinational logic networks from the sequential logic circuits."

What are 'sequential logic circuits' and 'combinatorial logic networks'?


summary_of_the_review: This paper offers valuable insights on GNN-based SAT solving and the proposed asymmetrical decoding layer is a welcome contribution.
Overall, I found the writing clear, with some exceptions. To me, the weakest aspect of this contribution is the empirical evaluation, which leaves some important questions unanswered.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2445/Reviewer_DaLz
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper is devoted to machine learning (ML) based methods for solving propositional satisfiability (SAT). In particular, the paper argues that the existing graph neural networks (GNN) based solutions in this domain suffer from a number of serious issues, one of them being their inability to detect variable dependencies, which prevents them from producing correct results on the formulas where this phenomenon is observed. The paper proposes to augment such solvers with a "memory component", which helps them assign a variable in an iterative fashion based on the values known up to the current point for other variables it "depends on". Experimental results obtained on a range of random formulas confirm the advantage of the proposed GNN architecture over the state of the art.
strength_and_weaknesses: I am somewhat ambivalent about this paper as it has pros and cons.

One of the upsides here is that the paper is clearly written. It has a very simple and inspiring motivation that demonstrates where the current solutions fall short. Another plus of this work is that it offers significant performance improvements with respect to the two competitors considered, namely, NeuroSAT and DG-DAGRNN.

Two of my main issues with this paper though include (1) the fact that the contribution seems to be too incremental, which makes it hard to justify the paper's acceptance to ICLR and (2) experimental results still being relatively weak. Let me elaborate on this below.

Regarding the contribution, the paper builds on the existing GNN-based SAT solvers and proposes a way to detect dependencies between variables. The entire idea and the approach setup are described in a single page, which really makes me wonder what the grounds could be for accepting the paper to ICLR.

More on the main idea, I wonder if the authors thought about how it "generalises" (not in the ML way) to other kinds of variable dependencies besides XOR clauses. The thing is practical CNF formulas are full of complicated variable dependencies, which make these formulas far beyond the reach of the existing GNN-based solvers. This way, tackling XOR specifically looks too limited. Granted the authors describe XOR as an example, but they still do not discuss any other kinds of problems that would benefit from their solution.

As for the experimental results, I believe it is not enough to compare a new GNN-based solver against prior solvers of the same type. One should try to compete with at least old-school GRASP and Chaff and see if there is any promise in what they propose, from a practical perspective. Here, I do not even mention a potential comparison against MiniSat, which is an old but still efficient clause learning based SAT solver. Also, considering only a couple of families of random formulas is quite disappointing too as those are hardly useful in practice.
clarity,_quality,_novelty_and_reproducibility: Presentation-wise the paper is nicely written. The discussion is clear and easy to follow even for a non-ML person like me. I like how the paper is motivated with a running example of XOR clauses and aims at solving this specific problem. To the best of my understanding, the proposed solution is novel. One major issue with the contribution is that it seems quite incremental and thus limited. In terms of reproducibility, the paper offers a repository with the source code and the dataset used to facilitate the reproducibility of the results.
summary_of_the_review: Given the above and despite my positive view of some of the aspects of the paper, I still believe that it does not offer contributions that would be sufficient to warrant a publication at a flagship AI/ML conference like ICLR.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough