Paper: Revisiting GNNs for Boolean Satisfiability (ICLR-2024)

Reviewer comment (Experiments):
Neither the samples nor the decimation techniques are subjected to ablation experiments.

Author response:
Thank you for pointing this out, we extended the table with experiments (Table 1) so that the impact of decimation and sampling can be separated. Concretely, there is now a row for runs with 32 random initialization samples and no decimation. These runs will have the same number of samples as the runs in the first row with 16 samples in the first pass and 1 sample for each decimated problem in the second pass. One can, therefore, see the effect of decimation which is simply the difference between the number of solved problems in total between these two rows. The effect of sampling can be seen by comparing to the second row for runs with just 1 sample and no decimation.

Reviewer comment (Writing):
The last sentence of the first paragraph in the introduction is not particularly convincing.

Author response:
You probably meant this sentence: “The presented ideas should be easily generalizable to other combinatorial domains where these approximation algorithms could be applied.” We believe this should be the case, because SDP relaxation is a very general technique, and relaxations for many other combinatorial problems can be found in the literature. If the GNN is able to discover this relaxation, it should be, together with the ideas we presented, applicable to these other combinatorial domains. Nevertheless, we removed this sentence, as it is not essential and we did not have a space to expand on it.

Reviewer comment (Theory):
The sampling technique seems to enhance model accuracy solely by initializing values across multiple embeddings, and its relationship with SDP appears weak.

Author response:
The initial embeddings are random unit vectors. In Appendix A.5 we included results from other experiments which show that the iterations of the GNN can be viewed as optimization steps of the embeddings, optimizing the SDP objective value. Therefore, the intuition is that if these vectors are initialized in different positions, they can converge to a different local optimum. This is how the SDP solver behaves. Some initializations may be more lucky.

Reviewer comment (Reproducibility):
Could you please clarify the nature of the initial embeddings? Are they generated randomly?

Author response:
The initial embeddings are random unit vectors. In Appendix A.5 we included results from other experiments which show that the iterations of the GNN can be viewed as optimization steps of the embeddings, optimizing the SDP objective value. Therefore, the intuition is that if these vectors are initialized in different positions, they can converge to a different local optimum. This is how the SDP solver behaves. Some initializations may be more lucky.

Reviewer comment (Experiments):
According to Section 5, the training instances are of very small scale. For the generated, random SAT instances, the number of variables are up to 40 variables. In fact, existing local search SAT algorithms are able to solve random satisfiable instances around phase-transition threshold with thousands of variables very efficiently. Hence, solving random instances with up to 40 variables are quite trivial.

Author response:
Thank you for your comments. We believe that there was a misunderstanding of the purpose of our paper. We were not proposing a method that would be able to compete with mainstream SAT solvers. Our aim was to shed light on the process by which NeuroSAT learns to solve SAT instances. We clearly state that in the introduction: “Although these learned solutions lag far behind the solvers used in practice
in terms of the size of problems they are able to solve, it is desirable to understand the process by which they arrive at their outputs.”

We demonstrated the similarities of the trained NeuroSAT and two well-known approximation algorithms. Using these connections we proposed several improvements, one of which enables us to train NeuroSAT in order of magnitude shorter time, which can enable quicker experimentation with NeuroSAT.

Many papers presented at this conference have similar flavors and show the behavior of the trained GNN only on toy examples (e.g. https://openreview.net/forum?id=rJxbJeHFPS , https://openreview.net/forum?id=hhvkdRdWt1F ).  These papers show how GNN are able to learn simple algorithms on small instances. It is clear that GNNs trained end-to-end are currently not able to compete with manually designed solvers. This kind of work fits well with ICLR as the conference is focused mainly on learning representations.

Reviewer comment (Experiments):
After reading Appendix A.3.2, it seems that the authors also do not introduce the number of variables for those generated, structured instances (i.e., those instances generated from the domains of Latin squares, Sudoku, and logical circuits). Actually, it is widely recognized that modern CDCL SAT solvers can solve structured SAT instances with tens of thousands of variables. Could you please claim the numbers of variables for those generated, structured instances?

Author response:
Regarding the number of variables of the structured formulas, they were included in Table 1, in the third column.

Reviewer comment (Theory):
It seems that your proposed method can only handle satisfiable instances. Could you discuss the behavior of your proposed method when dealing with unsatisfiable instances?

Author response:
Regarding unsatisfiable instances, we hoped it was clear from the text that our method cannot produce any guarantee of unsatisfiability. This is also the case for local search but it is still used in practice for instance within Cadical or Kissat. We also mentioned in the text that the solvers obtained from these approximation algorithms are incomplete. To make it clear, we now state it explicitly at the end of section 5.

Reviewer comment (Experiments):
The authors only compare their proposed method with NeuroSAT. However, CDCL solvers stand for the current state of the art in SAT solving. As a submission to a top-tier conference, lack a comparison against the real state of the art is unacceptable.

Author response:
Thank you for your comments. We believe that there was a misunderstanding of the purpose of our paper. We were not proposing a method that would be able to compete with mainstream SAT solvers. Our aim was to shed light on the process by which NeuroSAT learns to solve SAT instances. We clearly state that in the introduction: “Although these learned solutions lag far behind the solvers used in practice
in terms of the size of problems they are able to solve, it is desirable to understand the process by which they arrive at their outputs.”

We demonstrated the similarities of the trained NeuroSAT and two well-known approximation algorithms. Using these connections we proposed several improvements, one of which enables us to train NeuroSAT in order of magnitude shorter time, which can enable quicker experimentation with NeuroSAT.

Regarding the comment: “As a submission to a top-tier conference, lack of a comparison against the real state of the art is unacceptable.” Please note that Selsam et al. published their paper about NeuroSAT at this conference and it was already cited more than 300 times. We believe that it is paramount to understand what kind of algorithms are GNNs able to learn, irrespective of how practically scalable these learned algorithms are. 

Many papers presented at this conference have similar flavors and show the behavior of the trained GNN only on toy examples (e.g. https://openreview.net/forum?id=rJxbJeHFPS , https://openreview.net/forum?id=hhvkdRdWt1F ).  These papers show how GNN are able to learn simple algorithms on small instances. It is clear that GNNs trained end-to-end are currently not able to compete with manually designed solvers. This kind of work fits well with ICLR as the conference is focused mainly on learning representations.

Reviewer comment (Presentation):
The sentence "In Figure 4 in the Appendix, we recapitulate their visualization of embeddings with UMAP instead of PCA" is problematic because the connection between the figure and the following text is unclear. If the figure is important for understanding the main content, it should be moved to the main text; otherwise, the sentence should be removed. Additionally, it is unclear why UMAP was used instead of PCA.

Author response:
Thank you for pointing this out. We included a simplified version of this figure into the main text. The figure shows how the embeddings evolve during the message-passing process. For satisfiable formulas, these embeddings create two well-separated clusters at the end. The text explains that it is possible to extract the solution of the formula by running a clustering algorithm on the embeddings produced by the last step of the MP process (for SAT formulas there should be 2 distinct clusters). One cluster will correspond to literals which will take the value TRUE and the other to literals which will take the value FALSE. We used UMAP as the clusters were more distinct in comparison to the visualization obtained by PCA.

Reviewer comment (Experiments):
It is not clear whether sampling multiple initializations and decimation are two orthogonal improvements. If so, a separate empirical evaluation for each should be provided.

Author response:
Thank you for pointing this out, we extended the table with experiments (Table 1) so that the impact of decimation and sampling can be separated. Concretely, there is now a row for runs with 32 random initialization samples and no decimation. These runs will have the same number of samples as the runs in the first row with 16 samples in the first pass and 1 sample for each decimated problem in the second pass. One can, therefore, see the effect of decimation which is simply the difference between the number of solved problems in total between these two rows. The effect of sampling only can be seen by comparing to the second row for runs with just 1 sample and no decimation.

Reviewer comment (Experiments):
The experimental section does not address whether the (augmented) NeuroSAT architecture is competitive with other end-to-end approaches in some settings. Furthermore, it is unclear if curriculum learning could push predictive accuracy over 85% using a more expressive model, and whether the augmented method could better generalize to larger problems compared to the original NeuroSAT.

Author response:
Our main goal in this paper was to shed light on the inner workings of trained NeuroSAT. Even though we show practical improvements, their main purpose was to support our claim that the trained GNN behaves as an SDP solver (which can be initialized with different values). Decimation on the other hand support the connection to the BP. The practical benefit of the curriculum is that one can iterate experiments with NeuroSAT faster. 
We are also interested in understanding how other end-to-end approaches are able to predict satisfiability (i.e. whether they are also doing something similar to an SDP solver) but we leave this direction for future work.

Reviewer comment (Experiments):
Reporting the inference time of the standard versus decimation approaches is necessary for evaluation. It is also unclear why only 2 passes of decimation were evaluated, and what would happen if more passes were used.

Author response:
In Appendix A.6, we included a table with one more decimation step. The fourth pass did not solve any other problem in our test datasets. We will study this more closely in future work.  
We did not optimize the runtime of the decimation procedure. We decimate and obtain a new prediction for each sample separately. If we have 16 samples for which we obtain predictions in parallel and for each of the 16 decimated formulas we produce 1 sample, then this should take approximately 16x more time than without the decimation. Decimation could probably be parallelized but as these end-to-end approaches are currently not used in practice we did not explore such optimizations.

Reviewer comment (Experiments):
If multiple initializations and decimation are orthogonal improvements, ablation results for the two should be provided.

Author response:
Thank you for pointing this out, we extended the table with experiments (Table 1) so that the impact of decimation and sampling can be separated. Concretely, there is now a row for runs with 32 random initialization samples and no decimation. These runs will have the same number of samples as the runs in the first row with 16 samples in the first pass and 1 sample for each decimated problem in the second pass. One can, therefore, see the effect of decimation which is simply the difference between the number of solved problems in total between these two rows. The effect of sampling only can be seen by comparing to the second row for runs with just 1 sample and no decimation.

Reviewer comment (Experiments):
A more thorough evaluation of decimation should be provided, including experiments with more than 2 passes.

Author response:
In Appendix A.6, we included a table with one more decimation step. The fourth pass did not solve any other problem in our test datasets. We will study this more closely in future work.  
We did not optimize the runtime of the decimation procedure. We decimate and obtain a new prediction for each sample separately. If we have 16 samples for which we obtain predictions in parallel and for each of the 16 decimated formulas we produce 1 sample, then this should take approximately 16x more time than without the decimation. Decimation could probably be parallelized but as these end-to-end approaches are currently not used in practice we did not explore such optimizations.

Reviewer comment (Evaluation):
The runtime cost of the proposed approach compared to standard NeuroSAT should be reported.

Author response:
In Appendix A.6, we included a table with one more decimation step. The fourth pass did not solve any other problem in our test datasets. We will study this more closely in future work.  
We did not optimize the runtime of the decimation procedure. We decimate and obtain a new prediction for each sample separately. If we have 16 samples for which we obtain predictions in parallel and for each of the 16 decimated formulas we produce 1 sample, then this should take approximately 16x more time than without the decimation. Decimation could probably be parallelized but as these end-to-end approaches are currently not used in practice we did not explore such optimizations.

Reviewer comment (Experiments):
It should be clarified whether the augmented NeuroSAT method better generalizes to larger instances.

Author response:
The curriculum learning by itself does not seem to push the predictive accuracy. We saw that NeuraSAT with the curriculum can sometimes reach an accuracy of 87% (on the original test set) but we guess that NeuroSAT would also be able to reach this accuracy if trained long enough and with proper hyperparameters. We saw no difference in generalization to larger problems between the original NeuroSAT and the one trained with the curriculum.