Paper: NeuroBack: Improving CDCL SAT Solving using Graph Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
Regarding the baseline competitor. As described in the experiment part, the baseline competitor is kissat. Although kissat can be regarded as the latest breakthrough in the community of SAT solving, lots of kissat’s variants have been proposed since the introduction of kissat (as can be observed in the recent editions of SAT competitions). However, the authors only compare their solver with vanilla kissat. As a submission to a top-tier conference, this is not that thorough.

Author response:
As the latest significant breakthrough in SAT solving, Kissat represents a strong and widely recognized standard in the community.  By comparing our solver to the most recent well-established SAT solver, we aimed to provide a clear and straightforward evaluation of NeuroBack's capabilities. We acknowledge that numerous variants of Kissat have emerged since its introduction, many of which have demonstrated notable performance in recent SAT competitions. However, these variants often include specific optimizations or modifications tailored to competition settings, which may not generalize well across broader settings. Our objective was to evaluate the effectiveness of our neural approach in a more generalized context, which made comparing it with the vanilla Kissat more suitable than conducting an exhaustive comparison with each Kissat variant. Furthermore, we examined the top three Kissat variants in the SAT competitions over the last three years, and found that all of their optimization techniques are orthogonal to NeuroBack's phase initialization technique. Given NeuroBack's already demonstrated efficiency gains with vanilla Kissat, we anticipate it will also positively impact these Kissat variants. Our future research efforts will focus on empirically validating this hypothesis.

Reviewer comment (Experiments):
Regarding the evaluation on SAT Competition 2022’s datasets. Actually, as a submission in SAT solving, the tradition is to evaluate their proposed solver on the dataset from the latest edition of SAT Competition. In fact, as of the submission deadline of ICLR, the latest edition of SAT Competition is SAT Competition 2023. According to the official website of SAT Competition 2023 (https://satcompetition.github.io/2023/), the dataset of SAT Competition 2023 was published in July, 2023. Since the submission deadline of ICLR is due on September 28. 2023, there left two months for the authors to conduct the comparative experiments on the dataset of SAT Competition 2023. Hence, such lack of experiments is indeed a minus.

Author response:
Thank you for pointing this out. We have evaluated NeuroBack on SATCOMP-2023. Detailed results and analysis are now included and can be seen as supplementary material to the paper. In sum, NeuroBack-Kissat performs better than the baseline solver Kissat in both SATCOMP-2022 and SATCOMP-2023. Specifically, on SATCOMP-2023 problems, Kissat solved 198 problems and NeuroBack-Kissat 204, providing an improvement of 3.0%. 

In addition, since the default setting of Kissat is to simply initialize the phase of each variable as true, to further study NeuroBack effectiveness, we implemented an additional baseline solver, named Random-Kissat, which initializes the phase of each variable as either true or false at random. Detailed results and analysis are shown in the supplementary material of the paper. In summary, Random-Kissat performs better than Kissat in SATCOMP-2022, but worse in SATCOMP-2023; meanwhile, NeuroBack-Kissat performs better than Random-Kissat in both SATCOMP-2022 and SATCOMP-2023. More specifically, in SATCOMP-2022 problems, Random-Kissat solved 197 problems, whereas NeuroBack-Kissat solved 203, marking a 3.0% improvement. In SATCOMP-2023 problems, Random-Kissat solved 190 problems and NeuroBack-Kissat solved 204, demonstrating an improvement of 7.4%. 

Overall, these results suggest that the phase initialization provided by NeuroBack outperforms both the default and random phase initializations in Kissat, showing more effectiveness in solving SATCOMP-2022 and SATCOMP-2023 problems.

Reviewer comment (Experiments):
The paper says nothing about the usability of this heuristic for unsatisfiable instances. There are no backbones in those but I presume some variable phases may still be more useful in practice than the others.

Author response:
Thank you for bringing this up! In our experiments, we did not filter the benchmarks based on their satisfiability. Therefore, the 308 instances certainly contain unsatisfiable instances. Upon detailed analysis, we discovered that there are 106 unsatisfiable instances and 99 satisfiable instances that are solved by either Kissat or NeuroBack-Kissat. 

For the 106 solved unsatisfiable instances, NeuroBack-Kissat outperformed Kissat in 63 cases (59.4%) while Kissat outperformed NeuroBack-Kissat in only 35 instances (33.0%).  For the 99 solved satisfiable instances, NeuroBack-Kissat outperformed Kissat in 57 instances (57.6%), while Kissat outperformed NeuroBack-Kissat in 40 instances (40.4%). Interestingly, our analysis revealed that while NeuroBack-Kissat showed a slightly higher improvement rate in unsatisfiable instances compared to satisfiable ones (59.4% vs 57.6%), the extent of improvement was more significant in satisfiable instances. On average, NeuroBack-Kissat enhanced the performance of satisfiable instances by 62.4%, compared to an average improvement of only 9.0% in unsatisfiable instances.

The experimental results demonstrate that NeuroBack's predicted variable phases can enhance the efficiency in solving unsatisfiable instances. However, as expected, NeuroBack shows a more pronounced improvement in solving satisfiable instances. We will add these results in the next version of our paper. 

**Further Response to Concern 1.**

1) a discussion on how and why the approach helps solving unsatisfiable instances

In fact, NeuroBack predicts phases of all variables appearing in the majority of the satisfying assignments, not only the backbone phases. We certainly agree that how the predicted phases help in solving unsatisfiable instances is not entirely intuitive. We really appreciate your insightful question.  Here's our clarification: For unsatisfiable instances, NeuroBack's phase predictions can aid in directing the search towards the unsatisfiable part of the search space. While NeuroBack cannot satisfy all components of a given SAT problem, it may predict phases that satisfy certain components, thereby allowing the solver to concentrate on the unsatisfiable part. Furthermore, in modern SAT solvers like Kissat [1], an assignment that falsifies the fewer clauses is often preferred in the searching loop, allowing the solver to specifically target the unsatisfiable portions of the clause set. Consequently, the phases predicted by NeuroBack can facilitate identifying an assignment that reduces clause falsification, thereby enhancing solving unsatisfiable instances.

2) comment on the performance on SAT and UNSAT instances and differentiate the two

The results shown in the previous comment indicate that NeuroBack achieves more notable success in solving satisfiable instances compared to unsatisfiable ones. This distinction stems from the inherent nature of these problems. In satisfiable instances, a complete satisfying assignment exists, where each variable is assigned a phase leading to a solution. Conversely, in unsatisfiable instances, only partial satisfying assignments exist, with phases assigned to just a subset of variables. Consequently, the phases predicted by NeuroBack have a generally greater impact in resolving satisfiable instances. This is because, for these instances, the predicted phases can contribute directly to finding a satisfying assignment. In contrast, for unsatisfiable instances, the utility of predicted phases is somewhat restricted to identifying partial solutions or refining the search scope. This fundamental difference in the nature of satisfiable versus unsatisfiable problems underpins the varying degrees of effectiveness observed in NeuroBack's performance.

Reviewer comment (Experiments):
Despite the claimed experimental results, nothing is shown for the unsatisfiable instances. This and the point above can be joined together.

Author response:
Thank you for bringing this up! In our experiments, we did not filter the benchmarks based on their satisfiability. Therefore, the 308 instances certainly contain unsatisfiable instances. Upon detailed analysis, we discovered that there are 106 unsatisfiable instances and 99 satisfiable instances that are solved by either Kissat or NeuroBack-Kissat. 

For the 106 solved unsatisfiable instances, NeuroBack-Kissat outperformed Kissat in 63 cases (59.4%) while Kissat outperformed NeuroBack-Kissat in only 35 instances (33.0%).  For the 99 solved satisfiable instances, NeuroBack-Kissat outperformed Kissat in 57 instances (57.6%), while Kissat outperformed NeuroBack-Kissat in 40 instances (40.4%). Interestingly, our analysis revealed that while NeuroBack-Kissat showed a slightly higher improvement rate in unsatisfiable instances compared to satisfiable ones (59.4% vs 57.6%), the extent of improvement was more significant in satisfiable instances. On average, NeuroBack-Kissat enhanced the performance of satisfiable instances by 62.4%, compared to an average improvement of only 9.0% in unsatisfiable instances.

The experimental results demonstrate that NeuroBack's predicted variable phases can enhance the efficiency in solving unsatisfiable instances. However, as expected, NeuroBack shows a more pronounced improvement in solving satisfiable instances. We will add these results in the next version of our paper. 

**Further Response to Concern 1.**

1) a discussion on how and why the approach helps solving unsatisfiable instances

In fact, NeuroBack predicts phases of all variables appearing in the majority of the satisfying assignments, not only the backbone phases. We certainly agree that how the predicted phases help in solving unsatisfiable instances is not entirely intuitive. We really appreciate your insightful question.  Here's our clarification: For unsatisfiable instances, NeuroBack's phase predictions can aid in directing the search towards the unsatisfiable part of the search space. While NeuroBack cannot satisfy all components of a given SAT problem, it may predict phases that satisfy certain components, thereby allowing the solver to concentrate on the unsatisfiable part. Furthermore, in modern SAT solvers like Kissat [1], an assignment that falsifies the fewer clauses is often preferred in the searching loop, allowing the solver to specifically target the unsatisfiable portions of the clause set. Consequently, the phases predicted by NeuroBack can facilitate identifying an assignment that reduces clause falsification, thereby enhancing solving unsatisfiable instances.

2) comment on the performance on SAT and UNSAT instances and differentiate the two

The results shown in the previous comment indicate that NeuroBack achieves more notable success in solving satisfiable instances compared to unsatisfiable ones. This distinction stems from the inherent nature of these problems. In satisfiable instances, a complete satisfying assignment exists, where each variable is assigned a phase leading to a solution. Conversely, in unsatisfiable instances, only partial satisfying assignments exist, with phases assigned to just a subset of variables. Consequently, the phases predicted by NeuroBack have a generally greater impact in resolving satisfiable instances. This is because, for these instances, the predicted phases can contribute directly to finding a satisfying assignment. In contrast, for unsatisfiable instances, the utility of predicted phases is somewhat restricted to identifying partial solutions or refining the search scope. This fundamental difference in the nature of satisfiable versus unsatisfiable problems underpins the varying degrees of effectiveness observed in NeuroBack's performance.

Reviewer comment (Evaluation):
Although Neuroback+Kissat solves 10 more instances out of 308, there are 92 more (308 + 92 = 400) where Neuroback fails to do anything useful. Hence, the proposed solver configuration must clearly lose to Kissat on those 92 instances as it spends additional time with no effect. If this understanding of mine is correct then the results aren't so positive.

Author response:
Given that the NeuroBack model has a relatively small size, the primary memory consumption during model inference stems from the size of the graph representations of SAT formulas. To effectively manage memory contention, we have adopted a straightforward yet efficient approach: if the size of a SAT formula surpasses a predetermined size threshold, we default to using Kissat for solving the formula. This size checking is very quick, typically taking a few milliseconds. Therefore, the overhead for this process in the remaining 82 instances is minimal and can be considered negligible.

**Further Response to Concern 2.**

Thank you for pointing this out! In our experimental setup, which includes a machine equipped with 256GB of memory running 64 solver instances in parallel, we have conservatively set the SAT formula size threshold at 135 MB. This ensures that the memory usage of each solver instance does not exceed our specified memory threshold of 10GB. This threshold setting is based on our practical experience. Increasing this threshold could potentially lead to memory contention issues. Users might choose to adjust the formula size threshold based on their machine’s memory capacity. Alternatively, they might simply establish a memory threshold for each solver instance based on their machine's memory capacity and allow model inference to proceed until this threshold is reached, which typically incurs an overhead of no more than a few seconds. We will add this clarification to the next version of our paper.

Reviewer comment (Experiments):
Can you comment on the use of this heuristic for unsatisfiable instances? Have you tried this? If yes, what is the performance compared to Kissat? If not, why?

Author response:
Thank you for bringing this up! In our experiments, we did not filter the benchmarks based on their satisfiability. Therefore, the 308 instances certainly contain unsatisfiable instances. Upon detailed analysis, we discovered that there are 106 unsatisfiable instances and 99 satisfiable instances that are solved by either Kissat or NeuroBack-Kissat. 

For the 106 solved unsatisfiable instances, NeuroBack-Kissat outperformed Kissat in 63 cases (59.4%) while Kissat outperformed NeuroBack-Kissat in only 35 instances (33.0%).  For the 99 solved satisfiable instances, NeuroBack-Kissat outperformed Kissat in 57 instances (57.6%), while Kissat outperformed NeuroBack-Kissat in 40 instances (40.4%). Interestingly, our analysis revealed that while NeuroBack-Kissat showed a slightly higher improvement rate in unsatisfiable instances compared to satisfiable ones (59.4% vs 57.6%), the extent of improvement was more significant in satisfiable instances. On average, NeuroBack-Kissat enhanced the performance of satisfiable instances by 62.4%, compared to an average improvement of only 9.0% in unsatisfiable instances.

The experimental results demonstrate that NeuroBack's predicted variable phases can enhance the efficiency in solving unsatisfiable instances. However, as expected, NeuroBack shows a more pronounced improvement in solving satisfiable instances. We will add these results in the next version of our paper. 

**Further Response to Concern 1.**

1) a discussion on how and why the approach helps solving unsatisfiable instances

In fact, NeuroBack predicts phases of all variables appearing in the majority of the satisfying assignments, not only the backbone phases. We certainly agree that how the predicted phases help in solving unsatisfiable instances is not entirely intuitive. We really appreciate your insightful question.  Here's our clarification: For unsatisfiable instances, NeuroBack's phase predictions can aid in directing the search towards the unsatisfiable part of the search space. While NeuroBack cannot satisfy all components of a given SAT problem, it may predict phases that satisfy certain components, thereby allowing the solver to concentrate on the unsatisfiable part. Furthermore, in modern SAT solvers like Kissat [1], an assignment that falsifies the fewer clauses is often preferred in the searching loop, allowing the solver to specifically target the unsatisfiable portions of the clause set. Consequently, the phases predicted by NeuroBack can facilitate identifying an assignment that reduces clause falsification, thereby enhancing solving unsatisfiable instances.

2) comment on the performance on SAT and UNSAT instances and differentiate the two

The results shown in the previous comment indicate that NeuroBack achieves more notable success in solving satisfiable instances compared to unsatisfiable ones. This distinction stems from the inherent nature of these problems. In satisfiable instances, a complete satisfying assignment exists, where each variable is assigned a phase leading to a solution. Conversely, in unsatisfiable instances, only partial satisfying assignments exist, with phases assigned to just a subset of variables. Consequently, the phases predicted by NeuroBack have a generally greater impact in resolving satisfiable instances. This is because, for these instances, the predicted phases can contribute directly to finding a satisfying assignment. In contrast, for unsatisfiable instances, the utility of predicted phases is somewhat restricted to identifying partial solutions or refining the search scope. This fundamental difference in the nature of satisfiable versus unsatisfiable problems underpins the varying degrees of effectiveness observed in NeuroBack's performance.

Reviewer comment (Evaluation):
Can you comment on losing on 82 instances to Kissat if we consider all the 400 instances in the benchmark set? Am I missing anything?

Author response:
Given that the NeuroBack model has a relatively small size, the primary memory consumption during model inference stems from the size of the graph representations of SAT formulas. To effectively manage memory contention, we have adopted a straightforward yet efficient approach: if the size of a SAT formula surpasses a predetermined size threshold, we default to using Kissat for solving the formula. This size checking is very quick, typically taking a few milliseconds. Therefore, the overhead for this process in the remaining 82 instances is minimal and can be considered negligible.

**Further Response to Concern 2.**

Thank you for pointing this out! In our experimental setup, which includes a machine equipped with 256GB of memory running 64 solver instances in parallel, we have conservatively set the SAT formula size threshold at 135 MB. This ensures that the memory usage of each solver instance does not exceed our specified memory threshold of 10GB. This threshold setting is based on our practical experience. Increasing this threshold could potentially lead to memory contention issues. Users might choose to adjust the formula size threshold based on their machine’s memory capacity. Alternatively, they might simply establish a memory threshold for each solver instance based on their machine's memory capacity and allow model inference to proceed until this threshold is reached, which typically incurs an overhead of no more than a few seconds. We will add this clarification to the next version of our paper.

Reviewer comment (Experiments):
While training on backdoor variables yields a good predictor for the value of all variables, why this approach works is still a bit mysterious. It would be helpful to have more experiments showing that initialization is actually better than a random assignment. Therefore I would like to see some behavior studies of NeuroBack on different kinds of benchmarks. Possible ways include comparing the distance from this initial assignment given by NeuroBack with the nearest solution, or evaluating the prediction accuracy of the value of a specific variable given by NeuroBack with the majority value of this variable in all solutions (this requires listing all solutions of a formula, which is only doable on small formulas).

Author response:
**Concern 1: More experiments comparing neural initialization to random assignment**

To address the reviewer's concern, we implemented a random initialization solver based on Kissat, named Random-Kissat, which initializes the phase of each variable to true or false randomly. We also included benchmarks from the most recent SAT competition, SATCOMP-2023, as "different kinds of benchmarks'' requested by the reviewer. Detailed results and analysis are now included and can be viewed in the supplementary material of the paper. 

In summary, Random-Kissat performs better than Kissat in SATCOMP-2022, but worse in SATCOMP-2023; meanwhile, NeuroBack-Kissat performs better than Random-Kissat in both SATCOMP-2022 and SATCOMP-2023. More specifically, in SATCOMP-2022 problems, Random-Kissat solved 197 problems, whereas NeuroBack-Kissat solved 203, making a 3.0% improvement. In SATCOMP-2023 problems, Random-Kissat solved 190 problems and NeuroBack-Kissat solved 204, demonstrating an improvement of 7.4%. Overall, these results indicate that the phase initialization provided by NeuroBack is more effective than random assignment in both SATCOMP-2022 and SATCOMP-2023 problems.

Reviewer comment (Novelty):
I am not quite convinced by the claimed advantage that NeuroBack only needs to be called once. This is a trivial property for any initialization approach for SAT solving. I believe that a more interactive collaboration between GNN and SAT solvers can further improve the performance of STA solvers, even if GNN needs to be called multiple times (correct me if I am wrong). I would be happy to change the evaluation if my concerns can be addressed.

Author response:
**Concern 2: not convinced by the claim that NeuroBack only needs to be called once**

We partially agree with the reviewer about the benefits of interactive collaboration between GNN and SAT solvers, where the GNN refines its predictions based on the current state of SAT solving, potentially improving neural prediction accuracy. However, it is crucial to consider the potential performance bottleneck arising from such interactions. The increased demand for GPU resources in parallel SAT solving, a prevalent approach in industrial applications for tackling complex problems, could significantly slow down the overall solving process. As more solver runs are executed in parallel, the competition for GPU resources intensifies. Lacking GPU resources may become a major bottleneck in achieving the expected parallel performance. Notably, if the GNN already demonstrates high prediction accuracy before the start of SAT solving, the frequent use of GPU resources for minor adjustments in predictions through online inferences would be less efficient than relying on the initial offline predictions.

The reviewer's concern arises from the application of NeuroBack's phase prediction. 
As outlined in the discussion and future work sections, our current approach only focuses on applying NeuroBack's phase predictions to initialize the variable phases in SAT solvers. However, there exists various approaches for applying NeuroBack predictions, either statically or dynamically, each potentially enhancing the SAT solving efficiency. For instance, NeuroBack phase predictions could be leveraged dynamically during rephasing to adjust or diversify the exploration of the search space. In future research, we aim to delve deeper into the broader applications of NeuroBack predictions, exploiting NeuroBack's full potential in SAT solving.