Paper: G4SATBench: Benchmarking and Advancing SAT Solving with Graph Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
G4SATBench contains only small scale formulas with at most 400 variables. However, industrial-level SAT formulas and formulas in SAT competitions can easily have thousands or even millions of variables. In order to make GNN-for-SAT approach practically applicable, the benchmarks used in the evaluation should also consider real-world level cases, such as large SAT formulas. I would encourage the authors to include such cases in G4SATBench.

Author response:
We acknowledge the potential value of including industrial-scale SAT instances in G4SATBench. However, our decision to exclude these instances is based on several considerations:

- Firstly, the size of industrial SAT instances often exceeds 1 million variables and clauses, which poses significant challenges for training GNN models effectively within GPU memory constraints. Furthermore, they are always limited in number (e.g., the industrial instances in SAT competitions are less than 1,000 instances), making it harder to train a good GNN model, as a diverse and ample dataset is critical for effective training.
- Secondly, we find that small-sized industrial instances in SAT Competitions often exhibit a high degree of similarity in their clauses, particularly within the same domain. This limits the ability of GNN models trained and evaluated on these instances to demonstrate true generalization across varied SAT problems.
- Thirdly, to partially address this, G4SATBench incorporates pseudo-industrial generators (CA and PS) to emulate some characteristics of industrial instances. This approach allows us to simulate the structure complexity of industrial scenarios while maintaining manageable dataset sizes and instance sizes.
- Lastly, our core objective is to establish a comprehensive framework for evaluating GNN-based SAT solvers and investigate how and what can (standalone) GNN models learn to solve SAT. Our results show that GNN models can develop a novel solving heuristic akin to local search, but fail to learn or mimic CDCL heuristic. Consequently, for broader applicability in industrial instances, we suggest future research could develop *neural-guided* SAT solvers that integrate the strengths of GNN models with modern CDCL-based approaches. We expect that our insights from G4SATBench will be instrumental in guiding such advancements.

Reviewer comment (Experiments):
GNN models evaluated in the study only contain basic ones, including NeuroSAT (the first GNN-for-SAT model), GCN, GGNN, and GIN. It would be interesting to compare GNN models applied by recent GNN-for-SAT approaches, and see how they perform in G4SATBench, instead of only considering basic GNN models.

Author response:
We observe that many existing GNN architectures for SAT solving only perform minor modifications to the original NeuroSAT [1,2], and the difference in performance between their architectures and NeuroSAT is marginal (experiments in G4SATBench also demonstrate that the performance between different GNN models is always similar). While there are some indeed novel GNN architectures for SAT solving [3,4], they are tailored for instances *not in CNF* and specialized to specific applications (like CircuitSAT), rather than general purposes. Due to this specialized nature, we've chosen to keep them outside the current scope of G4SATBench. Although using more advanced GNN models could achieve better performance, we believe our findings on the learning ability and limitations of GNN-based SAT solvers are still valid for these static GNN models, given their shared fundamental message-passing framework.

Reviewer comment (Experiments):
For experiments comparing with CDCL heuristics, the authors apply contrastive pretraining to pretrain the representation of CNF formulas to be close to their augmented ones with learned clauses. I have some doubts on the experimental setup. The learned clauses are not static and can vary depending on which CDCL solver we use. Studying whether GNN can learn these learned clauses is not quite reasonable to study if CNN can mimic CDCL. Furthermore, even though GNN can infer a bunch of learned clauses, can we say that CNN can learn a solving strategy akin to CDCL? As we know that CDCL not only involves clause learning, but also variable branching, etc.

Author response:
To demonstrate whether GNN models could effectively learn the CDCL heuristic, we first train a GNN model on the original SAT instances in the training set (referred to as model 1) and then evaluate their performance on the (clause learning) augmented instances in the testing set. Note that CDCL-based (or even backtracking-based) SAT solvers could easily solve these augmented instances in very few search steps (since the learned clauses effectively reduce the search space). Therefore, if GNNs are capable of learning the CDCL heuristic, we expect them to demonstrate a similarly high performance on these augmented instances. However, model 1 exhibits similar or, in many cases, inferior performance on the augmented instances compared to its performance on the original testing instances, suggesting that GNN models are not learning the CDCL heuristic.

Further, we train another GNN model specifically on the augmented SAT instances in the training set (model 2). and evaluate them on the augmented instances in the testing set. Experiments show that the performance of model 2 on the augmented testing set is notably better than that of model 1. Therefore, we hypothesize if the GNN models could learn the representation of the original instance to be similar to its augmented counterpart, their performance might improve. To explore this possibility, we use contrastive pretraining strategies [5], trying to make GNN models embed the clause learning heuristic in the latent space (i.e., learn a better representation like its clause learning augmented one). However, even with this approach, the GNN models faced challenges in effectively adopting the CDCL heuristic. This might be attributed to the inherent limitations of static GNN message passing, which struggles to adapt to the dynamic nature of clause learning in SAT solving, where additional nodes and edges dynamically modify the graph representation of an instance.

Reviewer comment (Experiments):
Why only small scale SAT formulas are included in the benchmark?

Author response:
We acknowledge the potential value of including industrial-scale SAT instances in G4SATBench. However, our decision to exclude these instances is based on several considerations:

- Firstly, the size of industrial SAT instances often exceeds 1 million variables and clauses, which poses significant challenges for training GNN models effectively within GPU memory constraints. Furthermore, they are always limited in number (e.g., the industrial instances in SAT competitions are less than 1,000 instances), making it harder to train a good GNN model, as a diverse and ample dataset is critical for effective training.
- Secondly, we find that small-sized industrial instances in SAT Competitions often exhibit a high degree of similarity in their clauses, particularly within the same domain. This limits the ability of GNN models trained and evaluated on these instances to demonstrate true generalization across varied SAT problems.
- Thirdly, to partially address this, G4SATBench incorporates pseudo-industrial generators (CA and PS) to emulate some characteristics of industrial instances. This approach allows us to simulate the structure complexity of industrial scenarios while maintaining manageable dataset sizes and instance sizes.
- Lastly, our core objective is to establish a comprehensive framework for evaluating GNN-based SAT solvers and investigate how and what can (standalone) GNN models learn to solve SAT. Our results show that GNN models can develop a novel solving heuristic akin to local search, but fail to learn or mimic CDCL heuristic. Consequently, for broader applicability in industrial instances, we suggest future research could develop *neural-guided* SAT solvers that integrate the strengths of GNN models with modern CDCL-based approaches. We expect that our insights from G4SATBench will be instrumental in guiding such advancements.

Reviewer comment (Experiments):
Why only basic GNN models are applied in the paper?

Author response:
We observe that many existing GNN architectures for SAT solving only perform minor modifications to the original NeuroSAT [1,2], and the difference in performance between their architectures and NeuroSAT is marginal (experiments in G4SATBench also demonstrate that the performance between different GNN models is always similar). While there are some indeed novel GNN architectures for SAT solving [3,4], they are tailored for instances *not in CNF* and specialized to specific applications (like CircuitSAT), rather than general purposes. Due to this specialized nature, we've chosen to keep them outside the current scope of G4SATBench. Although using more advanced GNN models could achieve better performance, we believe our findings on the learning ability and limitations of GNN-based SAT solvers are still valid for these static GNN models, given their shared fundamental message-passing framework.

Reviewer comment (Experiments):
What is the rationale behind the comparison between GNN and CDCL heuristics? Can you illustrate in more details?

Author response:
To demonstrate whether GNN models could effectively learn the CDCL heuristic, we first train a GNN model on the original SAT instances in the training set (referred to as model 1) and then evaluate their performance on the (clause learning) augmented instances in the testing set. Note that CDCL-based (or even backtracking-based) SAT solvers could easily solve these augmented instances in very few search steps (since the learned clauses effectively reduce the search space). Therefore, if GNNs are capable of learning the CDCL heuristic, we expect them to demonstrate a similarly high performance on these augmented instances. However, model 1 exhibits similar or, in many cases, inferior performance on the augmented instances compared to its performance on the original testing instances, suggesting that GNN models are not learning the CDCL heuristic.

Further, we train another GNN model specifically on the augmented SAT instances in the training set (model 2). and evaluate them on the augmented instances in the testing set. Experiments show that the performance of model 2 on the augmented testing set is notably better than that of model 1. Therefore, we hypothesize if the GNN models could learn the representation of the original instance to be similar to its augmented counterpart, their performance might improve. To explore this possibility, we use contrastive pretraining strategies [5], trying to make GNN models embed the clause learning heuristic in the latent space (i.e., learn a better representation like its clause learning augmented one). However, even with this approach, the GNN models faced challenges in effectively adopting the CDCL heuristic. This might be attributed to the inherent limitations of static GNN message passing, which struggles to adapt to the dynamic nature of clause learning in SAT solving, where additional nodes and edges dynamically modify the graph representation of an instance.

Reviewer comment (Experiments):
As described in this submission, the G4SATBench benchmark consists of 3 instance families, including random instances, pseudo-industrial instances, and combinatorial instances. However, SAT is so fascinating because of its great importance in real-world industrial applications. As a top-tier conference submission that focuses on proposing a SAT benchmark, it is needed to include industrial SAT instances. Hence, the lack of industrial SAT instances degrades the significance of this work.

Author response:
We acknowledge the potential value of including industrial-scale SAT instances in G4SATBench. However, our decision to exclude these instances is based on several considerations:

- Firstly, the size of industrial SAT instances often exceeds 1 million variables and clauses, which poses significant challenges for training GNN models effectively within GPU memory constraints. Furthermore, they are always limited in number (e.g., the industrial instances in SAT competitions are less than 1,000 instances), making it harder to train a good GNN model, as a diverse and ample dataset is critical for effective training.
- Secondly, we find that small-sized industrial instances in SAT Competitions often exhibit a high degree of similarity in their clauses, particularly within the same domain. This limits the ability of GNN models trained and evaluated on these instances to demonstrate true generalization across varied SAT problems.
- Thirdly, to partially address this, G4SATBench incorporates pseudo-industrial generators (CA and PS) to emulate some characteristics of industrial instances. This approach allows us to simulate the structure complexity of industrial scenarios while maintaining manageable dataset sizes and instance sizes.
- Lastly, our core objective is to establish a comprehensive framework for evaluating GNN-based SAT solvers and investigate how and what can (standalone) GNN models learn to solve SAT. Our results show that GNN models can develop a novel solving heuristic akin to local search, but fail to learn or mimic CDCL heuristic. Consequently, for broader applicability in industrial instances, we suggest future research could develop *neural-guided* SAT solvers that integrate the strengths of GNN models with modern CDCL-based approaches. We expect that our insights from G4SATBench will be instrumental in guiding such advancements.

Reviewer comment (Novelty):
It seems that the technical merit of this work is a bit thin. This paper describes a SAT benchmark, rather than proposing new SAT solving techniques. Also, the authors claim that an insight obtained from this paper is that GNN models can effectively learn a simple greedy local search strategy (such as GSAT algorithm), but it is difficult to learn backtracking search strategy (such as DPLL and CDCL). Since it is well-known that, for solving application SAT instances, CDCL-based SAT solvers greatly outperform local search SAT solvers, the insight of this work seems to argue that GNN-based SAT solvers are not promising to solve application SAT instances, which also degrades the significance of this work.

Author response:
One of our key contributions is to shed light on the capabilities and limitations of (standalone) GNN-based SAT solvers. While existing studies have mainly focused on developing various GNN architectures and methods to enhance solver performance, our work uniquely delves into understanding the mechanisms of how GNNs can learn to solve SAT problems. We acknowledge that CDCL-based SAT solvers currently outperform local search SAT solvers in most practical SAT applications. Given this nature, in Appendix E, we highlight the potential for future research to build *neural-guided* SAT solvers by integrating GNN models with modern CDCL-based solvers. For instance, GNN models could be employed as advanced local search heuristics, activated during the restart phases of CDCL-based SAT solvers. Or, leveraging the strong performance of GNN models in predicting unsat-core variables, as evidenced in G4SATBench, we could utilize these predictions to inform the branching decisions of CDCL-based solvers, thereby enhancing their efficiency. The goal of G4SATBench is not necessarily to develop a state-of-the-art SAT solver but rather to offer valuable insights into the strategies of GNN models for SAT solving. We hope these insights can lay the groundwork for future innovations and hybrid approaches in this area.

Reviewer comment (Evaluation):
Despite the thorough comparison within GNN-based methods, the benchmarking against traditional SAT-solving heuristics remains unclear. For this study to significantly impact SAT-solving, a more explicit comparison using time-to-solve metrics (preferably both arithmetic and geometric means, in accordance with standard practices) with leading industrial solvers or mainstream heuristics is needed.

Author response:
The primary objective of G4SATBench is to establish a comprehensive framework for evaluating GNN-based approaches to SAT solving and shed light on the learning capabilities and strategies of GNNs in SAT solving. To this end, our current framework does not include direct time-to-solve metric comparisons with modern SAT solvers. To enable such comparisons, we further conduct additional experiments using the local search solver Sparrow and CDCL SAT solver CaDiCaL on the instances in G4SATBench. Each experiment is repeated three times with different random seeds, and the mean results are reported.

We first set Sparrow to output the same number of solutions (i.e., 32) as NeuroSAT in our G4SATBench experiments. The following tables show the solving accuracy and running time of Sparrow:

Solving accuracy of Sparrow (%):

| Difficulty | SR    | 3-SAT | CA    | PS    | K-Clique | K-Domset | K-Vercov |
|------------|-------|-------|-------|-------|----------|----------|----------|
| Easy       | 56.03 | 52.09 | 85.48 | 77.25 | 53.68    | 37.15    | 31.61    |
| Medium     | 1.64  | 1.85  | 12.68 | 10.72 | 12.99    | 1.27     | 0.53     |

Running time of Sparrow (seconds per instance)

| Difficulty | SR    | 3-SAT | CA    | PS    | K-Clique | K-Domset | K-Vercov |
|------------|-------|-------|-------|-------|----------|----------|----------|
| Easy       | 0.005 | 0.005 | 0.006 | 0.006 | 0.005    | 0.005    | 0.005    |
| Medium     | 0.008 | 0.007 | 0.006 | 0.006 | 0.006    | 0.007    | 0.006    |

We then allow Sparrow and CaDiCaL to run without constraints to fully solve the satisfiable instances in G4SATBench and report the running time of each solver:

Running time of Sparrow (seconds per instance)

| Difficulty | SR    | 3-SAT | CA    | PS    | K-Clique | K-Domset | K-Vercov |
|------------|-------|-------|-------|-------|----------|----------|----------|
| Easy       | 0.007 | 0.007 | 0.008 | 0.008 | 0.007    | 0.009    | 0.009    |
| Medium     | 0.013 | 0.013 | 0.010 | 0.013 | 0.013    | 0.012    | 0.011    |

Running time of CaDiCaL (seconds per instance)

| Difficulty | SR    | 3-SAT | CA    | PS    | K-Clique | K-Domset | K-Vercov |
|------------|-------|-------|-------|-------|----------|----------|----------|
| Easy       | 0.013 | 0.013 | 0.012 | 0.011 | 0.014    | 0.012    | 0.011    |
| Medium     | 0.014 | 0.043 | 0.016 | 0.018 | 0.015    | 0.013    | 0.012    |

It's important to note that both CaDiCaL and Sparrow are highly optimized SAT solvers with advanced solving heuristics and engineering features. As a result, current *standalone* GNN-based SAT solvers face challenges competing with them. However, our findings indicate that when constrained to output the same number of solutions, GNN-based heuristics perform more effectively than modern local search solvers like Sparrow. Yet, when such a constraint is removed, as stated in our experiment section, GNN models may struggle to explore the vast solution space as efficiently as Sparrow and CaDiCaL.

Reviewer comment (Evaluation):
There is a lack of detailed discussion on where GNN-based SAT solvers fall short when compared to state-of-the-art (SOTA) SAT solvers. The authors should elaborate on the deficiencies of GNN-based methods and propose what advancements are necessary to surpass traditional SOTA SAT solvers.

Author response:
Our experimental results reveal that *standalone* GNN-based SAT solvers can develop a novel solving heuristic similar to local search. However, they currently fall short in emulating the CDCL heuristic, primarily due to the static nature of their message-passing framework that struggles to encapsulate the dynamic clause learning process inherent in CDCL. This limitation becomes evident when comparing their performance with CDCL-based solvers, particularly on large-scale industrial SAT instances where CDCL solvers excel.

To bridge this gap, as outlined in Appendix E, one avenue to enhance *standalone* GNN-based SAT solvers is to leverage *dynamic* GNN architectures to learn the CDCL heuristic within the latent space. Furthermore, our experiments demonstrate that GNN models excel in unsat-core variable prediction, pointing to another promising path toward achieving a state-of-the-art solver. This could be achieved by integrating GNN models with modern CDCL-based SAT solvers, combining their strengths for more effective SAT solving. For instance, GNN models could be employed initially to predict unsat-core variables, guiding CDCL-based SAT solvers to prioritize these variables in their branching strategies. Alternatively, GNN-based SAT solvers could be viewed as novel local search heuristics, complementing CDCL solvers in tackling SAT instances that are challenging for the CDCL heuristic alone.

Reviewer comment (Writing):
There appears to be a typographical error in the paragraph following Equation 1. Aggr(a)gation → Aggr(e)gation.

Author response:
We thank you for pointing out our typographical error. This has been corrected to "Aggregation" in the revision.

Reviewer comment (Writing):
In Section 4.3, "Training Objectives," the relationship between S(x), S_min, and S_max could benefit from clarification. Additionally, the use of N(a) within this context is ambiguous.

Author response:
$S_{max}$, $S_{min}$, $N$ are the smoothed approximations for the logical operators OR ($\lor$), AND ($\land$), and Not ($\neg$), respectively. Given a CNF formula $\phi$ and a predicted assignment $x$, we employ these smoothed functions to replace the standard logical operators in $\phi$ and substitute the variable in $\phi$ with the corresponding values from $x$ to calculate its satisfiability score $S_{\phi}(x)$. We have provided a clearer explanation of this part in the revision.