Paper: MixSATGEN: Learning Graph Mixing for SAT Instance Generation (ICLR-2024)

Reviewer comment (Evaluation):
Experimental results show that the proposed method is not always superior to HardSATGEN. Especially, the running time evaluation results in Tab.2 show that the proposed method fails to preserve the computational difficulties of hard instances. The results seem to contradict the claim of the paper.

Author response:
> **Q1: Experimental results show that the proposed method is not always superior to HardSATGEN.**

Please note that reproducing computational hardness in the SAT instance generation task is considerably challenging which has not been achieved until HardSATGEN [1]. Its success stems from its fine-grained control over the instance structure, particularly for the unsatisfiable core structures, as well as the postprocessing design involved to maintain the dominant effect of the unsat core structure.


While the design in [1] is effective, it confines the method to unsatisfiable instances only. Since real-world datasets generally incorporate both satisfiable and unsatisfiable instances, merely learning from the unsat instances can lead to certain negative impacts on downstream tasks like hyperparameter tuning, as evidenced by Table 3 in the new revision. In contrast, MixSAT surpasses this limitation, demonstrating generality to both satisfiable and unsatisfiable instances. MixSAT is also free from the node merge-split framework of previous learning-based methods [1-3], naturally meeting the hardness analysis and requirements set forth in [1].  Notably, our performance on satisfiable instances is significantly superior to all other methods. 

We acknowledge that our effectiveness on unsatisfiable instances is not as strong, especially considering [1] already provides a tailored solution for unsatisfiable problems. However, we believe the two methods can complement each other. As evidence, we have supplemented the study with hyperparameter optimization experiments and present the combined effects of both methods in Table 3, which show the significant effectiveness of our methods in practice as a boosting tool complementary to HardSATGEN.

Reviewer comment (Experiments):
Li et al. (2023) reported that hyperparameter tuning with instances generated by HardSATGEN improves performance on real-world problems. Since a primal objective of generating SAT instances is to tune the parameters of solvers, the paper should report the results of using the instances generated by MixSAT to tune the parameters of existing solvers.

Author response:
> **Q2: Hyperparameter tuning with the generated instances.**

Thanks for the valuable point. We supplement the hyperparameter tuning results on the state-of-the-art SAT solver Kissat in Table 3 of the new manuscript. Due to the complexity of the table format, we refrain from presenting it in this response. 

We adopt one of the Bayesian optimizers HEBO to tune the solver. The tuned hyperparameters include restart interval, reduce interval, and decay, which pose impacts on the frequency of restarting, frequency of learned clause reduction, and per mille scores decay respectively. For each method, the Bayesian optimizer tries 200 sets of hyperparameters and we select the top 5 hyperparameters to evaluate on test instances. The best runtime results are presented in the table. Due to the costly Bayesian optimization process, we select instances with runtimes under 500 seconds for experiments. 

Note in Table 2, MixSAT reproduces computational hardness well on satisfiable instances, while HardSATGEN [1] can only learn from unsatisfiable instances. Table 3 shows the matched performance where MixSAT and HardSATGEN perform the best for satisfiable and unsatisfiable instances respectively. Thus, we also consider simultaneously adopting MixSAT learned from satisfiable instances and HardSATGEN learned from unsatisfiable instances for tuning, which presents the best overall performance. In general, the generated instances by the proposed methods bring a significant solver runtime reduction of over 50% to the solver.

Reviewer comment (Reproducibility):
The proposed method generates instances by interpolating two instances. Therefore, the performance of the proposed method depends on how we make a pair. However, the details of making pairs are not reported in the paper.

Author response:
> **Q4: The performance of the proposed method depends on how we make a pair. However, the details of making pairs are not reported in the paper.**


Firstly, it is essential to note that the core methodology of our approach centers around the design of the interpolation method, which is non-trivial when applied to graph structures, especially in the case of large-scale bipartite graphs with distinctive structures. In the paper, we elaborate on how we design the graph interpolation method to ensure consistency in generating data that aligns with the original distribution, in structural (through matching) and hardness (through matching with randomness) aspects.


On the other hand, of course pair selection method is an integral part of our methodology. The process of how we filter pairs has been detailed in Sec. 2.3 of the original paper, with the specific design outlined as follows: 

We introduce an entropy-based filter to first select potentially valuable pairs for mixing (since the similarity of different pairs can vary much, not all pairs can yield quality matching results), which assesses and filters pairs using the row-wise entropy of the soft alignment matrix $\mathbf{S}\in\mathbb{R}^{n_1\times n_2}$:
$$h = -\sum_{i=0}^{n_1} \sum_{j=0}^{n_2}s'\_{ij}\log{s'\_{ij}} \quad \text{where} \quad s'\_{ij} = \frac{s\_{ij}}{\sum\_{k}s\_{ik}}$$
which implies the sharpness of the soft alignment matrix between $G_1$ and $G_2$. A smaller value of $h$ denotes more distinct matching relations, leading to a potentially better pair for mixing.

Reviewer comment (Experiments):
It is said that the dataset was taken from SATLIB and SAT Competition 2021. However, how the authors select easy and hard [a-h] instances from a large amount of instances seems not explained in the paper. The paper should show the reason behind selecting these instances to make the evaluation fair.

Author response:
> **Q5: The reason behind selecting these instances as datasets to make the evaluation fair.**

When selecting instances to construct the dataset for evaluation, we consider the following factors: 

1) It should include instances derived from real-world application scenarios to ensure the practical value of our experimental validation. Therefore, we choose instances from SAT competition benchmarks, which are encoded from real-world scenarios. 
 
2) We aim to maintain an equal proportion of satisfiable and unsatisfiable instances in our data to verify the general applicability of MixSAT. Hence, we maintain a 1:1 ratio of sat to unsat instances in the dataset. 

3) Recognizing the significance of computational hardness in experimental validation, we specifically select instances with higher computational hardness to constitute the Hard dataset.

Reviewer comment (Experiments):
The paper says the proposed method is more efficient (page 2), but no experimental results support the claim.

Author response:
> **Q6: Experimental support for "the proposed method is more efficient".**


To demonstrate the efficiency advantage of MixSAT over previous state-of-the-art learning-based methods that generate from scratch, we compared G2SAT[2] with MixSAT ($\lambda=0.1, \tau=1.0$) for single instance generation, which is adopted as the standard training scheme in [3].


The pipelines consist of two processes: training and generating. The training process only needs to be performed once for a particular dataset, while the generating process can produce formulas repeatedly according to the target quantity. We measure the time for training and generating one specific formula. As shown below and also in Table 8 of the updated revision,  MixSAT is significantly more efficient than G2SAT in both processes. Specifically, for the generating phase, MixSAT costs only approximately 1/2000 runtime compared to G2SAT.


|       |     **Training**      |   **Generating**    |
|:----------:|:---------------------:|:-------------------:|
| **G2SAT**  | 4467.05 $\pm$ 2315.81 | 2242.07 $\pm$ 1280.28 |
| **MixSAT** | 1760.03 $\pm$ 457.89  |   1.44 $\pm$ 0.22   |

Reviewer comment (Evaluation):
The paper says that MaxSAT introduces some randomness to maintain computational hardness. However, experimental results reported in Table 2 show that randomness does not help much in preserving hardness.

Author response:
> **Q7: Experimental support for "MixSAT introduces some randomness to maintain computational hardness".**


Note Table 2 illustrates the average solver runtime of all pairs across each instance. In our analysis, a tunable level of randomness is introduced to prevent hardness degradation caused by the interaction and conflicts of substructures. However, some deterministically generated formulas ($\lambda=0$) have already effectively maintained the hardness, surpassing the entirely random ones (due to inherent randomness). Consequently, the average solver runtime cannot effectively reflect the improvement of hardness led by introducing randomness.

To straightforwardly reiterate the effect of randomness on the computational hardness of generating formulas, below and also in Table 5 of the new revision presents the results on instance pairs where MixSAT exhibits some degree of hardness degradation. We gradually increased the $\lambda$ value to enhance the randomness. As seen, when the randomness increases,  the hardness of the initially degraded pairs is recovered to a certain extent compared with the ground truth. Note although pair $e$-$f$ fails to maintain hardness on both deterministic ($\lambda=0$) and purely random generations, it gains some hardness when partial randomness is introduced ($\lambda=0.5$).

| **Randomness**      | **pair $a$-$c$**  | **pair $b$-$d$**    | **pair $d$-$c$**   | **pair $e$-$f$** |
|:-------------------:|:-----------------:|:-------------------:|:------------------:|:----------------:|
| **Ground Truth**    | 254.22            | 1382.71             | 852.85             | 2629.03          |
| **$\lambda=0$**     | 0.09 $\pm$ 0.01     | 52.73 $\pm$ 3.87      | 117.83 $\pm$ 129.39  | 0.31 $\pm$ 0.01    |
| **$\lambda=0.1$**   | 0.29 $\pm$ 0.01     | 138.11 $\pm$ 18.68    | 199.63 $\pm$ 190.70  | 33.95 $\pm$ 0.03   |
| **$\lambda=0.5$**   | 0.19 $\pm$ 0.02     | 23.14 $\pm$ 0.74      | 122.32 $\pm$ 12.33   | 500.12 $\pm$ 4.60  |
| **$\lambda=1.0$**   | 3.59 $\pm$ 0.06     | 442.11 $\pm$ 3.53     | 188.93 $\pm$ 18.49   | 21.42 $\pm$ 2.41   |
| **$\lambda=10.0$**  | 39.88 $\pm$ 6.91    | 332.05 $\pm$ 25.57    | 1530.78 $\pm$ 838.56 | 2.05 $\pm$ 1.80    |
| **$\lambda=100.0$** | 130.32 $\pm$ 63.88  | 472.43 $\pm$ 13.31    | 1429.93 $\pm$ 308.41 | 0.69 $\pm$ 0.68    |
| **Random**          | 215.73 $\pm$ 120.95 | 1116.42 $\pm$ 1203.59 | 695.24 $\pm$  799.58  | 11.49 $\pm$ 9.12   |

Reviewer comment (Evaluation):
Experimental results look rather weak to me as they seem to be a mixed bag. At least, I don't see a clear advantage of the proposed approach over the state of the art in terms of the hardness of the generated formulas.

Author response:
Although MixSAT performs weaker than HardSATGEN [1] on unsat instances, it outperforms other methods. Please note that reproducing computational hardness in the SAT instance generation task is considerably challenging which has not been achieved until [1]. Its success stems from its fine-grained control over the instance structure, particularly for the unsatisfiable core structures, as well as the postprocessing design involved to maintain the dominant effect of the unsat core structure.

While the design in [1] is effective, it confines the method to unsatisfiable instances only. Since real-world datasets generally incorporate both satisfiable and unsatisfiable instances, merely learning from the unsat instances can lead to certain negative impacts on downstream tasks like hyperparameter tuning, as evidenced by Table 3 in the current revision. In contrast, MixSAT surpasses this limitation, demonstrating generality to both satisfiable and unsatisfiable instances. MixSAT is also free from the node merge-split framework of previous learning-based methods [1-3], naturally meeting the hardness analysis and requirements set forth in [1].  Notably, our performance on satisfiable instances is significantly superior to all other methods. 

We acknowledge that our effectiveness on unsatisfiable instances is not as strong, especially considering [1] already provides a tailored solution for unsatisfiable problems. However, we believe the two methods can complement each other. As evidence, we have supplemented the study with hyperparameter optimization experiments and present the combined effects of both methods in Table 3, which shows the significant effectiveness of our methods in practice as a boosting tool complementary to HardSATGEN.

Reviewer comment (Evaluation):
A key question in this line of research is whether the formulas generated automatically exhibit the properties intrinsic to "industrial instances". This is important given the widespread use of modern SAT solvers in industrial settings. So I would like to ask the authors whether the ability of their approach to capture this.

Author response:
The data we adopt in our experiments are exactly derived from real-world scenarios (from SAT Competition Benchmarks). Note the disparity between industrial data and real-world data can be viewed as minimal, as both originate from practical application scenarios. The experiments are designed to evaluate whether the generated instances exhibit the properties intrinsic to real-world instances. The evaluation for the real-world resemblance is two-fold: 1) structural properties compared to the reference real-world instances (Sec. 3.2); 2) computational properties compared to the reference real-world instances (Sec. 3.3). In the rebuttal period, we also supplement solver hyperparameter tuning results in Sec. 3.4, which indicates that the generated instances maintain close properties to the reference real-world instances so that hyperparameter tuning led by them could bring a significant solver runtime reduction of over 50% to the solver. The evaluation has already shown the effectiveness of MixSAT in generating instances that exhibit the properties intrinsic to real-world instances.

Reviewer comment (Theory):
Can you comment on the rationale of (at least some of) the steps your approach performs?

Author response:
Thanks for the note. We totally agree with your assertion that the rationale behind every effort of the methodology design should be articulated and elucidated in the paper. 

We believe this principle should be a consensus across both the mainstream SAT research and the ML4SAT research. In fact, we try to make efforts to achieve such clarity in the original paper. Indeed, we acknowledge that certain points may not have been sufficiently clarified, leading to confusion. Here we would like to elucidate the underlying rationale for each design choice in our methodology. We hope this clarification addresses your concerns and fortifies the overall clarity of our work.

- **Why mixing?** Previous node split-merge framework for SAT instance generation exhibits poor performance in computational hardness resemblance. Though more fine-grained controls in [1] on structures can improve the performance, it also introduces applicability limitations. Thus we resort to a different graph interpolation approach that directly manipulates the raw structures of SAT instances in place, which is achieved by mixing substructures of SAT instances. This design is more efficient, more adaptive, and can naturally avoid the hardness degradation causes in the split-merge framework as analyzed in [1]. (Page 2)

- **Why matching?** The exchanged substructures should be matched such that the mixing process can largely preserve the original essential patterns and the mixed instances remain within the underlying distribution of reference instances regarding both structural similarity and computational hardness. (Page 3)

- **Why adopt the neural SAT solving task as a pretraining task?** It is for the sake of hardness-awareness. This pretraining task enables the model to not only capture the graph structure but also naturally discover the inherent properties regarding the computational hardness of the instances. (Page 4)

- **Why perform online optimization with matching loss?** The pretraining task guarantees plausible initial node features, but it is not directly related to the matching purpose. The online optimization with matching loss is to directly maximize the similarity between the node features of the two SAT instances. (Page 5)

- **Why introduce randomness?** Learning-based methods excel at identifying similarities within structures. With the replacement on top of the learned variable correspondence map, similar substructures from two reference graphs can easily interact and probably lead to conflicts and consequent hardness degradation. Thus, beyond the regular matching pipeline, a tunable level of randomness could avoid hardness degradation. (Page 5)

- **Why use Gumbel trick to introduce randomness?** Introducing randomness to assignments is non-trivial, while Gumbel trick on discrete assignment distributions is well studied in [4] with satisfying sampling property that $H(\mathbf{M}+\mathbf{\Gamma})\sim\mathcal{G.M.}(\mathbf{M})$, where $\mathbf{\Gamma}$ denotes a Gumbel noise matrix, $\mathcal{G.M.}(\mathbf{M})$ is the Gumbel-Matching distribution with parameter $\mathbf{M}$, and $H(\cdot)$ denotes the matching operator providing the best matching results, which can be achieved by the Hungarian algorithm for the linear matching case. (Page 5)

- **Why this mixing algorithm?** The mixing algorithm we propose in this paper is simple and natural. It directly utilizes the variable matching confidence obtained in the matching process, which increases the probability of replacing similar structures. We perform substitutions at the clause level because clauses are the smallest substructure of the SAT formula, thus they correspond to the smallest interpolation step in the interpolation scenario. (Page 6)

We have revised the manuscript accordingly to increase the presentation clarity of our paper. Please let us know if you have any other confusion regarding the approach rationale.

Reviewer comment (Presentation):
Minor points related to the presentation and evaluation.

Author response:
We sincerely appreciate your thorough review. We will revise the manuscript accordingly to elevate it into a more refined paper.

Reviewer comment (Presentation):
I would also discuss the limitations of MixSAT.

Author response:
Currently, the primary limitation of our approach lies in the fact that our hardness reproducing performance on unsatisfiable examples, though better than other methods, is not as effective as that achieved by HardSATGEN [1]. However, please note that reproducing computational hardness in the SAT instance generation task is considerably challenging which has not been achieved until [1]. Its success stems from its fine-grained control over the instance structure, particularly for the unsatisfiable core structures, as well as the postprocessing design involved to maintain the dominant effect of the unsat core structure.

While the design in [1] is effective, it confines the method to unsatisfiable instances only. Since real-world datasets generally incorporate both satisfiable and unsatisfiable instances, merely learning from the unsat instances can lead to certain negative impacts on downstream tasks like hyperparameter tuning, as evidenced by Table 3. In contrast, MixSAT surpasses this limitation, demonstrating generality to both satisfiable and unsatisfiable instances. MixSAT is also free from the node merge-split framework of previous learning-based methods [1-3], naturally meeting the hardness analysis and requirements set forth in [1].  Notably, our performance on satisfiable instances is significantly superior to all other methods. 

On the other hand, we believe the two methods can complement each other. As evidence, we have supplemented the study with hyperparameter optimization experiments and present the combined effects of both methods in Table 3 of the current revision, which shows the significant effectiveness of our methods in practice as a boosting tool complementary to HardSATGEN.

Reviewer comment (Evaluation):
Besides structural properties and hardness, does MixSAT preserve the SAT/UNSAT ratio of the original data?

Author response:
Thanks for the comment. The results of satisfiability phase preservation for generated formulas are presented below and also in Table 10 of the current revision. It is observed that despite being trained on a dataset with a one-to-one ratio of satisfiable to unsatisfiable instances, previous methods exhibit a tendency to generate a specific satisfiability phase. Notably, instances generated by G2SAT, GCN2S, and HardSATGEN are all unsatisfiable, while PS generates exclusively satisfiable instances. Among all the methods, MixSAT maintains the closest phase ratio to the original reference instances.

Unsatisfiable ratio of the generated instances:

|  **Method**  | **EASY** | **HARD** |
|:-:|:-:|:-:|
| Ground Truth | 50.00%  | 50.00%  |
|  CA  |  87.5%  | 75.00%  |
|  PS  |  0.00%  | 0.00%  |
|  G2SAT | 100.00% | 100.00% |
| GCN2S | 100.00% | 100.00% |
|  HardSATGEN  |  100%   | 100.00% |
| MixSAT | 23.61%  | 43.41%  |

Reviewer comment (Evaluation):
What are the limitations of your approach?

Author response:
Currently, the primary limitation of our approach lies in the fact that our hardness reproducing performance on unsatisfiable examples, though better than other methods, is not as effective as that achieved by HardSATGEN [1]. However, please note that reproducing computational hardness in the SAT instance generation task is considerably challenging which has not been achieved until [1]. Its success stems from its fine-grained control over the instance structure, particularly for the unsatisfiable core structures, as well as the postprocessing design involved to maintain the dominant effect of the unsat core structure.

While the design in [1] is effective, it confines the method to unsatisfiable instances only. Since real-world datasets generally incorporate both satisfiable and unsatisfiable instances, merely learning from the unsat instances can lead to certain negative impacts on downstream tasks like hyperparameter tuning, as evidenced by Table 3. In contrast, MixSAT surpasses this limitation, demonstrating generality to both satisfiable and unsatisfiable instances. MixSAT is also free from the node merge-split framework of previous learning-based methods [1-3], naturally meeting the hardness analysis and requirements set forth in [1].  Notably, our performance on satisfiable instances is significantly superior to all other methods. 

On the other hand, we believe the two methods can complement each other. As evidence, we have supplemented the study with hyperparameter optimization experiments and present the combined effects of both methods in Table 3 of the current revision, which shows the significant effectiveness of our methods in practice as a boosting tool complementary to HardSATGEN.

Reviewer comment (Evaluation):
Do you observe a degratation in performance when interpolating formulas generated by MixSAT?

Author response:
To evaluate the effect of secondary interpolation on the performance of formulas generated by MixSAT, we select two representative instance pairs and apply MixSAT ($\lambda=0.1,\tau=1$) to obtain their 5% mixing ratio formulas. We then use these formulas as the start formulas and interpolate them with their corresponding goal formulas, resulting in 10% mixing ratio formulas. We compare these formulas with the original formulas, the 5% start formulas, and the 10% formulas obtained by one interpolation. The comparison is shown below and also in Table 9 of the current revision, where we can see that the secondary interpolation does not degrade the performance of the formulas, as they maintain the same level of hardness as the one-interpolated formulas. This demonstrates the robustness of MixSAT in generating hard formulas through interpolation.

|   | **Ground Truth** |  **5%**  | **Interpolated 10%** | **10%**  |
|:-:|:-:|:-:|:-:|:-:|
| **hard $b$-$a$** | 1382.71 | 1023.25 $\pm$ 4.69  | 1001.97 $\pm$ 7.28 | 1042.53 $\pm$ 4.78 |
| **hard $d$-$c$** | 852.85 | 199.63 $\pm$ 190.70 | 6.13 $\pm$ 3.14 | 3.56 $\pm$ 1.69  |

Reviewer comment (Experiments):
Is it possible to learn a binary classifier that accurately discriminates original vs. MixSAT instances?

Author response:
Due to the limited data availability in specific SAT scenarios (which is also the motivation for designing generation algorithms), the current dataset size is insufficient to support the training of a classification network. In fact, we believe this situation is widely acknowledged in literature [1,2].

We understand your concern, as classification is a method to discern whether the generated distribution approximates the real one. Indeed, in our experiments, out of the same consideration, we have already validated the structural and hardness similarities between generated and real instances. Additionally, during the rebuttal period, we have highlighted MixSAT's superior performance in solver tuning. We believe these results sufficiently support the approximation of the distribution between the data generated by our method and real-world data.

Reviewer comment (Writing):
"e.g. in EDA." — What does EDA stand for?

Author response:
EDA stands for "Electronic Design Automation", a practical application scenario of SAT problem, where SAT instances exhibit scarcity. Sorry for the confusion and we have fixed it in the revision.

Reviewer comment (Reproducibility):
Adding the pseudocode of MixSAT would greatly help.

Author response:
Thanks for the comment. We supplement the pseudocode in Appendix C to further enhance our methodology presentation clarity.