Paper:
We present GQSAT, a branching heuristic in a Boolean SAT solver trained with value-based reinforcement learning (RL) using Graph Neural Networks for function approximation. Solvers using GQSAT are complete SAT solvers that either provide a satisfying assignment or a proof of unsatisfiability, which is required for many SAT applications. The branching heuristic commonly used in SAT solvers today suffers from bad decisions during their warm-up period, whereas GQSAT has been trained to examine the structure of the particular problem instance to make better decisions at the beginning of the search. Training GQSAT is data efficient and does not require elaborate dataset preparation or feature engineering to train. We train GQSAT on small SAT problems using RL interfacing with an existing SAT solver. We show that GQSAT is able to reduce the number of iterations required to solve SAT problems by 2-3X, and it generalizes to unsatisfiable SAT instances, as well as to problems with 5X more variables than it was trained on. We also show that, to a lesser extent, it generalizes to SAT problems from different domains by evaluating it on graph coloring. Our experiments show that augmenting SAT solvers with agents trained with RL and graph neural networks can improve performance on the SAT search problem.

Review:
The paper proposes learning a branching heuristic to be used inside the SAT solver MiniSat using reinforcement learning. The state is represented as a graph representation of the Boolean formula as in previous works, and the policy is parameterized as a graph neural network. At each step of an episode the policy selects a variable to branch on and assigns a value to it. The episode terminates once the solver finds a satisfying assignment or proves unsatisfiability. The reward function encourages the policy to reach terminal state in as few steps as possible. The policy is trained using DQN. Results on randomly generated SAT instances show that the learned policy is able to solve problems with fewer steps than VSIDS, the branching heuristic commonly used by state-of-the-art solvers.
Pros:
- The paper is nicely written and easy to read.
- The general idea of using RL to learn distribution-specific branching heuristics is a very interesting research problem, and SAT is a difficult test case for it.
- The experiments provide interesting insights, especially Figure 2 and the graph coloring results in Table 3.
Cons:
- Showing improvements in the number of steps compared to VSIDS is not interesting because VSIDS as implemented in MiniSat and state-of-the-art solvers like Glucose has been tuned to minimize running time rather than number of steps.  A better comparison would be to compare against a branching heuristic that is designed to be step-efficient -- e.g., the branching heuristic GGB proposed in Chapter 3 of Liang’s PhD thesis (available here: https://drive.google.com/file/d/1RzJtmdbjFeT2N84WDWQkBfQoGPF-qRoT/view?usp=sharing). GGB is more expensive than VSIDS, but if the time needed for branching is removed, a solver using GGB is faster than one using VSIDS (see figure 3.1 in the thesis).
- A better discussion of how to scale up the proposed approach to instances with millions of variables is needed. Although most ML papers on SAT deal with at most hundreds of variables, such small instances are trivial for the state-of-the-art solvers. The real challenge is to scale up to the instance sizes that are considered in SAT competitions. For example, Selsam and Bjorner 2019 https://arxiv.org/pdf/1903.04671.pdf tackle such instances in their NeuroCore work. (While the paper references that work, it doesn’t compare to it.) There is no attempt to address the scalability issue in this work. Apart from considering the challenges of successfully learning on large instances (briefly discussed in the conclusion), there should be an analysis on the inference cost of the graph neural network (which scales linearly with the number of variables and clauses) to make a single branching decision vs. that of VSIDS, and what that implies on how much reduction in the number of steps a learned branching policy would need to achieve before it can provide time savings over VSIDS. It may turn out that the reductions required of a learned branching policy are implausibly large. Without a better understanding of these challenges, it is not clear that learning can help much to improve the state-of-the-art SAT solvers.