Learning to Find Proofs and Theorems by Learning to Reﬁne Search Strategies The Case of Loop Invariant Synthesis
Abstract
We propose a new approach to automated theorem proving where an AlphaZerostyle agent is self-training to reﬁne a generic high-level expert strategy expressed as a nondeterministic program.
An analogous teacher agent is self-training to generate tasks of suitable relevance and difﬁculty for the learner.
This allows leveraging minimal amounts of domain knowledge to tackle problems for which training data is unavailable or hard to synthesize.
As an example application, we illustrate our approach on the problem of loop invariant synthesis for imperative programs and use neural networks to reﬁne both the teacher and solver strategies.
Introduction
Augmenting tactic-based interactive theorem provers with neural guidance has been the focus of increased attention in recent years [1, 2, 3, 4, 5].
The dominant approach uses imitation learning on corpora of formalized mathematics.
However, despite recent efforts involving self-supervised pre-training [5] or data-augmentation [6], this approach is limited by the conspicuous scarcity of human-produced training data.
An alternative approach inspired by the success of AlphaZero [7] is to use reinforcement learning and let an agent self-train to interact with a theorem prover via trial and error [8, 9].
However, previous attempts of doing so have been hampered by two fundamental issues:
• First, existing tactic-based theorem provers offer inﬁnite action spaces not amenable to random exploration.
Also, they are optimized for formalizing the outcome of human insights concisely but often fail to deﬁne a tractable search space for deriving those insights in the ﬁrst place [10].
For example, using tactics effectively often requires providing insights in advance that are more easily found later in the search process (e
g. constructing a term to instantiate an existential quantiﬁer [11]).
More generally, most of the human process of discovering proofs, which involves trial and error, sketching, and abductive reasoning is not captured by standard prover tactics and so we can hardly expect a reinforcement learning agent to learn this process through sheer interaction with a tactic-based prover.
Second, although reinforcement learning alleviates the need for human proofs during training, the agent must still be provided learning tasks of suitable relevance, diversity, difﬁculty, and generalizability in the form of theorem statements to be proved.
This is in contrast with applications of AlphaZero in board games, where the symmetric nature of games such as Chess or Go enables leveraging symmetric self-play as an inﬁnite source of training tasks.
In this paper, we suggest a novel approach to learning automated theorem proving that does not rely on human-provided proofs and theorems.
Instead, a teacher agent is trained to generate interesting and relevant tasks while a solver agent is co-trained to solve them.
Our core insight is that both agents can be implemented by using reinforcement learning to reﬁne high-level search strategies [10] written by assume x >= 1; y = 0; while (y < 1000) { x = x + y;
y = y + 1; } assert x >=
y;
A desirable invariant is a formula I [ x, y ] such that: ∀ x, y   x ≥ 1 ∧ y =
→ I [ x, y ] (holds initially)
y <
1000 ∧ I [ x, y ]
→ I [ x + y, y + 1] (preserved) y ≥ 1000 ∧ I [ x, y ] → x ≥ y (implies post)
Such an invariant is I [ x, y ] :=
( x ≥ y ∧ x ≥
∧ y ≥ 0) experts in the form of nondeterministic programs. Our paper also deﬁnes novel design principles and language constructs for expressing such strategies.
These include conditional generative strategies as a template for implementing teachers, abductive reasoning as a design principle that is made scalable by self-learned guidance, and strategy events as an abstraction that enables easier reward engineering and better sample-efﬁciency.
This paper introduces our proposed framework and describes a wellcontained yet challenging setting, namely the automatic veriﬁcation of imperative programs with loops. We frame our contributions in a way to emphasize general applicability.
An empirical evaluation in other application domains is left to future work.
Approach 2.1 Background: Verifying Imperative Programs with Loops
Suppose we are given a program such as the one in Figure 1 and we want to prove that the ﬁnal assertion always holds.
The way to proceed is to ﬁnd a predicate called a loop invariant with the following properties: i) it is true before the loop is entered, ii) it is preserved by the loop body when the loop guard holds and iii) it implies the ﬁnal assertion when the loop guard does not hold.
By “preserved”, we mean that if the invariant is true before executing the loop body then it is also true afterwards.
Finding loop invariants is the most crucial aspect of program veriﬁcation [12, 13] and still resists automation [14].
Despite the difﬁculty of automatically synthesizing loop invariants, humans do so routinely using well-understood search strategies.
For example, in the case of the example in Figure 1, one would ﬁrst try to prove the postcondition x ≥ y itself as an invariant and then note that it is not preserved by the loop body as the following implication does not hold: y < 1000 ∧ x ≥ y (cid:54)→ x + y ≥ y + 1 .
However, simplifying the right-hand side, we see that the proof works if we can show x ≥ 1 to be an invariant itself.
Using a similar form of abductive reasoning, we ﬁnd that this in turn requires y ≥to be an invariant and we end up proposing x ≥ y ∧ x ≥ 1 ∧ y ≥ 0 as a loop invariant.
2.2 Expressing Search Strategies as Nondeterministic Programs
The search strategy we followed for the example above can be formalized as a nondeterministic program, which is shown in Figure 2.
As a nondeterministic program, it features a choose operator that takes a list of objects as an input and selects one of them nondeterministically.
A nondeterministic program can be reﬁned into a deterministic one by providing an external oracle to implement the choose operator.
It also deﬁnes a search tree that can be explored with or without a guiding heuristic.
In this work, we use neural networks to implement such oracles and guiding heuristics.
In addition, a key aspect of this strategy is to infer missing assumptions under which a currently failing proof obligation would hold.
This form of reasoning is called abductive reasoning and it is fundamental in the way humans search for proofs [16, 17].
It is implemented in a separate abduct procedure that takes a formula as an input and returns either Valid or a list of abduction candidates.
For example, the abduct procedure fails to prove the implication x ≥
→ x + y ≥ 1 but may suggest x + y ≥ 1 , x < 0 and y ≥ 1 as possible missing assumptions.
The use of abductive reasoning for theorem proving and loop invariant synthesis speciﬁcally has been proposed in the past [18, 19].
However, abduction procedures are hard to implement [20] and typically only available for speciﬁc decidable theories.
Also, using them in proof search tends to scale poorly in the absence of
1 def solver( init: Formula, guard: Formula, body: Program, post: Formula) -> Formula:
: return [
[Figure] [Figure]
[Figure] [Figure]
See Appendix A for more details and for a full listing of the invariant synthesis strategy that we use in our experiments. good heuristics to rank and ﬁlter abduction candidates.
Our proposed framework makes abductive reasoning practical by addressing both issues: it allows leveraging self-learned guidance to select abduction candidates and it allows implementing abduction procedures as nondeterministic programs that are amenable to learning themselves.
Another notable feature of the strategy in Figure 2 is the use of the reward operator on line 17 to incentivize ﬁnding short invariants (short proofs are desirable in general as they tend to be easier to check, interpret and generalize).
The reward operator can be used multiple times and at any point of the strategy execution.
An implicit reward of 1 or -1 is emitted when the strategy successfully returns or fails respectively.
An optimal execution of a nondeterministic strategy is one that maximizes the cumulative amount of collected rewards.
In this example, we bound the maximal invariant size penalty in such a way to ensure that ﬁnding a proof is always rewarded more than failing at doing so.
As argued by Selsam [10], deﬁning search strategies as nondeterministic programs provides a natural and ﬂexible way for experts to leverage domain-speciﬁc knowledge while outsourcing difﬁcult proof decisions to search algorithms and learned heuristics.
This paradigm differs from the standard paradigm of tactic-based theorem provers in which an external entity must orchestrate stateless tactics that do not call for any form of interactive guidance internally.
In contrast, the nondeterministic programming approach inverts control and has strategies call external oracles rather than the other way around, which allows for a much tighter control of the resulting search space.
2.3 Reﬁning Nondeterministic Strategies with Reinforcement Learning
A nondeterministic program induces a (deterministic) Markov Decision Process (MDP) where intermediate states are choice points and ﬁnal states are either success states (the program returns successfully) or failure states (an assertion is violated or choose is called on an empty list of choices).
Standard search algorithms can be used to navigate this MDP but doing so in an efﬁcient and scalable way requires strong heuristics for guiding search.
In this work, we propose to learn such heuristics in a self-supervised fashion using the AlphaZero algorithm [7, 21].
In AlphaZero, a neural network is trained to map any state to a probability distribution over available actions along with a value estimate (i.e. an estimate of the expected sum of future rewards).
The neural network alone can be used as a policy for navigating the MDP.
However, a stronger policy results from combining the network with a tree search algorithm such as Monte-Carlo
Tree Search (MCTS) [22].
The key insight underlying AlphaZero is that the network can be trained via an iterative improvement process where it is successively i) used as an MCTS heuristic to try and solve problems and then ii) updated using gradient descent so as to better predict the outcome of each attempt along with the action selected by MCTS on all states encountered along the way.
More details on AlphaZero can be found in the literature [7].
Using AlphaZero, we can reﬁne nondeterministic proof search strategies without external proof examples to learn from.
However, AlphaZero must still be provided with a set of training problems to be solved.
Training problems should be diverse, relevant and numerous enough to allow proper generalization.
They should also be of varied difﬁculty to allow for learning to bootstrap.
2.4 Generating Training Problems with Conditional Generative Strategies
In theorem proving, high-quality training problems produced by humans are often not available in quantities even remotely matching the needs of reinforcement learning to properly generalize across problem instances.
A possible solution is to use procedural generation techniques to assemble large datasets of synthetic problems.
This works well in some speciﬁc areas [23] and particularly for problems that can be framed as inverse problems such as symbolic integration [24].
In other areas, generating interesting problems is as hard as solving existing ones, possibly harder.
This is the case in particular with the problem of loop invariant synthesis.
Here, a natural idea for generating problem instances would be to use a probabilistic grammar for repeatedly sampling triples consisting of a program, a loop invariant and an assertion and then reject all triples in which the properties deﬁning valid invariants do not hold.
Unfortunately, not only would doing so naively lead to a very high rejection rate, but the resulting dataset would be heavily biased towards trivial samples that are hardest to reject (e.g. programs where the ﬁnal assertion is the negation of the loop guard and where the invariant does not even matter).
In contrast, many classes of interesting problems from standard human-written benchmarks would only be sampled with an inﬁnitesimal probability.
In this work, we propose to generate problems using the same methodology we use to solve them: by leveraging reinforcement learning to reﬁne nondeterministic search strategies.
Speciﬁcally, experts can deﬁne teacher strategies in the form of stochastic and nondeterministic programs, each run of which either fails or successfully returns with a problem instance.
Reinforcement learning can then be used to reﬁne such programs with the two objectives of maximizing the diversity and interestingness of generated problems while avoiding rejection.
However, these objectives are naturally in tension since an agent may be locally incentivized to avoid generating particular types of problems that are easier to reject.
We introduce a design template for a class of strategies that avoid this obstacle.
We call such strategies conditional generative strategies .
A conditional generative strategy generates a problem in three steps: i) it samples a set of constraints that deﬁne desired features for the generated problem, ii) it nondeterministically generates a problem that respects as many constraints as possible and gets rewarded for doing so and iii) it applies a sequence of random and validity-preserving problem transformations as a way to further increase diversity.
We provide a high-level description of a such a teacher strategy for invariant synthesis in Figure 3.
The interest of such an architecture is that it enables decoupling the two conﬂicting objectives of generating valid and diverse problems: diversity is guaranteed by the ﬁrst and third steps while learning can focus on the second step.
This also allows using the exact same learning algorithm for training both teacher and solver agents (AlphaZero in this work).
2.5 Predicting Events Rather Than Rewards
In the standard AlphaZero setting, the network is trained to predict the value of encountered states, that is the expected sum of future rewards.
However, such a number conﬂates a lot of valuable
1 def teacher(rng: RandGen) -> Prog: cs = sample_constrs(rng) p = generate_prog(cs) p
=
transform(p, rng) p = hide_invariants(p) return p 7 8 def generate_prog(cs: Constrs): p =
Prog(" assume init; while (guard) { invariant inv_lin; invariant inv_aux; invariant inv_main; body; } assert post;") p = refine_guard(p, cs)
p = refine_inv(p, cs)
p = refine_body(p, cs) assert valid(inv_preserved(p))
p = refine_post(p, cs) assert valid(inv_post(p))
p = refine_init(p, cs) assert valid(inv_init(p))
nv = num_violations(p, cs) reward (max(-1.5, -0.5 * nv)) return p 29 30 def transform(p: Prog, rng:
RandGen): p = shuffle_formulas(p, rng) p = shuffle_instrs(p, rng) p =
add_useless_init(p, rng) p
=
add_useless_post(p, rng) ...
return p aux is a conjunction of atomic formulas that can be used in proving inv_main but not post . These three formulas are reﬁned nondeterministically by the call to refine_inv on line 19. After an invariant is chosen, a loop body is also generated nondeterministically (line 20) that preserves the invariant (line 21).
Finally, the num_violations function penalizes the presence of useless or redundant problem components.
For example, a penalty is applied if inv_main features a disjunct that can be removed without invalidating the problem.
Appendix B provides more details on how the refine_* functions can be implemented.
The simplest way to do so is to have a grammar and use the choose operator to select rules recursively.
However, fancier strategies can easily be implemented in the nondeterministic programming framework.
In particular, our implementation uses the abduct procedure introduced in Figure 2 to suggest values for constants and subformulas.
[Table]
The full list of constraints we used in our experiments is available in Appendix B. information. For example, suppose a teacher state is assigned a value of 0. Does 0 mean that the teacher is predicted to either fail or succeed with equal probability or that it will certainly succeed in generating a problem that violates two constraints?
And if so, which two?
Although this information is not relevant to MCTS, there are several reasons for having the network predict it anyway.
First, doing so can result in an increased sample efﬁciency by providing the network with more detailed feedback on its mistakes.
Such an effect has been previously demonstrated in the
KataGo project [25].
Second, it is interesting from an interpretability point of view and makes it easier to diagnose problems and weaknesses in a given network.
We found a third reason that is more subtle, which has to do with getting the combined beneﬁts of issuing intermediate and ﬁnal rewards.
Indeed, in the teacher strategy showed in Figure 3, we only penalize the agent for violating a constraint at the very end .
Intuitively, there is a lost opportunity here since many violations could be detected much earlier while the problem is still being reﬁned.
Emitting an intermediate reward at this point would certainly improve learning efﬁciency.
However, the same violation can be potentially detected at several different points in time and we must ensure that a reward is only issued the ﬁrst time.
This in turns makes the job of the value prediction network harder since it needs to keep track of whether or not each violation has been penalized already.
Encoding this information alone can be costly, and especially so for attention-based network architectures such as transformers with a quadratic inference cost.
Having the network predict constraint violations separately offers an elegant way out: all rewards are issued at the end but from the moment a constraint violation is detected, the network’s predicted probability for this violation is overridden to 1 whenever a value estimate is computed.
More formally, we propose to introduce the concept of an event .
Strategies can declare an arbitrary number of events e and each one is associated with a reward r e along with a maximal number of occurrences m e ( m e = 1 for constraint violation events) that can be counted towards the ﬁnal reward.
The reward operator introduced in Figure 2 is replaced by an event operator.
When a strategy terminates, a reward is issued implicitly with a value of − 1 in case of a failure and
[Equation]
in case of a success.
In this expression, n e denotes the number of calls made to event ( e ) during the whole episode and setting r min > − 1 guarantees that successes are always rewarded more than failures.
Note that it is important not to issue event penalties after a failure.
Otherwise, faced with a likely failure, an agent may be incentivized to simply give up and fail immediately to avoid further penalties in search of success.
Also, there are two reasons for bounding the maximal number of occurrences of event e that are counted towards the ﬁnal reward by a constant quantity m e .
First, this allows having a network with a ﬁnite softmax head predict the number of occurrences of e (as we see below).
Second, the m e = 1 case is particularly useful to model events such as constraint violations that can be detected multiple times but should only be penalized once.
In this framework, the value head of the network is tasked with predicting the following probabilities: i) the probability p 0 of a failure, ii) the probability p 1 of succeeding with minimal reward r min , iii) the probability p 2 = 1 − p 0 − p 1 of succeeding with a greater reward and iv) the probabilities p i e that min( n e , m e ) = i for all events e and 0 ≤ i ≤ m e .
A value estimate derives from these probabilities:
[Equation]
where ˆ p ie ∝ 1 { n e ≤ i } · p ie is a corrected probability estimate accounting for the number of times n e that event ( e ) was raised already.
In our invariant synthesis experiments, we use events in both the teacher and solver strategies to represent constraint violations ( m e = 1) and encourage short proofs by penalizing individual proof steps respectively ( m e > 1) .
Implementation and Engineering Contributions
This paper advocates for a hybrid approach to automated theorem proving where human experts from a variety of areas can formalize their knowledge succinctly in the form of nondeterministic proving and teaching strategies.
Reinforcement learning is leveraged to ﬁll in the blanks in all inevitable cases where the human intuition escapes formalization.
However, for this vision to be practical, writing strategies and iterating on them must be as frictionless as possible.
New tools and abstractions are needed to automate the process of converting nondeterministic programs into reinforcement learning environments, ease debugging and foster code reuse.
In this work, we accomplish a crucial step in this direction by introducing the Looprl theorem prover.
Looprl consists of the following collection of features
: • A domain speciﬁc language for writing strategies embedded in OCaml, with ﬁrst-class support for neural-guided nondeterministic programming.
Strategies are automatically compiled into reinforcement learning environments that can be explored in Python. • A graphical interface that can be used to interact with strategies manually while inspecting the neural network’s predictions and the behavior of MCTS (see Figure 4). • A library of utility functions for writing program veriﬁcation strategies.
This library includes an implementation of the abduct function introduced in Figure 2 for integer linear arithmetic (see details in Appendix F).
• A parallel, high-performance implementation of the (Gumbel) AlphaZero algorithm [7, 21] for reﬁning search strategies.
Our implementation uses Pytorch [26] and Ray [27].
It provides support for events (see Section 2.5) and unbounded, variable-size action spaces.
The ﬁrst point on developing a domain-speciﬁc language for writing strategies is particularly important and deserves more discussion.
In our experience, having such a language is not just a mere matter of convenience but one of feasibility.
In fact, we did try to implement an initial version of a teacher strategy for loop invariant synthesis by writing pseudocode on paper and manually compiling it into an MDP implemented in Python.
However, doing so resulted in a complexity explosion where any single-line change to the pseudocode could take days of work to implement.
The reason compiling a nondeterministic program into an MDP manually is so tedious is that the whole program state must be made explicit, which includes the program stack.
A tempting alternative would be to write nondeterministic code as normal Python code parametrized by an arbitrary choose function.
However, this does not allow using tree search on the resulting program since doing so would require cloning the whole execution state of a Python program.
Fortunately, there exists a solution to this problem in the ﬁeld of programming languages using search monads [28, 10] and which essentially enables writing arbitrary nondeterministic code and then reifying it into a search tree that can be manipulated explicitly.
In Looprl, we implement a domain speciﬁc language embedded in OCaml for expressing nondeterministic strategies.
OCaml is a natural choice because it is fast (about two orders of magnitude faster than Python for symbolic code such as proof inferences) and it has good support for monads and
Python interoperability.
Our language provides built-in support for the choose and event operators.
Also, it deﬁnes an intermediate graph-based format for representing data to be sent to neural networks in an architecture-agnostic way.
Any piece of information that is passed to the choose operator must be convertible to this format and feature suitable metadata ensuring a seamless integration with the
Looprl user interface and debugging tools.
Experiments
We tested Looprl on the problem of synthesizing loop invariants for single-loop imperative programs and evaluated it on the standard Code2Inv [29] benchmark.
This benchmark contains a set of 132 programs written in C. All programs feature a single loop along with a ﬁnal assertion to be proven.
They feature linear integer arithmetic and sometimes involve conditionals, assumptions and nondeterministic tests (some Code2Inv problems are in Appendix C).
Most existing invariant generation tools can only solve a subset of these problems [30].
To the best of our knowledge, only one existing tool can solve them all [31]. However, it does not use learning and relies on speciﬁc optimization techniques that are not generalizable beyond the setting of purely numerical programs.
We used Looprl to implement an invariant generation strategy, which we describe in Appendix A.
In a nutshell, it generalizes the simple strategy described in Figure 2 by adding features such as the ability to abduct disjunctive invariants or to conjecture invariant templates with placeholder constants to be reﬁned later using abduction.
To our surprise and although this generic strategy can be described in only one page of pseudocode, it is sufﬁcient to simply solve every Code2Inv benchmark problem in a few seconds when combined with the vanilla MCTS algorithm
[22] (with no learned heuristic).
We therefore evaluate a self-trained solver agent on its ability to solve as many Code2Inv problems as possible without resorting to any search.
At every decision point, it greedily takes the highest ranked action according to the network’s policy head and no backtracking is allowed.
We argue that such a metric is of particular interest and signiﬁcance.
Indeed, ﬁnding an invariant for a single loop is of limited interest of its own.
Rather, doing so is typically useful as a subtask in a hierarchy of increasingly complex and relevant problems.
The next level of this hierarchy may be to prove a piece of code with nested loops, and then to synthesize a function respecting a speciﬁcation, which is still several levels away from writing full-ﬂedged software systems.
Complex problems cannot be solved if backtracking search is needed at every level of this deep hierarchy.
4.1 Training Protocol
We train a teacher agent and a solver agent in sequence using the (Gumbel) AlphaZero algorithm.
Both agents use a similar 1.6M parameters Dynamic Graph Transformer network [32] (see architecture details in Appendix H).
The teacher agent is trained ﬁrst for 20 AlphaZero iterations.
During each iteration, it goes through 8000 problem generation episodes, using 64 MCTS simulations per step.
Then, the network is updated using samples from the k previous iterations with k increasing fromto 6 during training.
Each iteration simulates 800 additional episodes to generate validation samples that are not used to train the network but to control overﬁtting via early stopping.
After the teacher agent is trained, a dataset of 50K problems is gathered for training the solver, which includes a mix of 40K problems generated during the last ﬁve training iterations and 10K new ones that are generated without Gumbel exploration noise [21].
The solver agent is also trained for 20 iterations.
During each iteration, it attempts to solve 20K randomly selected problems using 32 MCTS simulations per step.
In addition, 5K additional problems are solved to generate validation samples.
See Appendix G for a complete list and explanation of all training hyperparameters.
A complete training run takes about 16h on a single machine with a 10-core Intel i9-10900KF processor, 64GB of RAM and a NVIDIA GeForce RTX 3080 GPU.
4.2 Experimental Results
We show training curves for the solver and teacher agents in Figures 5 and 6 respectively.
These curves record the evolution of the average reward collected by MCTS combined with the latest network during each iteration.
Error intervals are estimated based on two random seeds.
Note that the maximum theoretical reward obtainable by the teacher agent is < 1 since problem generation constraints may be sampled that are mutually incompatible, in which case the agent simply does its best to minimize the number of violated constraints.
The same holds for the solver agent since all
Iteration number − 1 .− 0 .0 .0 .1 .
A v e r ag e r e w a r d
Iteration number 0 .0 .0 .0 .0 .
A v e r ag e r e w a r d
[Table]
generated invariants are penalized proportionally to their size. The solver agent goes through a more modest reward gain during training (the y -axis is rescaled in Figure 6 to emphasize the trend).
This is reﬂective of the fact that most problems generated by the teacher can be solved by MCTS alone and so most of the training concentrates on learning to solve about 10% of hard problems.
We show the results of our evaluation on the Code2Inv benchmark in Table 2.
Three different agents are compared on the average ratio of benchmark problems for which they can successfully generate an invariant of minimal size without search or backtracking.
The ﬁrst agent is a baseline that simply selects proof actions at random.
The second agent is a solver network that was trained using problems generated by an untrained teacher (i.e. using MCTS but no network heuristic).
Finally, the third agent is a solver network trained using the full protocol of Section 4.1.
These results demonstrate the critical impact of learning for both the teacher and solver agents.
Unsurprisingly, an inferior teacher leads to an inferior solver with decreased generalization capabilities.
Related Work
Leveraging nondeterministic programming for proof search. The idea of combining nondeterministic expert strategies with neural oracles was proposed by Selsam [10, 33] as a possible angle for tackling the IMO Grand Challenge [34].
However, how to train such oracles remains an open problem.
One proposal [32] is to use supervised learning for training a universal oracle that is conditioned on the description of arbitrary search problems and can therefore use training data from a large number of heterogeneous sources.
In contrast, we propose using reinforcement learning with the insight that nondeterministic search strategies can be used to generate problems in addition to solving them.
Learning to generate synthetic theorems. Procedural generation techniques have been used for producing synthetic theorems [35, 36, 37, 24, 38].
These usually proceed in a backwards fashion by generating random proof trees from a given set of axioms and rules.
Wang et al. proposed to use supervised learning to learn how to guide this process towards generating more interesting theorems that are similar to those of a reference dataset [6].
Using reinforcement learning for loop invariant synthesis. Reinforcement learning has already been applied to the problem of loop invariant synthesis [29, 30], albeit in a very different way.
In the aforementioned work (which introduces the Code2Inv benchmark), a separate reinforcement learning agent is trained from scratch on every benchmark problem, using counterexamples from an SMT solver as a training signal.
This process takes minutes to hours for each problem to be solved.
In contrast, we train a single agent to generalize across problem instances so that new problems can be solved in a matter of milliseconds.
Using reinforcement learning for theorem proving. The HOList Zero [8] and TacticZero [9] systems use reinforcement learning to learn how to interact with tactic-based theorem provers without relying on human proofs.
However, they still rely on large human-produced corpora of formalized mathematical statements to be used as training tasks and are not yet competitive with approaches based on imitation learning.
The use of reinforcement learning has also been explored in the context of saturation-based or tableau-based provers for ﬁrst-order logic [39, 40].
However, learned heuristics in such settings must operate at a very low-level and are subject to a speed-accuracy tradeoff that is unfavorable to deep learning.
Conclusions and Future Work
This paper took a bold stride toward the challenge of self-learning automated theorem proving without relying on example theorems and proofs.
It advocates a hybrid approach to theorem proving where experts are provided a ﬂexible language to formalize their domain-speciﬁc knowledge in the form of generic nondeterministic teacher and solver strategies, leaving blanks to be ﬁlled by learning.
We demonstrated our framework by applying it to the problem of loop invariant synthesis.
Our agent solves all Code2Inv challenges.
More importantly, it learns to solve a majority of problems with no search at all despite never seeing these problems during training.
None of our core contributions, however, are speciﬁc to invariant synthesis.
These include: i) our key insight that nondeterministic programming and reinforcement learning can be similarly combined to implement solvers and teachers, ii) conditional generative strategies as a general template to write teachers, iii) abductive reasoning as a design principle that is made scalable by self-learned guidance, iv) strategy events for easier reward engineering and better sample-efﬁciency, and v) an implementation that establishes engineering foundations for making the whole approach practical.
We see this paper as a ﬁrst step toward a broader vision where interactive theorem provers allow users to write teacher and solver strategies for domains of mathematics and programming in a modular and distributed fashion.
A unique neural network can then be trained as an oracle for all these strategies, allowing better generalization and sample efﬁciency.
However, many challenges lie ahead.
In particular, while there is some evidence already [10, 41, 19] that effective solver strategies can be written at scale (interactive theorem provers already allow specifying custom search tactics [42, 43, 44] and we are merely suggesting a generalization of this popular feature), scaling up teacher strategies raises more questions.
For example, writing conditional generative strategies can be engineering-intensive since diversity is enforced extrinsically via manually-deﬁned constraints.
Future work will explore teachers that also integrate intrinsic reward signals, either via a curiosity mechanism or by having the solver directly reward the teacher for producing problems of suitable novelty and difﬁculty.
Broader Impact
Automated theorem proving for program veriﬁcation and program synthesis plays an important role for increasing the safety of critical pieces of software.
More automatic theorem provers enable a wider user base to beneﬁt from the advances to their software quality.
But automating theorem provers is a highly sophisticated challenge.
Learning from succinct expert advice in the form of strategies may be the sweet spot enabling more generic domain-speciﬁc automation for theorem provers. Acknowledgments and Disclosure of Funding This work was supported by the Alexander von Humboldt Professorship program, the National
Science Foundation under Award CNS-1739629 and by DARPA under Contract No. FA8750-18C-0092.
We thank Ruben Martins, Paul Gölz, Vincent Hellendoorn, Léonard Blier along with the anonymous reviewers at NeurIPS 2022 for their helpful comments and feedback.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In Figure 7, we provide some detailed pseudocode for the solver strategy that we implemented for the purpose of the experiments in Section 4. This strategy is similar to the simple one introduced in
{+Figure 
+}
but it comes with the following extensions: • Ability to abduct disjunctive invariants: Imagine a proof obligation fails and the abduct function returns a 1 , . . . , a n as possible missing assumptions. In some cases, none of those can be proved invariant but the disjunction of a subset of them can.
This is why the suggest_missing function deﬁned on line 28 is used to build an invariant candidate as a disjunction of at most 3 (atomic) abduction candidates.
• Ability to strengthen abducted invariants: An abducted invariant candidate may have to be strengthened before it can be proved invariant.
For example, the abduction engine may suggest x (cid:54) =
as a missing assumption but x (cid:54) = 0 may not be a valid invariant whereas x > 0 (or x < 0 ) is.
The call to strengthen on line 47 is used to optionally and nondeterministically strengthen an invariant candidate.
Our current strategy supports two kinds of strengthening: replacing a formula of the form A (cid:54)
=
B by either A > B or A < B or weakening an inequality of the form A ≤ B into A ≤ B + c ? where c ? is a nonnegative constant whose exact value is to be determined later (see next point).
• Ability to instantiate constants lazily via abduction: Invariant candidates can feature metavariables that denote unknown constants.
The constrs variable deﬁned in linecollects global constraints about these metavariables.
If a missing assumption is suggested that only features metavariables, then it is added as a constraint rather than as a new invariant candidate (line 42).
After an invariant is proved, the metavariables it contains are instantiated using concrete values in a way to satisfy all global constraints (see call to abduct_refinement on line 57).
A metavariable that appears in the invariant as an upper bound is instantiated with a value that is as low as possible, whereas a metavariable that appears as a lower bound is instantiated with a value that is as high as possible (this information is contained in the btype variable).
• Ability to conjecture invariant templates: In some cases, abduction alone is not enough to discover a missing invariant.
{+For example, if two disjunctive invariants I 
+}
and I 2 are needed for proving the postcondition, one may have to conjecture the ﬁrst one and then use abduction to ﬁnd the other.
The strategy in Figure 7 allows conjecturing invariant templates with metavariables to be instantiated via abduction (see previous point).
The conjectures function called on line 29 returns three kinds of conjecture candidates: i) conjectures of the form t (cid:12) c ?
where (cid:12) ∈ {≥ , = } and t = (cid:80) i a
i x
i is a linear combination of variables that is preserved by the loop body, ii) relaxations of the loop guard (e.g. x ≤ 10 + c ? with c ?
>if the loop guard is x ≤ 10 ) and iii) initial assumptions that only contain variables that are not modiﬁed by the loop body.
The solver strategy also emits two types of events (see Section 2.5) at lines 38 and 36 respectively.
Conjecturing events are associated with a reward of − 0 .
3 and abduction events are associated with a reward of − 0 .
Both kinds of events are counted at most four times ( m e = 4 ) and the minimum total reward delivered in case of a success is r min = 0 .
Hints on how to use our proposed solver strategy to solve Code2Inv benchmark problems are available in Appendix C.
B Details on the Teacher Strategy
The teacher strategy we use for loop invariant generation follows the structure introduced in Figure 3.
We provide additional details below:
• Sampling constraints:
The full list of available constraints is available in Table 3.
Values are sampled mostly independently for each constraint types.
In our implementation, we hardcode a small number of correlations (e.g. we are more likely to sample a disjunctive formula for inv_main if inv_lin is not used in order to keep things interesting).
We also reject a number of constraint combinations that are clearly uninteresting or unsatisﬁable.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
else :
[Figure]
[Figure]
(f
[Figure]
Ge
[Figure]
Metavar(c)
[Figure]
[Figure]
• Fixed constraints:
In addition to penalizing the violation of sampled constraints, the teacher implements ﬁxed hard constraints that are always enforced.
A list of all such constraints is available in Table 4.
Violating one of these constraints leads to an immediate failure along with a reward of -1.
• Reﬁning formulas and using abduction: Different parts of the problem template shown in Figure 3 are nondeterministically reﬁned in turn.
To reﬁne an atomic formula, a template is ﬁrst selected of the form x ? (cid:12) c ? or x ? (cid:12) y ?
where x ? and y ? are variable placeholders, c ? is a constant placeholder and (cid:12) ∈ { <, ≤ , >, ≥ , = , (cid:54) = } .
Each variable placeholder is then nondeterministically substituted by an existing or a fresh variable.
Constant placeholders are either instantiated with concrete constants (a set of 6 available numerical constants is sampled at the start of the teacher strategy along with constraints) or parameters (special variables that cannot be modiﬁed by the program).
Constant placeholders can also be left as-is and reﬁned later using abduction (e.g. before invariant preservation is checked, abduction is used to suggest values for the remaining constant placeholders).
Abduction is also used to suggest required parameter assumptions that are added to init (e.g. n >where n is a variable not modiﬁed in the program).
• Reﬁning programs: Subprograms are reﬁned by selecting a sequence of assignment templates of the form: x ? :
= c ? , x ? : = y ? , x ? :
= x ?
+ d ? , x ? :
= x ? − d ? , x ? :
= x ?
+ y ?
and x ? :
= c ?
− y ?
( d ? is a placeholder for a strictly positive constant).
A special skip template can be selected to stop adding assignments.
Variable and constant placeholders are handled in the same way they are handled in formulas.
Which templates are available is determined by the assignment-templates constraint (see Figure 3).
• Extra reﬁnement suggestions: Extra suggestions are added to the standard templates when reﬁning some formulas.
When adding a disjunct to the main invariant, the loop guard itself is added as a suggestion along with a relaxed version of it (using a placeholder constant).
When reﬁning the last disjunct of the postcondition, abduction is used to suggest candidates that are consequences of the current assumptions in the associated proof obligation.
Abduction is also used to suggest conjuncts for init .
• Detecting constraint violations early: Constraint violations are detected as early as possible to allow early feedback during search.
For example, whether or not the invariant is satisﬁable is checked right after the invariant has been reﬁned and before the loop body is reﬁned in turn.
Most constraints must be checked again whenever a new parameter assumption is added.
For example, an invariant 0 ≤ x ∧ x < n may be initially judged as satisﬁable.
However, abduction may later on suggest n < 0 as a parameter assumption, making it unsatisﬁable.
• Applying random transformations to generated problems: For increased diversity, a sequence of random transformations is applied to any problem generated by the teacher before it is returned.
We provide a list of all such transformations in Table 5.
C Examples of Code2Inv Problems
We show examples of Code2Inv benchmark problems in Table along with some hints on how they can be solved using the strategy detailed in Appendix A.
D Examples of Generated Teacher Problems
We show examples of challenge problems generated by our trained teacher agent in Table 7.
E Looprl UI Screenshots
We show examples of using the Looprl user interface to inspect the solver and teacher strategies in
Figures 8 and 9 respectively.
F Implementing Abduction
Both the teacher and solver strategies we use as examples in this paper rely on an abduct function that takes as an input a formula F and then either proves it valid or returns a (possibly empty) set of assumptions
A such that A → F can be proved to be valid.
Implementing such a function is hard in the general case and so an implementation of abduct may leverage nondeterminism.
However, because we are only dealing with arithmetic of limited complexity in this work, we implemented a fully-speciﬁed abduction procedure for linear integer arithmetic that relies on Fourier-Motzkin elimination [45].
When given a formula F , our abduction procedure ﬁrst rewrites it in conjunctive normal form as
F = (cid:86) ni
=1 (cid:87) j F ij where F ij are atomic formulas of the form (cid:80) k a k x k (cid:12) c where (cid:12) ∈ {≥ , = } and c, a k are integer constants.
F.1 Case where n =
If F can be expressed as a disjunction of atomic formulas, we can compute abduction suggestions as follows: i) one considers the negation of F , obtaining a set of atomic assumptions and ii) one
[Table]
[Table]
[Figure]
[Table]
derives as many consequences as possible from those assumptions using Fourier-Motzkin elimination (linearly combining inequalities so as to eliminate variables).
If a contradiction is derived, then F is valid.
If no contradiction is dervied given some timeout, then the negation of any derived consequence can be considered as an abduction candidate.
{+Example Suppose we want to compute abduction candidates for F = x ≥ 
+}
→ x + y ≥ 1 .
The conjunctive normal form of F is ( x < 0 ∨ x + y ≥ 1) .
{+Taking the negation yields ( x ≥ 
+}
∧ x + y < 1) , which we normalize into the set of assumptions { x ≥ 0 , − x − y ≥ 0 } (all variables are integers).
Then, we can take a Fourier-Motzkin step by adding these two assumptions and derive the following consequence: − y ≥ 0 .
After this, no other reasoning step is applicable end we end up with the following set of facts: { x ≥ 0 , − x − y ≥ 0 , − y ≥ 0 } .
We therefore suggest the following abduction candidates: x < 0 , x + y > 0 and y > 0 .
F.2 Case where n >
If the conjunctive normal form of F has two conjuncts or more, we apply the procedure above on each conjunct separately.
For example, suppose that F = G ∧ H , G admits a set of abduction candidates { A i } i and H admits a set of abduction candidates { B i } i .
One possibility would be to return all { A i ∧ B j } ij combinations as abduction candidates for F .
However, doing so can quickly result in a combinatorial explosion.
Therefore, our implementation does something different and returns the union of the { A i } i and { B i } i instead.
Doing so, it cannot provide the guarantee that any resulting abduction candidate A is sufﬁcient in implying F .
Rather, abduction candidates are seen as suggestions to unblock one part of the proof (i.e. enable proving one conjunct of F ) but not necessarily the whole proof.
G Training Hyperparameters
We provide an exhaustive list of all hyperparameter values used in our experiments in Table 8. x = 0; while (x < 5) { x = x + 1; if (y > z) { y = z; } } assert y < = z; Problem
Invariant: x < 5 ∨ y ≤ z .
This problem can be solved using our proposed strategy by directly abducting the correct disjunctive invariant when attempting to prove the postcondition.
assume x <= 10; assume y >= 0; while (*) { x = x + 10; y = y + 10; } assume x == 20; assert y !
= 0;
Problem
Invariant: x − y ≤ 10 .
The star ( * ) corresponds to a nondeterministic boolean value.
In this case, the loop body can be executed an arbitrary number of times.
This problem can be solved by conjecturing an invariant of the form x − y ≤ c ?
and then using abduction to reﬁne c ? .
assume n >= 0; i = 0; x = 0; y = 0; while (i < n) { i
= i + 1; if (*) { x = x + 1; y = y + 2; } else { x = x + 2;
y = y + 1; } } assert 3*n ==
x + y;
Problem
Invariant: 3 i = x + y ∧ i ≤ n .
This problem can be solved by conjecturing an invariant of the form 3 i − x − y =
c ? , reﬁning c ? through abduction and then abducting i ≤ n as a missing invariant while trying to prove the postcondition.
s = 0;
i = 1; while i <= n: i = i +s = s +assume s !
=assert s == n
Problem
Invariant: i − s = 1 ∧ ( i ≤ n + 1 ∨ s = 0) .
This problem can be solved by ﬁrst conjecturing an invariant of the form i − s = c ? , then using abduction to reﬁne c ? and ﬁnally abducting the disjunctive invariant i ≤ n + 1 ∨ s = 0 while trying to prove the postcondition
[Table]
[Table]
body-structure no-cond loop-guard-useful-for-post available-consts -9 -6 4assume
y =
=
x; while (x < 1) { invariant y =
= x; y = y + 1; x = x + 1; } assert y > 0;
In this example, the network has been tasked to generate an example of a program with a single atomic invariant and a loop guard that is useful to establish the postcondition but not the invariant itself.
[Table]
{+use-aux-invbody-structure no-cond allow-vcomp-in-prim-inv no-var-const-assign available-consts -8 -
+}
2assume x < y; assume x >= 5; while (*) { invariant x < y; invariant y >= 6 && x >= -1; x = x + 6; y
= y + x; } assert x < y;
In this example, the network has been tasked to generate an example that involves an auxilliary event with two conjuncts.
The auxilliary event can be useful to prove the main invariant but not the postcondition.
Assignments of the form x = y or x = c where x and y are variables and c is a constant are not allowed.
[Table]
disjunctive-post body-structure no-cond only-constr-incr available-consts
-55 -52 21assume x == 21; assume y
==
-52; while (*) { invariant -x - 3*y == 135;
y = y - 21; x = x + 63; } assert y !
=
21 || x =
=
In this example, the network has been tasked to ﬁnd an example of a problem involving a linear invariant with two variables, no loop guard and a disjunctive postcondition.
Note that an irrelevant loop guard may be added in the ﬁnal transformation stage of the teacher.
for clarity but those should of course be hidden before problems are sent to the solver agent.
These invariants only provide one way to solve the associated problems and alternative invariants may exist.
We disable the ﬁnal random transformations applied by the teacher for clarity and to avoid clutter from useless formulas and instructions.
Finally, some problem constraints are not listed for brevity unless their value is different from the default with highest probability.
[Figure]
[Table]
[Table]
[Table]
[Table]
Value estimates for unvisited children are de- creased by this value so as to encourage deeper search trees (see LC0 AlphaZero implementa- tion).
[Table]
Whether the [Table] an environ- ment [Table]
is taken
[Table]
Maximal size of the MCTS tree.
This parameter is relevant to avoid out-of-memory errors when reset-tree if false.
[Table]
The number of previous iterations from which samples are used to update the network at each iteration.
If the provided list is shorter than the total number of iterations, the last number in this list is duplicated as many times as necessary
[Table]
Hyperparameters for the solver agent
[Table]
AlphaZero agents
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Value estimates
[Table]
unvisited
[Table]  The maximum number of training epochs. [Table] H Network Architecture and Choice Points Encoding H.1 Network Architecture
Neural networks that are used as oracles for nondeterministic strategies take as an input a choice point and return i) a probability distribution over all available choices, ii) some success and failure probability estimates and iii) some event occurence probability estimates (see Section 2.5).
In turn, a choice point is represented as i) a probe [10] that encodes all relevant state information provided to choose (see Figure 7 for examples) and ii) a list of possible choices that were passed as arguments to choose .
Our proposed network architecture works as follows.
Given a choice point, the probe is encoded using a Dynamic Graph Transformer (DGT) neural network [32].
DGT networks are similar to
Transformers [46] but they allow leveraging some additional graph structure over the source tokens by associating edge types to learned attention biases.
Each choice is encoded separately using another
DGT.
It is then concatenated with the probe encoding and passed to a combiner network that outputs a score. Scores are normalized into a probability distribution using a softmax operation. The probe encoding is also passed to a value head that produces outcome and event predictions. Batches of choice points can be evaluated efﬁciently in parallel using scatter operations [47].
H.2 Encoding Programs and Formulas
All data that is to be passed to the neural network must be encodable into a sequence of tokens with an optional graph structure.
This is the case of programs and formulas in particular.
We use standard techniques to encode those [48, 49]:
• Abstract syntax trees (ASTs) are encoded into a sequence of tokens in polish notation order (e.g. x + 3 y is encoded as “ PLUS VAR(x) MUL CONST(3) VAR(y) ”).
The edges in the original AST are preserved as graph edges to be passed to the DGT network.
[Table]
• We use a different positional encoding scheme for tokens that leverages the tree structure of the underlying AST [49].
Doing so has been demonstrated to result in better generalization capabilities [50].
• Identiﬁer names are randomly mapped into a ﬁnite number of unique ids for each choice point.
Unique ids are encoded using a one-hot encoding scheme.
If a choice point introduces more names than there are unique ids available (rare), the last occuring names are made to share a single uid that is ﬂagged to indicate a naming conﬂict.
• Numerical constants with an absolute value greater than 3 are represented with generic POS_CONST and NEG_CONST tokens.
A binary encoding of their value is also provided to the network.
Moreover, special edges are added to compare all numerical constants involved in a program or formula (see Table 9).
• Following [48], we also add semantic edges to the encoding of programs and formulas that reﬂect the way information ﬂows within them.
Details can be consulted in Table 9.