Learning to Find Proofs and Theorems by Learning to Reﬁne Search Strategies
Abstract
We propose a new approach to automated theorem proving and deductive programsynthesis where an AlphaZero-style agent is self-training to reﬁne a high-levelexpert strategy expressed as a nondeterministic program.
An analogous teacheragent is self-training to generate tasks of suitable relevance and difﬁculty for thelearner.
This allows leveraging minimal amounts of domain knowledge to tackleproblems for which training data is unavailable or hard to synthesize.
We illustrateour approach on the problem of loop invariant synthesis for imperative programsand using neural networks to reﬁne both the teacher and solver strategies
Context and Motivation
Augmenting tactic-based interactive theorem provers with neural guidance has been the focus ofincreased attention in recent years [1, 2, 3, 4, 5].
The dominant approach uses imitation learning onlarge corpora of formalized mathematics.
However, despite recent efforts involving self-supervisedpre-training [5] or data-augmentation [6], this approach is limited by the conspicuous scarcity ofhuman-produced training data.
An alternative approach inspired by the success of AlphaZero [7] isto use reinforcement learning and let an agent self-train to interact with a theorem prover via trial anderror [8, 9].
However, previous attempts of doing so have been hampered by two fundamental issues:
• First, existing tactic-based theorem provers offer inﬁnite action spaces that are hardly
amenable to random exploration.
Also, they are optimized for formalizing the outcome
of human insights
concisely but often fail to deﬁne a tractable search space for deriving
those insights in the ﬁrst place [10].
For example, using tactics effectively often requires
providing insights in advance that are more easily found later in the search process (e
g.constructing an object to instantiate an existential quantiﬁer [11]).
More generally, most
of the human process of discovering proofs, which involves trial and error, sketching
and abductive reasoning is not captured by standard prover tactics and so we can hardly
expect a reinforcement learning agent to learn this process through sheer interaction with a
tactic-based prover
• Second, although reinforcement learning alleviates the need for human proofs during training,
the agent must still be provided learning tasks of suitable relevance, diversity, difﬁculty and
generalizability in the form of theorem statements to be proved.
This is in contrast with
applications of AlphaZero in
board games
, where the symmetric nature of games such as
Chess or Go enables leveraging symmetric self-play as an inﬁnite source of training tasks
In this paper, we propose a novel approach to learning automated theorem proving and deductiveprogram synthesis that does not rely on human-provided proofs and theorems.
Instead, a teacher and asolver agent are trained in a self-supervised manner to generate and prove theorems respectively.
Both agents leverage domain-speciﬁc expert knowledge in the form of high-level search strategies [10].
assume x >= 1; y = 0; while (y < 1000) { x = x + y;
y = y + 1; } assert x >=
y;
A desirable invariant is a formula I [ x, y ] such that: ∀ x, y   x ≥ 1 ∧ y =
→ I [ x, y ] (holds initially)
y <
1000 ∧ I [ x, y ]
→ I [ x + y, y + 1] (preserved) y ≥ 1000 ∧ I [ x, y ] → x ≥ y (implies post)
Such an invariant is I [ x, y ] :=
( x ≥ y ∧ x ≥
∧ y ≥ 0)
These strategies are expressed as nondeterministic programs and they deﬁne tractable search spacesfor reinforcement learning to explore.
Approach
In this paper, we explore and demonstrate our approach in a well-contained yet challenging setting,namely the veriﬁcation of single-loop imperative programs.
However, none of our core contributionsare speciﬁc to this scenario.
2.1 Background: Verifying Imperative Programs with Loops
Suppose we are given a program such as the one in Figure 1 and we want to prove that the ﬁnalassertion always holds.
The way to proceed is to ﬁnd a predicate called a loop invariant with thefollowing properties: i) it is true before the loop, ii) it is preserved by the loop body when the loopguard holds and iii) it implies the ﬁnal assertion when the loop guard does not hold.
By “preserved”,we mean that if I is true before executing the loop body then it is also true afterwards.
Finding loopinvariants is the most crucial aspect of program veriﬁcation [12, 13] and still resists automation [14].
Despite the difﬁculty of automatically synthesizing loop invariants, humans do so routinely usingwell-understood search strategies.
For example, in the case of the example in Figure 1, one wouldﬁrst try to prove the postcondition x ≥ y itself as an invariant and then note that it is not preserved bythe loop body as the following implication does not hold: y < 1000 ∧ x ≥ y (cid:54)→ x + y ≥ y + 1 .
However, simplifying the right-hand side, we see that the proof works if we can show x ≥ 1 to be aninvariant itself.
Using a similar form of abductive reasoning, we ﬁnd that this in turn requires y ≥ 0to be an invariant and we end up proposing x ≥ y ∧ x ≥ 1 ∧ y ≥ 0 as a loop invariant.
2.2 Expressing Search Strategies as Nondeterministic Programs
The search strategy we followed for the example above can be formalized as a nondeterministicprogram, which is shown in Figure 2.
As a nondeterministic program, it features a choose operatorthat takes a list of objects as an input and selects one of them nondeterministically.
A nondeterministicprogram can be reﬁned into a deterministic one by providing an external oracle to implement thechoose operator.
It also deﬁnes a search tree that can be explored with or without a guiding heuristic.
In this work, we use neural networks to implement such oracles and guiding heuristics.
In addition, a key aspect of this strategy is to infer missing assumptions under which a currentlyfailing proof obligation would hold.
This form of reasoning is called abductive reasoning and it isfundamental in the way humans search for proofs.
It is implemented in a separate abduct procedurethat takes a formula as an input and returns either Valid or a list of abduction candidates.
For example,the abduct procedure fails to prove the implication x ≥
→ x + y ≥ 1 but may suggest x + y ≥ 1 ,x < 0 and y ≥ 1 as possible missing assumptions.
The use of abductive reasoning for theoremproving and loop invariant synthesis speciﬁcally has been proposed in the past [16, 17].
However,abduction procedures are hard to implement [18] and typically only available for speciﬁc decidabletheories. Also, using them in proof search tends to scale poorly in the absence of good heuristics torank and ﬁlter abduction candidates. Our proposed framework makes abductive reasoning practicalby addressing both issues: it allows leveraging self-learned guidance
1 {+def solver( init: Formula, guard: Formula, body: Program, post: Formula) -> Formula: 
 : return [+}
[Figure]
[Figure] [Figure]
[Figure]
See Appendix A for more details and for a full listing of the invariant synthesis strategy that we use in our experiments. and it allows implementing abduction procedures as nondeterministic programs that are amenable tolearning themselves.
Another notable feature of the strategy in Figure 2 is the use of the reward operator on line 17 toincentivize ﬁnding short invariants (short proofs are desirable in general as they tend to be easier tocheck, interpret and generalize). The reward operator can be used multiple times and at any point ofthe strategy execution.
An implicit reward of 1 or -1 is emitted when the strategy successfully returnsor fails respectively.
An optimal execution of a nondeterministic strategy is one that maximizesthe cumulative amount of collected rewards.
In this example, we bound the maximal invariant sizepenalty in such a way to ensure that ﬁnding a proof is always rewarded more than failing at doing so.
As argued by Selsam [10], deﬁning search strategies as nondeterministic programs provides a naturaland ﬂexible way for experts to leverage domain-speciﬁc knowledge while outsourcing difﬁcult proofdecisions to search algorithms and learned heuristics.
This paradigm differs from the standardparadigm of tactic-based theorem provers in which an external entity must orchestrate statelesstactics that do not call for any form of user interaction internally.
In contrast, the nondeterministicprogramming approach inverts control and has strategies call external oracles rather than the otherway around, which allows for a much tighter control of the resulting search space.
2.3 Reﬁning Nondeterministic Strategies with Reinforcement Learning
A nondeterministic program induces a (deterministic) Markov Decision Process (MDP) whereintermediate states are choice points and ﬁnal states are either success states (the program returnssuccessfully) or failure states (an assertion is violated or choose is called on an empty list of choices).
Standard search algorithms can be used to navigate this MDP but doing so in an efﬁcient and scalableway requires strong heuristics for guiding search.
In this work, we propose to learn such heuristics in a self-supervised fashion using the AlphaZeroalgorithm [7, 19].
In AlphaZero, a neural network is trained to map any state to a probabilitydistribution over available actions along with a value estimate (i.e. an estimate of the expected sum offuture rewards).
The neural network alone can be used as a policy for navigating the MDP.
However, astronger policy results from combining the network with a tree search algorithm such as Monte-Carlo
Tree Search (MCTS) [20].
The key insight underlying AlphaZero is that the network can be trainedvia an iterative improvement process where it is successively i) used as an MCTS heuristic to try andsolve problems and then ii) updated using gradient descent so as to better predict the outcome ofeach attempt along with the action selected by MCTS on all states encountered along the way.
Moredetails on AlphaZero can be found in the literature [7].
Using AlphaZero, we can reﬁne nondeterministic proof search strategies without external proofexamples to learn from.
However, AlphaZero must still be provided with a set of training problemsto be solved.
Training problems should be diverse, relevant and numerous enough to allow propergeneralization.
They should also be of varied difﬁculty to allow for learning to bootstrap.
2.4 Generating Training Problems with Conditional Generative Strategies
In theorem proving and program synthesis, high-quality training problems produced by humans arenot available in quantities even remotely matching the needs of reinforcement learning to properlygeneralize across problem instances.
A possible solution is to use procedural generation techniquesto assemble large datasets of synthetic problems.
This works well in some speciﬁc areas [21] andparticularly for problems that can be framed as inverse problems such as symbolic integration [22].
In other areas, generating interesting problems is as hard as solving existing ones, possibly harder.
This is the case in particular with the problem of loop invariant synthesis.
Here, a natural idea forgenerating problem instances would be to use a probabilistic grammar for repeatedly sampling triplesconsisting of a program, a loop invariant and an assertion and then reject all triples in which theproperties deﬁning valid invariants do not hold.
Unfortunately, not only would doing so naively leadto a very high rejection rate, but the resulting dataset would be heavily biased towards trivial samplesthat are hardest to reject (e.g. programs where the ﬁnal assertion is the negation of the loop guard andwhere the invariant does not even matter).
In contrast, many classes of interesting problems fromstandard human-written benchmarks would only be sampled with an inﬁnitesimal probability.
In this work, we propose to generate problems using the same methodology we use to solve them: byleveraging reinforcement learning to reﬁne nondeterministic search strategies.
Speciﬁcally, experts can deﬁne teacher strategies in the form of stochastic and nondeterministic programs, each run of which either fails or successfully returns with a problem instance. Reinforcementlearning can then be used to reﬁne such programs with the two objectives of maximizing the diversityand interestingness of generated problems while avoiding rejection. However, these objectives arenaturally in tension since an agent may be locally incentivized to avoid generating particular types ofproblems that are easier to reject.
We introduce a design template for a class of strategies that avoidthis obstacle. We call such strategies conditional generative strategies .
A conditional generative strategy generates a problem in three steps: i) it samples a set of constraintsthat deﬁne desired features for the generated problem, ii) it nondeterministically generates a problemthat respects as many constraints as possible and gets rewarded for doing so and iii) it applies asequence of random and validity-preserving problem transformations as a way to further increasediversity.
We provide a high-level description of a such a teacher strategy for invariant synthesisin Figure 3.
The interest of such an architecture is that it enables decoupling the two conﬂictingobjectives of generating valid and diverse problems: diversity is guaranteed by the ﬁrst and thirdsteps while learning can focus on the second step. This also allows using the exact same learningalgorithm for training both teacher and solver agents (AlphaZero in this work).
2.5 Predicting Events Rather Than Rewards
In the standard AlphaZero setting, the network is trained to predict the value of encountered states,that is the expected sum of future rewards.
However, such a number conﬂates a lot of valuableinformation.
For example, suppose a teacher state is assigned a value of 0.
Does 0 mean that the
1 def teacher(rng: RandGen) -> Prog: cs = sample_constrs(rng) p = generate_prog(cs) return transform(p, rng) 5 6 def generate_prog(cs: Constrs): p =
Prog(" assume init; while (guard) { invariant inv_lin; invariant inv_aux; invariant inv_main; body; } assert post;") p = refine_guard(p, cs)
p = refine_inv(p, cs)
p = refine_body(p, cs) assert valid(inv_preserved(p))
p = refine_post(p, cs) assert valid(inv_post(p))
p = refine_init(p, cs) assert valid(inv_init(p))
nv = num_violations(p, cs) reward (max(-1.5, -0.5 * nv)) return p 27 28 def transform(p: Prog, rng:
RandGen): p = shuffle_formulas(p, rng) p =
add_useless_init(p, rng) p
=
add_useless_post(p, rng) ...
p = hide_invariants(p, rng) return p post .
The num_violations function also penalizes the presence of useless or redundant problem components.
For example, a penalty is applied if inv_main features a disjunct that can be removed without invalidating the problem.
Appendix B provides more details on how the refine_* functions can be implemented.
The simplest way to do so is to have a grammar and use the choose operator to select rules recursively.
However, fancier strategies can easily be implemented in the nondeterministic programming framework.
In particular, our implementation uses the abduct procedure introduced in
Figure 2 to suggest values for constants and subformulas. [Table]
The full list of constraints we used in our experiments is available in Appendix B. teacher is predicted to either fail or succeed with equal probability or that it will certainly succeed ingenerating a problem that violates two constraints?
And if so, which two?
Although this information is not relevant to MCTS, there are several reasons for having the networkpredict it anyway. First, doing so can result in an increased sample efﬁciency by providing the networkwith more detailed feedback on its mistakes. Such an effect has been previously demonstrated in the KataGo project [23].
Second, it is interesting from an interpretability point of view and makes iteasier to diagnose problems and weaknesses in a given network.
We found a third reason that is moresubtle, which has to do with getting the combined beneﬁts of issuing intermediate and ﬁnal rewards.
Indeed, in the teacher strategy showed in Figure 3, we only penalize the agent for violating a constraintat the very end .
Intuitively, there is a lost opportunity here since many violations could be detectedmuch earlier while the problem is still being reﬁned.
Emitting an intermediate reward at this pointwould certainly improve learning efﬁciency.
However, the same violation can be potentially detectedat several different points in time and we must ensure that a reward is only issued the ﬁrst time.
Thisin turns makes the job of the value prediction network harder since it needs to keep track of whetheror not each violation has been penalized already.
Encoding this information alone can be costly,and especially so for attention-based network architectures such as transformers with a quadraticinference cost.
Having the network predict constraint violations separately offers an elegant way out:all rewards are issued at the end but from the moment a constraint violation is detected, the network’spredicted probability for this violation is overridden to 1 whenever a value estimate is computed.
More formally, we introduce the concept of an event .
Strategies can declare an arbitrary number ofevents e and each one is associated with a reward r e along with a maximal number of occurrences m e( m e = 1 for constraint violation events) that can be counted towards the ﬁnal reward.
The rewardoperator introduced in Figure 2 is replaced by an event operator.
When a strategy terminates, areward is issued implicitly with a value of − 1 in case of a failure and
[Equation]
in case of a success.
In this expression, n e denotes the number of calls made to event ( e ) duringthe whole episode and setting r min > − 1 guarantees that successes are always rewarded more thanfailures.
Note that it is important not to issue event penalties after a failure.
Otherwise, faced with alikely failure, an agent may be incentivized to simply give up and fail immediately to avoid furtherpenalties in search of success.
Also, there are two reasons for bounding the maximal number ofoccurrences of event e that are counted towards the ﬁnal reward by a constant quantity m e .
First, thisallows having a network with a ﬁnite softmax head predict the number of occurrences of e (as we seebelow).
Second, the m e = 1 case is particularly useful to model events such as constraint violationsthat can be detected multiple times but should only be penalized once.
In this framework, the value head of the network is tasked with predicting the following probabilities:i) the probability p 0 of a failure, ii) the probability p 1 of succeeding with minimal reward r min , iii) theprobability p 2 = 1 − p 0 − p 1 of succeeding with a greater reward and iv) the probabilities p ie thatmin( n e , m e ) = i for all events e and 0 ≤ i ≤ m e .
A value estimate derives from these probabilities:
[Equation]
where ˆ p ie ∝ 1 { n e ≤ i } · p ie is a corrected probability estimate accounting for the number of times n ethat event ( e ) was raised already.
In our invariant synthesis experiments, we use events in both theteacher and solver strategies to represent constraint violations ( m e = 1) and penalize proof stepsrespectively ( m e > 1) .
Implementation and Engineering Contributions
This paper advocates for a hybrid approach to automated theorem proving where human experts froma variety of areas can formalize their knowledge succinctly in the form of nondeterministic provingand teaching strategies.
Reinforcement learning is leveraged to ﬁll in the blanks in all inevitable caseswhere the human intuition escapes formalization.
However, for this vision to be practical, writing strategies and iterating on them must be as frictionlessas possible.
New tools and abstractions are needed to automate the process of converting nondeterministic programs into reinforcement learning environments, ease debugging and foster code reuse.
In this work, we accomplish a crucial step in this direction by introducing the Looprl theorem prover.
Looprl consists of the following collection of features:
• A domain speciﬁc language for writing strategies embedded in OCaml, with ﬁrst-classsupport for neural-guided nondeterministic programming.
Strategies are automaticallycompiled into reinforcement learning environments that can be explored in Python.• A graphical interface that can be used to interact with strategies manually while inspectingthe neural network’s predictions and the behavior of MCTS (see Figure 4).• A library of utility functions for writing program veriﬁcation strategies.
This library includesan implementation of the abduct function introduced in Figure 2 for integer linear arithmetic(see details in Appendix F).•
A parallel, high-performance implementation of the (Gumbel) AlphaZero algorithm [7, 19]for reﬁning search strategies.
Our implementation uses Pytorch [24] and Ray [25].
Itprovides support for events (see section 2.5) and unbounded, variable-size action spaces.
The ﬁrst point on developing a domain-speciﬁc language for writing strategies is particularly importantand deserves more discussion.
In our experience, having such a language is not just a mere matterof convenience but one of feasibility.
In fact, we did try to implement an initial version of a teacherstrategy for loop invariant synthesis by writing pseudocode on paper and manually compiling it intoan MDP implemented in Python.
However, doing so resulted in a complexity explosion where anysingle-line change to the pseudocode could take days of work to implement.
The reason compiling a nondeterministic program into an MDP manually is so tedious is that thewhole program state must be made explicit, which includes the program stack.
A tempting alternativewould be to write nondeterministic code as normal Python code parametrized by an arbitrary choosefunction.
However, this does not allow using tree search on the resulting program since doing sowould require cloning the whole execution state of a Python program.
Fortunately, there exists asolution to this problem in the programming languages community, which involves a construct calleda search monad [26, 10] and which essentially enables writing arbitrary nondeterministic code andthen reifying it into a search tree that can be manipulated explicitly.
In Looprl, we implement a domain speciﬁc language embedded in OCaml for expressing nondeterministic strategies.
OCaml is a natural choice because it is fast (about two orders of magnitude fasterthan Python for symbolic code such as proof inferences) and it has good support for monads and
Python interoperability.
Our language provides built-in support for the choose and event operators.
Also, it deﬁnes an intermediate graph-based format for representing data to be sent to neural networksin an architecture-agnostic way.
Any piece of information that is passed to the choose operator mustbe convertible to this format and feature suitable metadata ensuring a seamless integration with the
Looprl user interface and debugging tools.
Looprl consists of about 15K lines of code written in
OCaml and Python.
It is available on Github.
Experiments
We tested Looprl on the problem of synthesizing loop invariants for single-loop imperative programsand evaluated it on the standard Code2Inv [27] benchmark.
This benchmark contains a set of132 programs written in C. All programs feature a single loop along with a ﬁnal assertion to beproven.
They feature linear integer arithmetic and sometimes involve conditionals, assumptionsand nondeterministic tests.
We provide examples of Code2Inv problems in Appendix C. Mostexisting invariant generation tools can only solve a subset of these problems [28].
To the best of ourknowledge, only one existing tool can solve them all [29], but only by using speciﬁc optimizationtechniques that are not generalizable beyond the setting of purely numerical programs.
We used Looprl to implement an invariant generation strategy, which we describe in Appendix A.
In anutshell, it generalizes the simple strategy described in Figure 2 by adding features such as the abilityto abduct disjunctive invariants or to conjecture invariant templates with placeholder constants to bereﬁned later using abduction.
To our surprise and although this generic strategy can be described inonly one page of pseudocode, it is sufﬁcient to simply solve every Code2inv benchmark problem in afew seconds when combined with the vanilla MCTS algorithm
[20] (with no learned heuristic).
We therefore evaluate a self-trained solver agent on its ability to solve as many Code2Inv problems aspossible without resorting to any search.
At every decision point, it can only take the highest rankedaction according to the network’s policy head and no backtracking is allowed.
We argue that sucha metric is the most skeptical measure for the quality of the resulting learning.
Indeed, ﬁnding aninvariant for a single loop is of limited interest of its own.
Rather, doing so is typically useful as asubtask in a hierarchy of increasingly complex and relevant problems.
The next level of this hierarchymay be to prove a piece of code with nested loops, and then to synthesize a function respecting aspeciﬁcation, which is still several levels away from writing full-ﬂedged software systems.
Complexproblems cannot be solved if backtracking search is needed at every level of this deep hierarchy.
4.1 Training Protocol
We train a teacher agent and a solver agent in sequence using the (Gumbel) AlphaZero algorithm.
Bothagents use a similar 1.6M parameters Dynamic Graph Transformer network [30] (see architecturedetails in Appendix H).
The teacher agent is trained ﬁrst for 20 AlphaZero iterations.
During eachiteration, it goes through 8000 problem generation episodes, using 64 MCTS simulations per step.
Then, the network is updated using samples from the k previous iterations with k evolving from 1 to6 during training.
Each iteration simulates 800 additional episodes to generate validation samplesthat are not used to train the network but to control overﬁtting via early stopping.
After the teacheragent is trained, a dataset of 50K problems is gathered for training the solver, which includes a mix of40K problems generated during the last ﬁve training iterations and 10K new ones that are generatedwithout Gumbel exploration noise [19].
The solver agent is also trained for 20 iterations.
Duringeach iteration, it attempts to solve 20K randomly selected problems using 32 MCTS simulations perstep.
In addition, 5K additional problems are solved to generate validation samples.
See Appendix Gfor a complete list and explanation of all training hyperparameters.
A complete training run takes about 16h on a single machine with a 10-core Intel i9-10900KFprocessor, 64GB of RAM and a NVIDIA GeForce RTX 3080 GPU.
4.2 Experimental Results
We show training curves for the solver and teacher agents in Figures 5 and 6 respectively.
Thesecurves record the evolution of the average reward collected by MCTS combined with the latestnetwork during each iteration.
Error intervals are estimated based on two random seeds.
Note thatthe maximum theoretical reward obtainable by the teacher agent is < 1 since problem generationconstraints may be sampled that are mutually incompatible, in which case the agent simply does itsbest to minimize the number of violated constraints.
The same holds for the solver agent since allgenerated invariants are penalized proportionally to their size.
The solver agent goes through a moremodest reward gain during training (the y -axis is rescaled in Figure 6 to emphasize the trend). This is Iteration number − 1 .− 0 .0 .0 .1 . A v e r ag e r e w a r d
Iteration number 0 .0 .0 .0 .0 .
A v e r ag e r e w a r d
Policy [Table] [Table] reﬂective of the fact that most problems generated by the teacher can be solved by MCTS alone andso most of the training concentrates on learning to solve about 10% of hard problems.
We show the results of our evaluation on the Code2Inv benchmark in Table 2.
Three different agentsare compared on the average ratio of benchmark problems for which they can successfully generatean invariant of minimal size without search or backtracking.
The ﬁrst agent is a baseline that simplyselects proof actions at random.
The second agent is a solver network that was trained using problemsgenerated by an untrained teacher (i.e. using MCTS but no network heuristic).
Finally, the third agentis a solver network trained using the full protocol described in section 4.1.
Unsurprisingly, an inferiorteacher leads to an inferior solver with decreased generalization capabilities.
Related Work
Leveraging nondeterministic programming for proof search The idea of combining nondeterministic expert strategies with neural oracles was proposed by Selsam [10, 31] as a possible angle fortackling the IMO Grand Challenge [32].
However, how to train such oracles remains an open problem.
One proposal [30] is to use supervised learning for training a universal oracle that is conditioned onthe description of arbitrary search problems and can therefore use training data from a large numberof heterogeneous sources.
In contrast, we propose using reinforcement learning with the insight thatnondeterministic search strategies can be used to generate problems in addition to solving them.
Learning to generate synthetic theorems Procedural generation techniques have been used forproducing synthetic theorems [33, 34, 35, 22, 36].
These usually proceed in a backwards fashion bygenerating random proof trees from a given set of axioms and rules.
Wang et al. proposed to usesupervised learning to learn how to guide this process towards generating more interesting theoremsthat are similar to those of a reference dataset [6].
Using reinforcement learning for loop invariant synthesis Reinforcement learning has alreadybeen applied to the problem of loop invariant synthesis [27, 28], albeit in a very different way.
In theaforementioned work (which introduces the Code2Inv benchmark), a separate reinforcement learningagent is trained from scratch on every benchmark problem, using counterexamples from an SMTsolver as a training signal.
This process takes minutes to hours for each problem to be solved.
In contrast, we train a single agent to generalize across problem instances so that new problems can besolved in a matter of milliseconds.
Using reinforcement learning for theorem proving The HOList Zero [8] and TacticZero
[9]systems use reinforcement learning to learn how to interact with tactic-based theorem provers withoutrelying on human proofs.
However, they still rely on large human-produced corpora of formalizedmathematical statements to be used as training tasks and are not yet competitive with approachesbased on imitation learning.
The use of reinforcement learning has also been explored in the contextof saturation-based or tableau-based provers for ﬁrst-order logic [37, 38].
However, learned heuristicsin such settings must operate at a very low-level and are subject to a speed-accuracy tradeoff that isunfavorable to deep learning.
Conclusions and Future Work
This paper took a bold stride toward the challenge of self-learning theorem proving without relyingon example theorems and proofs.
It advocates a hybrid approach to automated theorem provingwhere experts are provided a ﬂexible language to formalize their domain-speciﬁc knowledge in theform of nondeterministic teacher and solver strategies, leaving blanks to be ﬁlled by learning. We demonstrated our framework on the problem of loop invariant synthesis.
Our agent solves all Code2Inv challenges. More importantly, it learns to solve a majority of problems with no search atall despite never seeing these problems during training. None of our core contributions, however, are speciﬁc to invariant synthesis.
These include: i) our keyinsight that nondeterministic programming and reinforcement learning can be similarly combinedto implement solvers and teachers, ii) conditional generative strategies as a general template towrite teachers, iii) abductive reasoning as a design principle that is made scalable by self-learnedguidance, iv) strategy events for easier reward engineering and better sample-efﬁciency and v) animplementation that establishes engineering foundations for making the whole approach practical. We see this paper as a ﬁrst step toward a broader vision where interactive theorem provers allowusers to write teacher and solver strategies for various domains of mathematics and programming in amodular and distributed fashion. A unique neural network can then be trained as an oracle for allthese strategies, allowing better generalization and sample efﬁciency.
However, many challenges lieahead. In particular, while there is some evidence already [10, 39, 17] that effective solver strategiescan be written at scale (e.g. interactive theorem provers already allow specifying custom searchtactics [40] and we are merely suggesting a generalization of this popular feature), scaling up teacherstrategies raises more questions. For example, conditional generative strategies are quite engineeringintensive since diversity is enforced extrinsically via manually-deﬁned constraints.
Future work willexplore teachers that also integrate intrinsic reward signals, either via a curiosity mechanism or byhaving the solver directly reward the teacher for producing problems of suitable novelty and difﬁculty.
Broader Impact
Automated theorem proving for program veriﬁcation and program synthesis plays an important rolefor increasing the safety of critical pieces of software.
More automatic theorem provers enable awider user base to beneﬁt from the advances to their software quality.
But automating theoremprovers is a highly sophisticated challenge.
Learning from succinct expert advice in the form ofstrategies in Looprl may be the sweet spot enabling more generic domain-speciﬁc automation fortheorem provers.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]