HyperTree Proof Search for Neural Theorem Proving
AfﬁliationAddress
Abstract
We propose an online training procedure for a transformer-based automated theorem prover.
Our approach leverages a new search algorithm,
HyperTree Proof
Search (HTPS), inspired by the recent success of AlphaZero.
Our model learnsfrom previous proof searches through online training, allowing it to generalizeto domains far from the training distribution.
We report detailed ablations of ourpipeline’s main components by studying performance on three environments of increasing complexity.
In particular, we show that with HTPS alone, a model trainedon annotated proofs manages to prove 65 .
4% of a held-out set of Metamath theorems, signiﬁcantly outperforming the previous state of the art of 56 .
5% by GPT-f.
Online training on these unproved theorems increases accuracy to 82 .
With asimilar computational budget, we improve the state of the art on the Lean-basedminiF2F-curriculum dataset from 31% to 42% proving accuracy.
Introduction
The nineteenth century saw the emergence of proofs so involved that they could only be veriﬁed 14 by a handful of specialists.
This limited peer review process inevitably led to invalid proofs, withmistakes sometimes remaining undiscovered for years (e g. the erroneous proof of the Four Colour 16 Conjecture [1]).
Some mathematicians argue that the frontier of mathematics has reached such a levelof complexity that the traditional review process is no longer sufﬁcient, envisioning a future where 18 articles are submitted with formal proofs so that the correctness can be delegated to a computer [2].
Unfortunately, very few mathematicians have adopted formal systems in their work and as of today, 20 only a fraction of existing mathematics has been formalized.
Several obstacles have hindered thewidespread adoption of formal systems.
First, formalized mathematics are quite dissimilar from 22 traditional mathematics, rather closer to source code written in a programming language, whichmakes formal systems difﬁcult to use, especially for newcomers.
Second, formalizing an existing 24 proof still involves signiﬁcant effort and expertise (the formalization of the Kepler conjecture took over 20 person years to complete [3]) and even seemingly simple statements sometimes remain frustratingly challenging to formalize
To write a formal proof, mathematicians typically work with Interactive Theorem Provers (ITPs).
The most popular ITPs provide high-level “tactics” that can be applied on an input theorem (e.g. theinitial goal) to generate a set of subgoals, with the guarantee that proving all subgoals will result in a proof of the initial goal (reaching an empty set means the tactic solves the goal).
An example of a proof in Lean [4], an interactive theorem prover, is given in Figure 1 and the corresponding proof hypertree is illustrated in Figure 5 of the Appendix
In this paper, we aim at creating a prover that can automatically solve input theorems by generating a 34 sequence of suitable tactics without human interaction.
The backward procedure naturally suggests
[Figure]
a simple approach where a machine learning model trained to map goals to tactics interacts with 36 an ITP to build the proof of an input goal in a backward fashion.
The automated prover builds ahypergraph with the theorem to be proved as the root node, tactics as edges and subgoals as nodes.
The prover recursively expands leaves by generating tactics with our model until we ﬁnd a proof ofthe initial theorem.
A proof is then a hypertree rooted in the initial theorem whose leaves are empty 40 sets.
Search in this setting is analog to search in two player games.
However, particular challenges arise in theorem proving.
First, the action space, i.e. the amount 42 of possible “moves” in a given state, is inﬁnite (there is an unlimited number of tactics that can beapplied to a given theorem).
This requires sampling possible actions from a language model for 44 which training data is scarce.
Moreover, if all tactics sampled at a goal fail, we have no informationon what region of the probability space to sample next.
Second, in the context of theorem proving, 46 we need to provide a proof of all subgoals created by a tactic, whereas AlphaZero[5] for two playergames is allowed to focus on the most likely adversary moves.
This paper presents an in-depth study of our approach to overcome these difﬁculties and the resulting 49 model, Evariste.
In particular, we make the following contributions:
• A new MCTS-inspired search algorithm for ﬁnding proofs in unbalanced hypergraphs.•
A new environment (Equations) to easily prototype and understand the behavior of themodels we train and our proof search.•
A detailed ablation study and analysis of the different components used in our approach onthree different theorem proving environments.
We study how data is selected for training the
policy model after a successful or failed proof-search, what target should be used to train
the critic model, and the impact of online training vs. expert iteration.• State-of-the-art performance on all analyzed environments.
In particular, our model managesto prove over 82.6% of proofs in a held-out set of theorems from set.mm in Metamath, aswell as 58.6% on miniF2F-valid [6] in Lean.
Related work
Automated theorem proving has been a long-standing goal of artiﬁcial intelligence, with the earliest 62 work dating from the 1950s [7, 8].
We focus here on recent work closest to ours and defer additionalrelated work to Appendix B.
Neural theorem provers.
Recent work applying deep learning methods to theorem proving [9–11] 65 are the closest to this work and obtained impressive results on difﬁcult held-out sets for Metamathand Lean.
The main differences between their approach and ours are the proof-search algorithm we 67 propose, the training data we extract from proof-searches and our use of online training comparedto their expert iterations.
Another similar approach, Holophrasm [12], uses a different tree-search 69 algorithm while others [13, 14] learn the search policy along with the tactic model.
Unlike previousstudies that focus on a single proving environment (e.g. Metamath, Lean, or HOL-Light), we 71 extensively study the performance of our prover on three different formal languages, and found thatsome conclusions signiﬁcantly vary based on the considered environment.
MCTS and two player games.
AlphaZero [5] demonstrated good performances on two player 74 games, replacing the Monte-Carlo evaluations of MCTS [15] with evaluations from a deep neuralnetwork and guiding the search with an additional deep policy.
These ideas have been applied to ﬁrst 76 order logic proving in Kaliszyk et al.
[16] with gradient boosted trees as policy and value models.
Theorem proving can be thought of as computing game-theoretic value for positions in a min/max 78 tree: to prove a goal, we need one move (max) that leads to subgoals that are all proven (min).
[-(2 + (1 + 1))and 
-]
Thishas led to other tree search algorithms such as Proof Number Search [17] or a more recent version, 80 using a neural estimate of proof size: Wu et al.
Noticing heterogeneity in the arities of min ormax nodes, we propose a search method that goes down simultaneously in all children of min nodes, 82 such that every simulation can potentially result in a full proof-tree.
Online training from proof searches
In this section, we introduce our Hypertree Proof Search (HTPS) algorithm and describe how it is 85 used to generate training data for our model.
We then detail our online training method.
3.1 Hypertree Proof Search
Given a main goal g to automatically prove, HTPS is the algorithm that interacts with a policy 88 model P ✓ and a critic model c ✓ , and the theorem proving environment to ﬁnd a proof hypertree for g . {+
+}
Proof search progressively grows a hypergraph starting from g , iteratively repeating the three stepsillustrated in Figure 2: selection, expansion and back-propagation.
The main difference with other 91 search algorithms is our parallel descent in all subgoals of a tactic.
A proof is found when there existsa hypertree from the root to leaves that are empty sets. Selection [Table] g g
[Table]
.1) × 0.
v T (g)=(1 × 0.1) × 0. Back-propagation Expansion g g g v T (g 1 )=0. N(g 1 ,t 0 ) =1 W(g 1 ,t 0 )
=0.4 N(g 0 ,t 0 )
=1 W(g 0 ,t 0 )
=1x0.
{+v T (g 0 ) =1 × 0. N(g 0 ,t 0 ) =
+}
W(g 0 ,t 0 )
=
ggg 1 gggv T (g 4 )=0.4 N(g 4 ,t 1 )=
W(g 4 ,t 1 )=0 gggg 2 v T (g 2 )=
1 v T (g 3 )
=
0.1 gg 4 g 3 g 3 ggg 4 g 3 gg
[Figure]
gg
We assume a policy model P ✓ and critic model
c ✓ .
Conditioned on a goal, the policy model allows 94 the sampling of tactics, whereas the critic model estimates our ability to ﬁnd a proof for this goal.
Our proof search algorithm will be guided by these two models. Additionally, and similar to MCTS, 96 we store the visit count N ( g, t ) (the number of times the tactic t has been selected at node g ) and thetotal action value W ( g, t ) for each tactic t of a goal g . These statistics will be used in the selectionphase and accumulated during the back-propagation phase of the proof search described below. Selection The number of nodes in the proof hypergraph grows exponentially with the distance to 100 the root.
Thus, naive breadth-ﬁrst search is infeasible to ﬁnd deep proofs and some prioritizationcriteria is required to balance depth and breadth.
Similar to MCTS , we balance the policy model’s 102 prior with current estimates from the critic.
In particular, we experiment with two different searchpolicies: PUCT [19] and Regularized Policy (RP) [20] (algorithms are detailed in Appendix C.3).
A key difference between previous work and ours is that our proof search operates on a hypergraph.
Thus, whereas MCTS goes down a path from the root to an unexpanded node during its selectionphase, our algorithm will instead create a partial proof hypertree, leading to a set of either solved 107 or unexpanded nodes.
To do so, we recursively follow the arg-max of the search policy from theroot, until we reach the leaves of the current hypergraph (the detailed pseudo-code can be found in 109 Section C.4).
This selection step is illustrated in Figure 2.
In order to batch calls to the policy and critic models over more nodes to expand, we run several 111 selections sequentially, using a virtual loss [21, 5] to produce different partial proof-trees.
Note thatsolving all unexpanded leaves of any of these trees would immediately lead to a full proof of the root.
In the next section, we describe how nodes are expanded.
Expansion To expand a node g , we use the policy model to sample tactics that would make progress 115 on the goal.
Tactics are sampled in an auto-regressive fashion (token by token) by the decoder [22],based on the previously generated tokens, and on a representation of the goal provided by the encoder.
117 The generated tactics are then evaluated in the theorem proving environment.
Each valid tactic willlead to a set of new subgoals to solve, or to an empty set if the tactic solves the goal.
Finally, we 119 add a hyperedge for each valid tactic t i from the expanded node g to its (potentially empty) set ofchildren for this tactic { g 0 i , ..., g ki } .
Note that these children might already be part of the hypergraph.
121 For new nodes, visit counts N ( g, t ) and total action values W ( g, t ) are initialized to zero.
There arethree types of nodes in the hypergraph:
• Solved : at least one tactic leads to an empty set, or has all its children solved.
• Invalid : all tactics sampled from the policy model were rejected by the environment, or lead
to invalid nodes.
• Unsolved : neither solved nor invalid, some tactics have unexpanded descendants.
Note that the deﬁnitions for Solved or Invalid are recursive.
These status are updated throughout the 128 hypergraph anytime a hyperedge is added.
Tactics leading to invalid nodes are removed to preventsimulations from reaching infeasible nodes.
Once this is done, we back-propagate values from the 130 expanded nodes up to the root, as described in the next section.
Back-propagation For each expanded goal g in a simulated proof tree T , its value is set to v
{+T ( g ) = 1 if it is solved, and v T ( g ) = 
+}
if it is invalid.
Otherwise, its value is estimated by the critic model: v
T ( g ) =
c ✓ ( g ) .
This provides v T for all leaves of T and we can then back-propagate in topographic order (children before parents) through all nodes of T .
Interpreting the value of a node as the probability that it can be solved, since solving a goal requires solving all of its children subgoals, the value of a parent is the product of values of its children (we assume that the solvability of subgoals is independent, for simplicity):
[Equation]
In particular, the value of a goal g is the product of the values of all leaves in T that remain to be 132 solved to obtain a proof of g .
Once all values in T are computed, we increment the correspondingvisit count N ( g, t ) in the hypergraph as well as the total action values: W ( g, t )
+= v T ( g ) .
For agoal g , the estimated value for tactic t is then the mean of the total action value:
[Equation]
A fully detailed back-propagation step is illustrated in Figure 2.
3.2 Online training
Both the policy model P ✓ and the critic model
c ✓ are encoder-decoder transformers [23] with shared 138 weights ✓ , which are trained online on two different objectives.
The policy model P ✓ takes as input atokenized goal and generates tactics.
It is trained with a standard seq2seq objective [22], where we 140 minimize the cross-entropy loss of predicted tactic tokens conditioned on the input goal.
Our critic model c ✓ is used to predict ﬂoating point values representing how likely a goal is to be 142 solved.
We start decoding with a special token, restrict the output vocabulary to two tokens PROVABLEand UNPROVABLE , and evaluate the critic with c ✓ ( g )
=
P ( PROVABLE | g, CRITIC ) .
This objective is 144 identical to a seq2seq objective where the cross-entropy is minimized over the two special tokens.
Our online training uses a distributed learning architecture reminiscent of AlphaZero [19] or dis- 146 tributed reinforcement learning setups [24, 5].
A distributed data parallel trainer receives trainingdata from a set of asynchronous provers that run proof searches on theorems chosen by a controller.
148 Provers, in turn, continuously retrieve the latest model versions produced by the trainers in order toimprove the quality of their proof search.
This set-up is represented in Figure 7 of the Appendix.
Once a prover ﬁnishes a proof-search, we extract two types of training samples from its hypergraph:
Tactic samples.
At the end of a successful proof search, we extract (goal, tactic) pairs of a minimal 152 proof hypertree of the root node as training samples for the policy model.
We use a differentminimality criterion depending on the environment: number of proof steps for Metamath and 154 Equations and total tactic CPU time for Lean. We show that this selection has a large impact onperformances, other options such as selecting all solved nodes are investigated in Section 5.2.1.
The 156 policy model is trained with a standard seq2seq objective [22], where we minimize the cross-entropyloss of predicted tactic tokens conditioned on an input goal.
Critic samples.
In the proof search hypergraph, we select all nodes that are either solved, invalid 159 (all tactics failed or led to invalid nodes), or with a visit count higher than a threshold.
Then, we usec ( g ) = 1 as the training target for solved nodes.
For internal nodes, we use the ﬁnal estimated action 161 value c ( g ) =
W ( g, t ⇤ ) /N
( g, t ⇤ ) where t ⇤ is the tactic that maximizes the search policy at g .
Finally,for invalid nodes, we use c ( g ) = 0 .
The trainers receive training samples that are stored into two separate ﬁnite-size queues, one for each 164 objective.
When a queue is full, appending a new sample discards the oldest one.
In order to create abatch for a task, we uniformly select samples in the corresponding queue.
During online training, in addition from this generated data, we also sample from the supervised 167 datasets used for ﬁne-tuning our models (see 4.1) which provide high-quality data.
All trainingobjectives are weighted equally.
An overview of our full training pipeline is given in Appendix D.1.
Our proof-search depends on many hyper-parameters, and the optimal settings might not be the same 170 for all statements, making tuning impractical.
Thus, the controller samples these hyper-parametersfrom pre-deﬁned ranges (see Appendix D.3 for details) for each different proof-search attempt.
Experiments
In this section, we provide details about our experimental training and evaluation protocols.
We ﬁrst 174 describe the supervised datasets used to ﬁne-tune our policy models, as well as the tokenization used.
Then, we discuss the evaluation datasets and methodology.
In Appendix D.2, we provide additional 176 details about our model pretraining and architecture.
We develop and test our methods on three theorem proving environments: Metamath, Lean and 178 Equations.
Metamath [25] comes with a database of 35 k human-written theorems called set.mm .
Wealso evaluate our methods on the Lean proving environment, which provides a level of automation 180 that is helpful to solve more complex theorems.
Lean comes with a human-written library of 27 ktheorems called Mathlib [26].
4.1 Model ﬁne-tuning and supervised datasets
Starting the HTPS procedure described in Section 3 from a randomly initialized model would be sub- 184 optimal, as no valid tactic would ever be sampled from the policy model.
Thus, starting the online training from a non-trivial model is critical.
To this end, we ﬁrst ﬁne-tune our policy model P ✓ on a super- 186 vised dataset of theorems speciﬁc to each environment.
We refer to this model as the supervised model. Metamath In Metamath
[Table]
goal-tactic pairs; more statistics about the training data are provided in Table 1.
Tokenization in 190 Metamath is trivial, as statements are composed of space-separated tokens.
Lean Following [11], we extract a supervised dataset from the Mathlib library and co-train with the 192 dataset of proof-artifacts of Han et al.
[10] to reduce overﬁtting.
To facilitate experimentation andreproducibility, we use ﬁxed versions of Lean, Mathlib, and miniF2F (c.f. Appendix D.5).
Finally, 194 we add another supervised co-training task by converting to Lean a synthetic dataset of theoremsgenerated by the Equations environment (c.f. Appendix E.6).
Statistics about the training set are 196 available in Table 1.
Equations Finally, we developed the Equations environment as a simpler analogue to existing 198 proving environments.
Its expressivity is restricted to manipulating mathematical expressions (e.g.equalities or inequalities) with simple rules (e.g. A + B = B + A , or A < B )  B <  A ).
200 Unlike Metamath or Lean, the Equations environment does not come with a dataset of manuallyannotated proofs of theorems.
Instead, we generate supervised data on the ﬂy using the random 202 graph generator described in Appendix E.5.
As the model quickly reaches perfect accuracy on thesesynthetic theorems, we only leverage statements from the Identities split during online training.
4.2 Evaluation settings and protocol
In Polu et al.
[11], the model is ﬁne-tuned on theorems from the training set and expert iteration is done 206 on theorems from different sources: train theorems, synthetic statements, and an extra curriculum ofstatements without proofs (miniF2F-curriculum).
The produced model is then evaluated on unseen 208 statements, namely the validation and test splits of the miniF2F dataset [6].
In this work, we also consider the transductive setup: on a corpus of unproved statements available 210 at train time, how many proofs can our method learn to generate?
This protocol is also sensible, asallowing the model to learn from a failed proof-search can lead to more focused exploration on the 212 next attempt, proving more statements overall than a model that would not be trained online.
Following [9], we also evaluate the pass@k by running k proof searches on the evaluated statements 214 with the policy and critic obtained by online training.
Given the many evaluations presented in thiswork, we only run them once.
We give more details on the hyper-parameters used in Appendix D.3. {+
+}
In the transductive setup, we also report the cumulative pass rate , i [Table] e. the proportion of theoremssolved at least once during online training.
Results
In this section, we present our results and study the moving parts of our pipeline through ablations.
220 We compare our results with GPT-f which represents the state of the art on Metamath and Lean.
[Table]
5.1 Main results5.1.1 Lean
In Lean, we run our experiments on A100 GPUs with 32 trainers and 200 provers.
Each prover 224 runs our Lean API on 48 CPU cores.
Unlike Polu et al.
[11], we sample statements equally frommathlib-train and miniF2F-curriculum, to avoid giving too much importance to statements from a 226 different domain than the target.
Results can be found in Table 2.
After 1 day of training, eachstatement from miniF2F-curriculum has been sampled on average 250 times, and 110 out of the 327 228 statements have been solved.
Our model outperforms GPT-f on miniF2F-test, with an approximately10 ⇥ training time speed-up.
After 7 days, we solve 139 statements of miniF2F-curriculum ( 100 for 230 GPT-f), and observe further improvements on miniF2F-valid or miniF2F-test.
For other evaluations, we depart from the set-up of Polu et al.
[11], directly using the statements 232 from the miniF2F-valid split in our online training, obtaining 58.6% cumulative pass rate.
We thenevaluate the ﬁnal model on miniF2F-test, reaching 41% pass@64, against 36.6% for GPT-f. Without 234 the synthetic data co-training task, the performance drops to 54 .
9% cumulative pass rate on theminiF2F-valid split, and 38 .
5% pass@64 on the miniF2F-test split.
Examples of proofs found by our 236 model can be found in Appendix F.
5.1.2 Metamath
On Metamath, we train our model on V100 GPUs, with 128 trainers and 256 provers, whereas 239 ablations are run on 16 trainers and 32 provers.
We report our results in Table 3 for the supervisedmodel and for a model trained with online training.
During online training, we sample statements 241 from the training and from the validation splits of set.mm equally.
Online training dramatically improves performances on valid statements, going from a 61% pass@8 243 to a cumulative pass rate of 82 .
This improvement cannot solely be explained by the high numberof attempts on validation theorems during training.
Indeed, the ablation in Figure 3 (right) shows 245 that Evariste signiﬁcantly outperforms a supervised model with the same number of attempts.
Thesupervised model plateaus at 66% while Evariste keeps improving beyond 74% after 7 days of 247 training, showing that the model is able to learn from previous proof searches through online training.
On test theorems, for which statements were not provided during online training, the accuracy 249 increased by 10% compared to the supervised model, from 55 . 8% to 65 .
6% accuracy.
The supervisedmodel obtains a pass@32 accuracy of 65 . 4% (resp.
2% ) on the validation (resp.
test) splits,compared to GPT-f’s 56 .
5% (resp.
2% ) after expert iteration.
[Table]
In Equations, we run our main experiment with 32 trainers and 64 provers, whereas ablations are run 254 on 16 trainers and 32 provers.
In this environment, the model easily learns the training distribution ofour random generator, and solves all synthetically generated problems.
Thus, online training is run 256 on the Identities statements only.
Our main experiment reaches a cumulative pass rate of 91 .
3% onthe Identities split, while a supervised model never exceeds 36% even after a similar number of proof 258 attempts.
In Appendix 9, we give examples of Identities statements proved during online training, aswell as the size and depth of proofs found by the model.
In particular, Evariste managed to ﬁnd the proof of complex mathematical statements, 261 such as sinh( x/ 2)
= sinh( x ) / p 2(1 + cosh( x )) and tan(3 x )(  3(tan( x ))
= tan( x )  262 (tan( x )) 2 tan( x ) that required 82 and 117 proof steps respectively, showing the abilities of HTPS to prioritize subgoals and guide the search in very large proof graphs.
This shows that online training is 264 able to adapt our policy and critic models to a completely new domain, going from automaticallygenerated statements to identities found in math books.
Examples to understand the gap between 266 these two domains can be found in Appendix E.
5.2 Ablation study
In this section [Table] we present an ablation study on several components of our system.
Since Lean 269 experiments are CPU intensive, we run most of our ablations on the Equations and Metamathenvironments.
On Lean, we ran experiments on a smaller subset of hyper-parameters that consistently 271 performed well on the other environments.
5.2.1 Online training data for tactic objective
[Table]
The way we ﬁlter tactics sent to the trainers has a large impact on ﬁnal performances.
We investigated 274 several ﬁltering methods and report the results in Table 4.
The ﬁrst method is similar to the one usedin AlphaZero and exposed in [19]: we select all nodes of the proof search hypergraph where the visit 276 count is above a certain threshold and we ﬁlter tactics above a given search policy score.
At trainingtime, tactics are sampled according to the ﬁltered search policy.
With this method the model reaches 278 51.6% pass@8 on the valid set of Metamath and 37.5% cumulative pass rate on Equations.
We then experimented with other ﬁltering criteria, selecting only goal-tactic pairs that are part of 280 proofs: either a proof of the root node, or of any solved node in the hypergraph.
Then, we learn fromall possible proofs, or only from proofs that are minimal according to a criteria (number of proof 282 steps for Equations and Metamath, cumulative CPU time for Lean).
Learning only from minimal proofs always leads to improved performance, regardless of the selected 284 roots.
Learning from the minimal proofs of all solved nodes, we reach a cumulative pass rate of78.1% on Equations, compared to 40.6% when learning from all proofs.
On Metamath, only learning 286 from the root’s minimal proof gives the best result on the valid set, reaching a pass@8 of 68.6%.
5.2.2 Critic
[Table]
To measure the impact of our critic model, we run an experiment where the proof search is only 289 guided by the policy model.
In particular, during the back-propagation phase, we set v T ( g ) = 0 .
5for leaves of T .
In that context, our model is no longer trained with a critic objective.
We report the 291 results in Table 5.
We ﬁnd that using a critic model improves the performance signiﬁcantly, by 5 .
2%and 12 .
5% on Metamath and Equations respectively.
As mentioned in Section 3, to train the critic objective, we set the training targets as c ( g )
= 1 for 294 solved nodes, c ( g ) = 0 for invalid nodes and c ( g ) =
W ( g, t ⇤ ) /N
( g, t ⇤ ) where t ⇤ is the tactic thatmaximizes the search policy at g , for internal nodes.
We also tested a hard critic estimation of the 296 target values, following Polu and Sutskever [9], where c ( g ) = 1 for solved nodes and c ( g ) =
forboth invalid and internal nodes.
We report results in Table 5.
For both Metamath and Equations, 298 HTPS critic targets allow Evariste to reach its best performance.
In Equations, the model reaches acumulative pass rate of 78.1%, compared to 63.1% with hard critic estimates.
In Equations, using 300 hard critic targets gives worse performances than having no critic model at all, showing that thesetargets are a bad estimation: setting all internal nodes to zero is too pessimistic.
5.2.3 Fixed proof search parameters
We study the impact of sampling HTPS hyper-parameters for each attempt during online training.
We 304 run experiments with ﬁxed, chosen search parameters for Equations and Metamath to compare withrandom sampling, and report results in Table 5.
Evariste achieves better performances than the model 306 trained with ﬁxed search parameters on Metamath test set and Equations Identities , reaching 78.1%pass rate compared to 73.8% in Equations Identities .
5.2.4 Model update frequency during online training
In our online training procedure, the policy and critic models are updated every ﬁve minutes on the 310 provers.
We measure the impact of the frequency of these updates by trying different refresh rates: 5minutes, 1 hour, 6 hours for Equations, and no updates at all for both Equations and Metamath.
We 312 report the cumulative pass rate over training hours in Figure 3.
The higher the refresh rate, the betterthe cumulative pass rate over time, conﬁrming the beneﬁts of online training over expert iteration.
Conclusion
In this work, we introduce HTPS, an AlphaZero-inspired proof search algorithm for automated 316 theorem proving, along with an online training procedure.
We run an extensive study of our pipeline,and present state-of-the-art results on multiple proving environments.
We show that online training 318 provides large speed-ups over expert iteration, and allows generalization of the policy and criticmodels to completely new domains.
Despite large number of attempts per theorem, proving the 320 entirety of datasets like miniF2F remains elusive, and generating data from proof-search on thecurrently available corpora will likely be insufﬁcient in the long term.
As manually annotated formal 322 datasets are limited, another way of providing exploration and additional training data (in the spirit ofself-play for two player games) is required.
Automated generation of new theorems is likely to be 324 one of the future milestones.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] Section 5.1 includes our resultsas well as their limitations.(c)
Did you discuss any potential negative societal impacts of your work?
[No] Beyondenvironmental cost of training models, we do not see any direct path from the researchpresented in this paper to negative applications.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code forthe Equations environment will be open-sourced.
We also plan to make our trainedmodel publicly available to help people in the formal community.
Code for the overalldistributed training architecture is tied to our infrastructure and will be difﬁcult toopen-source.(b) Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?
[Yes] We provide all training details in Section D.2 of the Appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] As stated in Section 4.2, we report the result of onetraining and one evaluation as running all evaluations multiple times would be toocostly.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[No] License is available on the repositoriesassociated with each asset.(c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A]No new assets in this work.(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]