F ORMAL M ATHEMATICS S TATEMENT C URRICULUM L EARNING
A BSTRACT
We explore the use of expert iteration in the context of language modeling applied to formal mathematics.
We show that at same compute budget, expert iteration, by which we mean proof search interleaved with learning, dramatically outperforms proof search only.
We also observe that when applied to a collection of formal statements of sufficiently varied difficulty, expert iteration is capable of finding and solving a curriculum of increasingly difficult problems, without the need for associated ground-truth proofs.
Finally, by applying this expert iteration to a manually curated set of problem statements, we surpass previous state-of-the-art on the miniF2F benchmark, automatically solving multiple challenging problems drawn from high school olympiads
I NTRODUCTION
Deep learning has enjoyed spectacular success in many domains, including language (Brown et al., 2020;
Devlin et al., 2019; Wu et al., 2016), vision (Radford et al., 2021; Tan & Le, 2019), and image generation (Ramesh et al., 2021;
Karras et al., 2019).
One domain where deep learning has not yet enjoyed a comparable success is in tasks that require extensive planning and symbolic reasoning , with the exception of two-player games (Silver et al., 2016; 2017; Berner et al., 2019; Vinyals et al., 2019).
In such games, deep learning systems exhibit a considerable degree of reasoning, especially when trained with self-play combined with a search procedure such as Monte Carlo Tree Search (MCTS) (Browne et al., 2012).
But the resulting reasoning abilities achieved are limited due to the relatively narrow scope of games.
As such, theorem proving in interactive proof assistants, or formal mathematics, appears as an interesting game-like domain to tackle due to its increased scope.
The typical tasks consist of generating a machine-checkable proof given a formal statements.
Like games, formal mathematics has an automated way of determining whether a trajectory ( i.e. a proof) is successful ( i.e. formally correct).
But the vast scope of formal mathematics means that any strong reasoning result obtained in it will be more meaningful than comparable results in games ( e.g. finding proofs to mathematical conjectures), and could even be applicable to important practical problems ( e.g. software verification).
However, tackling formal mathematics involves two main challenges that we must address in order to continue making progress:
Infinite action space Not only does formal mathematics have an extremely large search space (like
Go (Silver et al., 2016) for example), it also has an infinite action space.
At each step of proof search, the model must choose not from a well-behaved finite set of actions, but a complex and infinite set of tactics, potentially involving exogenous mathematical terms that have to be generated (e.g., generating a mathematical statement to be used as a witness, an object used steps such as “there exists an x ...”, or a cut, the introduction and the chaining of a lemma in the middle of a proof).
No direct self-play setup In formal mathematics, a prover is not playing against an opponent but against a set of statements to prove.
When faced with a statement that is just too hard, there is no obvious reframing of the formal mathematics setup that will let the prover generate intermediary easier statements to tackle first.
This asymmetry prevents naive application of the symmetric self-play algorithms commonly used in 2-player games.
These two differences make a naive application of reinforcement learning to formal mathematics leave a large room for improvement (Whalen, 2016; Winands et al., 2008).
Past work proposed to address the infinite action space problem by sampling from a language model (Polu & Sutskever, 2020), while training such language model requires a large dataset of statements with proof.
This paper focuses on this second problem and our basis for addressing it is the observation that the key role of self-play is to provide an unsupervised curriculum.
We propose instead to supply auxiliary sets of problem statements (without requiring proofs) of varying difficulty.
We empirically show that, when the difficulty of these auxiliary problems is varied enough, a simple expert iteration procedure is able to solve a curriculum of increasingly difficult problems, eventually generalizing to our target distribution.
We show that this works with both automatically-generated and manually-curated auxiliary distributions of problems and leverage this to achieve state-of-the-art on the miniF2F benchmark.
Our results suggest that continuous self-improvement in formal mathematics can potentially be reduced to the problem of generating such sets of formal statements, which we have done in part manually in this work, but could eventually be scaled in the future with more automation (such as more domain-specific statements generator or even informal to formal machine translation).
miniF2F benchmark In this work, we target the miniF2F (Zheng et al., 2022) benchmark, which consists of 244 validation and 244 test formalized statements of mathematical problems from various competitions.
We believe it to be a better measure of mathematical reasoning compared to a formal library-derived split.
Also, the extreme scarcity in formal libraries of this type of problems makes it an ideal test-bed for the expert iteration methodology studied in this paper.
R ELATED W ORK
Our work strongly relies on, and can be seen as a natural continuation of the work presented in the original GPT-f paper (Polu & Sutskever, 2020) which studies the use of language models to generate tactics, the PACT paper (Han et al., 2022) which applies GPT-f to Lean and studies the benefits from co-training on self-supervised objectives, and the miniF2F benchmark (Zheng et al., 2022).
We present additional related work in Appendix A.
F ORMAL E NVIRONMENT
We choose Lean (de Moura et al., 2015; lea) as our formal environment.
Unlike Metamath (Megill & Wheeler, 2019) , which has been studied in the original GPT-f paper (Polu & Sutskever, 2020),
Lean benefits from high-level tactics which were shown to be beneficial in the context of the miniF2F benchmark.
Also, Lean has recently received a lot of attention from the mathematical community, thanks to projects such as the Perfectoid Spaces (Buzzard et al., 2019) and the Liquid Tensor experiment (Scholze, 2020), and benefits from a vibrant community of hundreds of contributors to its main mathematical library called mathlib .
We refer to the PACT paper’s Background section (Han et al., 2022) for a detailed introduction to Lean in the context of neural theorem proving.
We refer to
Appendix D for an illustration of miniF2F input and Lean environment.
lean-gym
In the PACT paper (Han et al., 2022), proof search is performed by the Lean runtime using the L EAN S TEP environment, with a generic backend interface to models.
While easy to use–one just needs to plug in their model–this approach makes it difficult to alter and iterate on the search procedure because it is programmed in Lean (which is not designed or intended for cluster-wide parallelised
I/O intensive tasks), and the coupling of the search procedure with the Lean runtime introduces challenges when scaling to a large number of parallel workers.
To solve these issues we implemented lean-gym 1 – a simple REPL interface over the standard input/output implemented in Lean directly.
We present lean-gym ’s API and discuss some of its advantages and limitations in Appendix B.
Proof extraction We rely on the proof extraction methodology presented in the PACT paper (Han et al., 2022) to extract human tactic proof steps from mathlib (the tactic dataset) as well as the various other proof artifacts ( mix1 and mix2 datasets).
We also extract mathlib-{train, valid, test} , the set of statements from mathlib along the split proposed in Han et al.
(2022) (the validation and test splits of tactic, mix1, mix2 being aligned with mathlib-{valid, test} as the splits are determined by declaration name hashes (across all data sources including proof-term mining) as opposed to individual proof steps or data-points.
E XPERT I TERATION
Expert iteration was introduced in Silver et al.
(2017) and broadly consists in iteratively training models on their previously sampled trajectories, to achieve continuous improvement.
In this section we present our expert iteration methodology, including the models and pre-training strategies.
We use decoder-only Transformers similar to GPT-3 (Brown et al., 2020).
Throughout this paper we focus on a model with 36 layers and 774 million trainable parameters (referred to as the 700m model in the GPT-f paper (Polu & Sutskever, 2020)).
4. P RE -T RAINING
We pre-train our models successively on GPT-3’s post-processed version of CommonCrawl (for 300B tokens) and an updated version of WebMath (Polu & Sutskever, 2020) (for 72B tokens) whose mix is presented in Appendix C.
4. T RAINING OBJECTIVES
Proofstep objective The proofstep objective , introduced in Polu & Sutskever (2020), consists in generating a PROOFSTEP (a Lean tactic) given a GOAL (a Lean tactic state).
We also condition this objective on the current DECLARATION (a Lean theorem name), which remains the same throughout a proof search: DECL <DECLARATION> GOAL <
GOAL> PROOFSTEP <PROOFSTEP> .
The rationale for conditioning on the declaration name is to hint our models on the position of the current declaration in the mathlib library.
It can be considered as a weak proxy signal for the large amount of information not shown to the model (the full environment consisting of the available imports and currently open declarations such as module names, notations, declared instances, ...).
The declaration name lets models at least in principle memorize and then retrieve some of that information, knowing that lean-gym errors if a theorem or definition that is not available in the environment associated with the current declaration is used by tactics generated by our models.
Also note that conversely to Polu & Sutskever (2020) and like Han et al.
(2022) <
GOAL> is not necessarily a single goal but a Lean tactic state, which possibly comprises multiple goals.
Proofsize objective We depart from Polu & Sutskever (2020) and use a proofsize objective to guide our proof searches, which consists in generating one token that represents a proof size estimate bucket for the current goal (Lean tactic state): DECL <DECLARATION> GOAL <GOAL> PROOFSIZE <
PROOFSIZE_BUCKET_TOKEN>
For a given goal g , either the goal was proved as part of the proof search and we denote its proof size (the number of tactic applications (compounded Lean tactics counting as one))
as ps ( g ) , or the goal was not proved in which case we assign the goal to a bucket that virtually represents "infinite" proof sizes.
We use 11 buckets B = 0 ... 10 and compute the proofsize bucket b ( g ) for a goal g by assigning infinite proof sizes to bucket 0 , all proof sizes over 20 to bucket 1 and linearly projecting proof sizes lower than 20 on the remaining buckets 2 , ..., 10 ( 10 being the bucket for the shortest proof sizes).
In practice, when training and sampling from the model, we map B to the tokens ’A’ ... ’K’.
To value goals as we run proof searches, we sample the proofsize bucket token and record the probability p b ( g ) for each viable bucket and use them to get a weighted average with the following formula: v ( g ) =
1# B (cid:80) b ∈ B p b ( g ) · b .
As an example, if the model assigns p 0 = 1 (hence p b ̸ =
=
0 ) then v ( g ) = 0 .
Conversely
[Table] The rationale for using this proofsize objective instead of the outcome objective described in Polu & Sutskever (2020) is that (i) it achieves better performance compared to the outcome objective (see Table 1) [Table] and (ii) it prioritizes goals that potentially lead to shorter proofs during proof search, creating an intrinsic incentive for the system to converge towards shorter proofs.
Similarly to Polu &
Sutskever (2020) we favor this token-based approach to the introduction of a separate value head to keep the overall architecture simple.
This way the proofsize objective can be implemented by simply augmenting the training dataset and without any architectural change.
4. B OOTSTRAPPING
Bootstrapping consists in the steps required to train an initial model on both the proofstep objective and the proofsize objective .
Given a pre-trained model on WebMath , we fine-tune it on the tactic dataset extracted from mathlib as well as the proof artifacts dataset mix1 as described in Han et al. (2022).
This initial model, which we denote θ 0 is solely trained on the proofstep objective .
We use the validation splits of the tactic and m1 datasets to early-stop training.
Note that this is our only use of mathlib-valid to influence the training process throughout this paper.
To generate data for the proofsize objective , we use θ 0 to sample proofs for statements from mathlibtrain .
For each statement from mathlib-train (25k) we attempt a = 1 proof searches using the cumulative logprob priority search described in Polu & Sutskever (2020) (which does not require a trained value function) using d = 512 expansions and e = 8 samples per expansion.
We denote the set of successful proof searches created in this process as S 0 .
Using S
we generate dataset D 0 by concatenating: (i) the initial tactic dataset ( proofstep objective ), (ii) a deduplicated set of proofsteps extracted from the proofs in S 0 ( proofstep objective ) and (iii) a deduplicated set of proofsize tuples (goals and proofsize) extracted from the full proof searches in S( proofsize objective ).
Note that the full proof searches in S 0 include goals that are visited but eventually remain unproved, which provides useful negative examples for the trained value function (even if these negatives may include provable goals that simply were not prioritized by the search).
Also note that S 0 doesn’t include failed proof searches.
We fine-tune θ 0 on D 0 for exactly one epoch (no use of validation data for early-stopping) to obtain our initial model θ 1 trained on both the proofstep objective and the proofsize objective .
θ 0 is used in our expert iteration setup as base model to fine-tune from at each iteration, and θ 1 is our first iterated model or mathlib bootstrapped model trained on both objectives.
We report in Table 1 the pass rates of θ 0 and θ 1 on mathlib-valid and miniF2F-valid and compare with previously reported pass rates for equivalent amounts of compute.
As reported in Polu & Sutskever (2020), training a value function to guide search greatly improves the pass rates of θ 1 on mathlib-valid .
Interestingly, the gap between θ 0 and θ 1 on miniF2F-valid is not as significant, demonstrating that training a value function on proofs sampled from mathlib-train has limited transfer to miniF2F-valid .
The main differences with Zheng et al.
(2022), potentially explaining the gap on miniF2F-valid ( 27 . 6% vs 23 .
9% ), consists in the new pre-training described in Section 4.1 as well as the use of a more recent mathlib checkpoint for the mix1 , mix2 and tactic datasets.
4. I TERATED SAMPLING AND TRAINING
Our expert iteration process takes as input: (i) a set of formal statements St , (ii) a function a : St −→ N indicating the number of proof search attempts to run per statement at each iteration, (iii) a base model θ 0 to fine-tune from at each iteration, and (iv) a mathlib bootstrapped model θ 1 trained on both objectives.
A high-level illustration of the iterated sampling and training is available in Appendix E.
Each iteration k consists in sampling proof searches for statements in St using θ k , filtering successful proof searches S k to extract a new dataset D k , and fine-tuning θ 0 on it to obtain θ k +1 , on which we can iterate.
To sample proof searches from St we use the best-first search described in Polu &
Sutskever (2020) with the value function described in Section 4.2.
We attempt a proof searches for each statement s ( s ∈ St ) with d = 512 expansions and e = 8 samples per expansion.
We denote the set of successful proof searches for iteration k as S k .
Using S k we generate datasets D k by concatenating: (i) the initial tactic dataset ( proofstep objective ), (ii) a deduplicated set of proofsteps extracted from the proofs in (cid:83) 1 ≤ i ≤ k S k ( proofstep objective ), and (iii) a deduplicated set of proofsize tuples (goals and proofsize) extracted from the full proof searches in (cid:83) 1 ≤ i ≤ k S k ( proofsize objective ).
We use a global deduplication across iterations for both proofsteps and proofsize tuples which we found to be important to maintain the stability of the expert iteration procedure.
This global deduplication is somewhat equivalent for each statement to growing a unique proof tree by aggregating all the proof searches that have been run for it across iterations.
This virtual proof tree accumulates a growing number of positive proof paths and visited goals that remain unproven.
We use these goals as negative examples for the proofsize objective , labeling them with an infinite proofsize.
Positive goals are deduplicated keeping the minimum proof sizes across proof searches.
Finally θ k is obtained by fine-tuning θ 0 for exactly one epoch on D k .
Note that the initial tactic dataset is included in each D k , despite θ 0 being already trained on it (along with mix1 ).
We found this repetition to be beneficial overall (as it adds the mathlib extracted proofsteps to our deduplicated per statements virtual proof trees) despite it leading to a slight overfit on the tactic dataset in terms of validation loss.
4. E XPERT ITERATION ON mathlib-train
In this section we propose to set St to the statements in mathlib-train , run our expert iteration process with it and report performance on both mathlib-valid and miniF2F-valid .
Performance is reported in terms of pass rate (percentage of successful proof searches) as a function of the number of attempts per statement, noted pass@k where k is the number of attempts per statement at test time.
To reduce noise in these metrics we run more than k attempts at test time (generally 32 to compute pass @1 and pass @8 ), averaging across attempts as needed to obtain a smoother pass@k value.
Given the large number of statements in mathlib-train (25k) we uniformly set a = 1 and use θand θ 1 as described in Section 4.3 and report pass@1 and pass@8 across 8 iterations in Figure 1.
The pass@1 on mathlib-valid goes from 56 .
3% for θ 1 to 62 .
6% for θ 9 .
The performance steadily improves and follows a clear logarithmic scaling law on mathlib-valid .
It is also notable that, initially, transfer to out-of-distribution miniF2F-valid appears limited but eventually kicks in as we reach better performance on mathlib-valid .
This demonstrates that the expert iteration process does not just overfit to mathlib but also leads to improved performance on out-of-distribution statements.
We define the cumulative pass rate at iteration k as the pass rate consisting of all proof searches up to iteration k .
Since we set a = 16 for evaluation on mathlib-valid and miniF2F-valid at each iteration, the cumulative pass rate at iteration k can be seen as a noisy ensembled pass@16k (multiple models ( θ k ), no averaging).
In Figure 2, we report this cumulative pass rate for two iteration loops, our normal one and a sampling-only loop where we skip re-training the model between iterations and solely sample from θ 1 .
This directly compares test-time compute scaling (scaling proof search attempts) to expert iteration scaling (interleaved training on new data sampled from mathlib-train ) and provides a very clear visualization of the gains of expert iteration.
For a fair comparison, we also report an adjusted compute line which approximates the test-time performance we would get at each iteration if we were to focus all the additional compute used by expert iteration (sampling proofs from mathlib-train as well as re-training models at each iteration) towards solely running proof searches against mathlib-valid .
As shown by Figure the scaling exponent of expert iteration is substantially higher than the scaling exponent associated with solely scaling test-time compute (running more proof searches), demonstrating the clear benefit of expert iteration.
We’ll denote the fully iterated model from this section as θ mathlib.
Even in the presence of ground-truth proofs for each of the statements in mathlib-train ( tactic dataset) expert iteration generates data that further improves the performance of the model.
The number of statements proved in mathlib-train goes from 17390 ( 67 . 8% ) at iteration 1 to( 76 . 0% ) at iteration 9, while the average proof length of these statements goes from 4 .
8 to 4 . 0 .
We hypothesize that this continuously improving performance through expert iteration stems from two effects: (i) the model finding new original proofs for the same statements and (ii)
the model closing marginally harder statements at each iteration – which in turn provides more useful training data for the next iteration.
By iteration 9 , the model is trained on more than 90% generated data.
We present in Appendix I a few examples of original proofs found by our models on mathlib-train compared with their ground-truth versions.
To verify our hypothesis that expert iteration is capable of closing a curriculum of increasingly difficult problems out of a set of problem statements, and that this capability is independent of having access to ground-truth proofs, we propose in the next section to study expert iteration applied to a synthetically generated set of problems for which we have fine-grained control on the difficulty of each statement.
S TATEMENT CURRICULUM LEARNING
In this section we focus on running expert iteration on synthetic statements generated by an inequality generator.
The use of synthetic statements enables us to control the difficulty of each statement to present evidence that expert iteration can hill-climb the intrinsic difficulty gradient of the resulting set of statements.
In particular, we show that, at fixed compute budget, expert iteration eventually closes proofs of hard statements that remain completely out of reach of simply sampling proof searches without interleaved training.
5. S YNTHETIC INEQUALITY GENERATOR
We designed a synthetic inequality statement generator for Lean in the spirit of the INT (Wu et al., 2021) generator.
The generator consists in generating inequalities from well known inequality theorems (AM-GM, Trivial inequality, Cauchy-Schwarz, Bernoulli, Young, Hölder) and composing them.
It is driven by two difficulty parameters: N D which controls depth of composition of inequalities and
N S which controls the complexity of the input expressions to the composed inequalities.
We provide details on its implementation in Appendix F.
Using this generator we generate a curriculum of 5600 inequality statements (for which we don’t have proofs), 100 for each values of 0 ≤ N S ≤ 7 and 0 ≤ N D ≤ 6 .
We denote this set of statements as synth-ineq .
To bootstrap our models capabilities on this specific task, we also generate 100 statements of low difficulty ( N D = 1 and N S = 5 ) and formalize a proof for each of these statements.
We refer to this dataset as synth-ineq-train .
In the rest of this paper we adjunct this training dataset to the tactic dataset used to train our models.
5. E XPERT ITERATION ON SYNTHETIC INEQUALITY STATEMENTS
In this section we propose to set St to the union of the statements in mathlib-train and synth-ineq .
Again, we uniformly set a = 1 and use θ 0 and θ 1 as described in Section 4.3, except that they are now also trained on synth-ineq-train .
Similarly to the previous section, we report in Figure 3 the cumulative pass rate for two loops, our standard expert iteration loop, and a proof search only loop where we do not interleave training between iterations.
The pass rates are reported split by values of N D (pooling together 0 ≤ N S ≤ 7 ) which we found to be the main driver for difficulty.
[Equation]
(cid:23) (cid:24) (cid:25) (cid:26) (cid:27) (cid:28) (cid:19) (cid:20)(cid:19) (cid:21)(cid:19) (cid:22)(cid:19) (cid:23)(cid:19) (cid:24)(cid:19) (cid:20) (cid:21) (cid:22) (cid:23) (cid:24) (cid:25) (cid:26) (cid:27) (cid:28) (cid:19) (cid:20)(cid:19) (cid:21)(cid:19) (cid:22)(cid:19) (cid:23)(cid:19) (cid:24)(cid:19) (cid:19) (cid:20) (cid:21) (cid:22) (cid:23) (cid:24) (cid:25) ([SHUW(cid:3)LWHUDWLRQ 6DPSOH(cid:3)RQO\
Despite the challenging nature of these synthetic inequalities, Figure 3 demonstrates that expert iteration is capable of learning the intrinsic curriculum induced by synth-ineq .
In particular, expert iteration is capable of closing 6 problems of difficulty N D = 6 without having been provided with any seed ground-truth proof for this difficulty level.
Note that difficulty N D
= 6 remains completely out of reach of simply scaling the number of attempts per statements (the sample only loop remaining stuck at 0 for N D = 6 ).
This confirms on our synthetic statements dataset synth-ineq that not only expert iteration is capable of learning the curricula occurring in a set of statements, but this process also enables the emergence of new capabilities without the need for ground-truth proofs (ability to close, highly challenging, deeply composed inequalities).
T ARGETING miniF2F
Motivated by the results from Section 5, we curated and manually formalized a set of math exercises denoted as miniF2F-curriculum to target miniF2F .
miniF2F-curriculum contains 327 statements from various sources, with their provenance and analysis detailed in Appendix G. miniF2F statements being quite out of distribution compared to mathlib statements (which typically are generic theorems and lemmas), we hypothesized that if the difficulty of miniF2F-curriculum was made varied enough, expert iteration could potentially leverage it to effectively shift our models’ distribution closer to miniF2F ’s, and in turn, improve their eventual performance on it.
6. T RANSFER TO miniF2F
In this section we propose to set St to the union of the statements in mathlib-train , synth-ineq and miniF2F-curriculum .
We uniformly set a = 1 on mathlib-train and synth-ineq and a = 8 on miniF2F-curriculum and use θ 0 and θ 1 as described in Section 5.
Similarly to previous sections, we report in Figure 4 (left) the cumulative pass rate on miniF2F-valid of our full curriculum expert iteration loop and compare them with the mathlib-train only expert iteration from Section 4.5.
Since more compute is deployed in our full-curriculum loop (more statements), we also report a mathlib-train only loop taking a = 2 .
At the end of the expert iteration, 100 out of the 327 statements from miniF2F-curriculum end up being closed, suggesting a lack of density in our manually formalized set of statement.
We also report in Figure 4 (right) the pass@1 and pass@8 for our full curriculum expert iteration loop.
The steady improvement on miniF2F-valid shows that the expert iteration procedure we propose does not overfit on the statements that compose the curriculum it uses.
Despite the potential inefficiency of our curriculum, the improved performance associated with its use demonstrates, as hypothesized, an effective transfer between miniF2F-curriculum, synth-ineq and miniF2F-valid through expert iteration.
We will denote the fully iterated model from this section as θ full 9 .
(cid:20) (cid:21) (cid:22) (cid:23)
[Figure]
(cid:20)
[Figure]
6. R ESULTS
We report in Table 2 the pass rates on mathlib-{valid, test} and miniF2F-{valid, test} for the models trained in previous sections, namely θ 1 , θ mathlib, and θ full 9 .
We achieve a 47 .
3% pass rate (using a =attempts) on miniF2F-valid and a 36 . 6% pass rate on miniF2F-test , substantially improving from the previous state-of-the-art (Zheng et al., 2022).
These results include the resolution of 26 AMC12 problems, 6 AIME problems and 2 IMO-adapted problems.
Out of these statements, 4 AMC12 problems ( amc12b_2020_p5 , amc12a_2009_p9 ,
[Table]
amc12a_2003_p24 , amc12b_2003_p17 ), 2 AIME problems ( aime_1984_p1 , aime_1990_p4 ), and 2 IMO-adapted problems ( imo_1961_p1 2 , imo_1964_p2 ) are uniquely solved by expert iterated models, the two IMO-adapted and the two AIME problems being uniquely solved by θ full 9 .
We provide a selection of the proofs found by our models for these statements as well as a qualitative analysis of them in Appendix J. Also, we achieve a new state-of-the-art: higher than 75% pass rate (using a = 64 attempts) on mathlib-{valid, test} , suggesting that our models could potentially be effectively leveraged as proof assistants in the formalization efforts associated with mathlib .
D ISCUSSION AND L IMITATION
Throughout this paper, we used a single model size (774m trainable parameters).
We refer readers to Appendix H for more discussion on model size, compute budget and training time.
Despite our models’ capability, as discussed in Appendix J.1, to generate cuts and witnesses, we believe that their current main limitation lies in their inability (under our proposed search procedure) to chain more than 2 or 3 non-trivial steps of mathematical reasoning, preventing them from consistently solving challenging olympiad problems.
We’ve been repeatedly impressed by the complexity of some of the proofsteps generated by our models.
But, proofs requiring many of such reasoning steps remain beyond our current compute horizon.
Even if we solved a selection of challenging olympiad problems, our models are still far from being competitive with the brightest students in these competitions.
While our models have demonstrated some capabilities to generate cuts, the cuts they generate are often shallow (they involve only a few proofsteps and don’t necessarily deeply change the structure of the proof–we refer the reader to the Cut-Elimination theorem and Carbone & Semmes (1996) for a discussion of the influence of cuts on proof size).
We believe that studying language models’ ability to generate cuts, and designing search procedures that leverage that capability (related ideas can be found in Czechowski et al.
(2021)), are interesting avenues of research to alleviate this limitation.
C ONCLUSION
In this paper we presented an expert iteration procedure for GPT-f
(Polu & Sutskever, 2020), demonstrating that it is capable of solving a curriculum of increasingly difficult problems out of a set of formal statements of sufficiently varied difficulty.
Our results suggest that the lack of self-play in the formal mathematics setup can be effectively compensated for by automatically/manually curated sets of formal statements, which are much cheaper to formalize than full proofs.
Finally, we hope that the statement curriculum learning methodology we presented in this work will help accelerate progress in automated reasoning, especially if scaled with automated generation and curation of formal statements in the future.
A R ELATED W ORK
Deep learning applied to premise selection and proof guidance Early applications of deep learning to formal mathematics focused primarily on premise selection and proof guidance.
DeepMath (Irving et al., 2016) explored the use of CNNs and RNNs to predict whether a premise is useful to demonstrate a given conjecture.
Their results were later improved with FormulaNet (Wang et al., 2017) by the use of graph neural networks, reminiscent of NeuroSAT
(Selsam et al., 2019).
Proof guidance consists in selecting the next clause to process inside an automated theorem prover.
Loos et al.
(2017) investigated the use of models similar to DeepMath’s for proof guidance and demonstrated a significant uplift on the Mizar library.
More recently Firoiu et al.
(2021) demonstrated the potential of deep learning techniques to be competitive with E prover’s heuristics when applied to resolution calculus while training on fully synthetic data.
Deep learning applied to automated theorem-proving HOList (Bansal et al., 2019b) proposes a formal environment based on HOL Light.
They achieve their best performance (Bansal et al., 2019a) with a GNN model designed for premise selection and the use of exploration.
The same team studied the use of a skip-tree objective with Transformers on formal statements (Rabe et al., 2021), demonstrating, along with GPT-f
(Polu & Sutskever, 2020), the potential of leveraging Transformers for formal reasoning.
GamePad (Huang et al., 2019) and CoqGymn/ASTactic (Yang & Deng, 2019) introduce environments based on the Coq theorem prover.
ASTactic generates tactics as programs by sequentially expanding a partial abstract syntax tree.
Urban & Jakubuv (2020) studied the capability of GPT-2 to produce useful conjectures for the Mizar library and IsarStep (Li et al., 2021) explored the synthesis of intermediate propositions in declarative proofs for Isabelle/HOL using Transformers.
Targeting miniF2F Lample et al.
(2022) designed HyperTree Proof Search (HTPS), an online training procedure targeting Lean, Metamath and hand-crafted environment named Equations.
Lample et al.
(2022) report 41% pass-rate on miniF2F-test and 42.5% pass-rate on miniF2F-curriculum in
Lean (de Moura et al., 2015; lea) setup.
Thor (Jiang et al., 2022) combined language model and
Sledgehammer (Paulson, 2010) and achieved 29.9% pass-rate on miniF2F-test in Isabelle setup, which is later improved to 35.2% by Wu et al.
leveraging autoformalization and expert iteration.
B L EAN - GYM
lean-gym presents the following API:
• init-search : declaration → tactic _ state .
Takes a declaration name (a theorem name from the loaded library) and initializes a search while setting the run-time environment at that particular declaration.
It returns the initial tactic state along with a fresh search_id and tactic_state_id .
• run_tac : ( tactic _ state, tactic ) → tactic _ state .
Takes a search_id and a tactic_state_id to identify a tactic state, as well as a tactic string to apply to it.
It returns a new tactic state and its associated tactic_state_id .
Below is an example in-terminal trace demonstrating the use of lean-gym ’s
REPL interface: $ lean --run src/repl.lean ["init_search", ["int.prime.dvd_mul", ""]] { "error":null, "search_id":"0", "tactic_state":" ⊢ ∀ {m n : Z } {p : N }, nat.prime p
→ ↑ p | m * n → p | m.nat_abs ∨ p | n.nat_abs", "tactic_state_id":"0" } ...
["run_tac",["1","1","apply (nat.prime.dvd_mul hp).mp"]] {
"error":null, "search_id":"1", "tactic_state":"m n : Z , p : N , hp : nat.prime p, h : ↑ p | m * n ⊢ p | m.nat_abs * n.nat_abs", "tactic_state_id":"2" } ...
Using lean-gym is virtually equivalent to opening a Lean editor at a specific theorem, deleting its proof and interacting with Lean to reconstruct it.
Providing a REPL interface over the standard input/output makes it very easy to integrate lean-gym from any programming language.
Writing a wrapper in Python, as an example, only takes a few dozen lines of code.
Since lean-gym is a Lean program, managing the loaded libraries is done directly using Lean’s own infrastructure (using leanpkg.toml ), making it quite straightforward to have access to both mathlib and miniF2F statements from the same lean-gym instance.
Note that lean-gym is stateful, meaning that distributing proof searches on multiple lean-gym instances requires to track which instance is associated with which proof search.
In practice, we were able to scale the use of lean-gym to thousands of cores running thousands of proof searches in parallel.
Finally, lean-gym ’s REPL interface is blocking, preventing inner-proof search parallelization, though this limitation can probably be removed in the future.
C W EB M ATH
Our updated WebMath pre-training dataset consists in the mix presented in table 3.
[Table]
As demonstrated in table 3, we empirically up-weighted (compared to their token size) parts of
WebMath with high-quality mathematical content while making sure they don’t overfit (despite running >1 epochs for some of them).
We also included PACT mix2 directly in the WebMath pre-training to avoid having to sequence more than two pre-training phases to prepare Lean models.
D E XAMPLE OF MINI F2F INPUT , L EAN ENVIRONMENT AND M ODEL OUTPUT
We illustrate an example of the interaction between Lean environment and our model.
In the figure shown below, the model has 1 output for each current goal (corresponding to 1 expand budget).
The model could have expand budget bigger than 1, in which case the search procedure becomes a tree.
E I LLUSTRATION OF E XPERT I TERATION
F S YNTHETIC INEQUALITIES F. D ESIGN
The generator consists of three phases:
Seed expressions generation The first phase consists in generating seed expressions for which we track the sign.
We start by initializing an expression set E composed of tuples of expressions and sign constraints, by generating n v variable names (letters) assumed strictly positive as well as n n integers (for which we know the sign).
For N S rounds, we compose elements of E using unary ( log ( · ) , log (1 / · ) , sqrt ( · ) ) or binary operations ( + , − , × , /, ∧ , max, min ) for which we can deduce the sign based on the sign condition of the input expression(s) and re-inject the resulting expression and sign constraint in E .
This produces a set E of signed seed expressions of size n v + n n + N S .
Inequality composition The second phase consists in generating inequalities from well known inequality theorems (AM-GM, Trivial inequality, Cauchy-Schwarz, Bernoulli, Young, Hölder) taking as input to these theorems expressions from E based on the sign constraints required for each theorem.
We finally compose these inequalities N D times using compositions theorems detailed in F.2.
The resulting inequality is a composed inequality of depth N D based on n v + n n + N S seed expressions.
Simplification We finally post-process these inequalities so that they are parsable by Lean and run them through Lean’s simp tactic for a final simplification.
N D and N S together control for the difficulty of the resulting inequality.
N D controls depth of composition, while N S controls for obfuscation as it increases the complexity of the input expressions to the composed inequalities.
When sampling inequalities, we n n = 4 and randomly sample 2 ≤ n v ≤ 8 at each generation.
We report below examples of generated inequalities for various values of N D and N S .
F. L IST OF INEQUALITY COMPOSITION THEOREMS
Below is the list of theorem names from mathlib that we use to compose inequalities together.
One third of the time, we only transform the current composed inequality with one of the following theorems:
• neg_le_neg • inv_le_inv • mul_self_le_mul_self • div_le_one_of_le
We otherwise compose the current composed inequality with a newly generated inequality using the following theorems:
• mul_le_mul • add_le_add • div_le_div • mul_le_mul_of_nonneg • le_mul_of_ratio
[Equation]
[Table]
) (h5 : 0 < f) : (1: R ) + (99: R ) *
c + (a / f / (6: R ) + a * (a / f) /
((d ^ 2 + a ^ 2 / f ^ 2) * (real.log ((59: R ) + f) ^ 2 + c ^ 2))) ≤ ((a / f) ^ ((3: R ) / (2: R )) /
((3: R ) / (2: R ))
+ a ^ 3 / (3: R )) /
(real.log ((59: R ) + f) *
d + a / f * c) ^ 2 * (c / (c / (70: R )))
+ a / f
+ (c + (1: R )) ^ 99 :=
sorry
G MINI F2F- CURRICULUM
The 327 statements of miniF2F-curriculum 3 are manually formalized from:
• AOPS Books (Lehoczky & Rusczyk, a;b) : 302 examples and exercises.
The books are classic problem solving textbooks for students in grades 7-12 preparing for contests such as AMCs and AIMEs.
We skipped problems that were too challenging to formalize due to missing infrastructure in mathlib or non-suitable format for formalization (see section Formalization effort and challenges in Zheng et al.
• MATH
(Hendrycks et al., 2021) dataset : 25 problems.
All problems were drawn from the train split of the dataset, focusing on difficulty 5 problems ( miniF2F only contains problems from the test split).
We verified (based on problem provenance and manual inspection of statements) that miniF2Fcurriculum had an empty intersection with miniF2F-{test, valid} .
We refer to Zheng et al.
for more details on the formalization procedure and the typical time needed for it as these problems were formalized in similar conditions.
H M ODEL S IZE
Other than the single model size we use in the experiment reported in the main text (774m trainable parameters), we briefly experimented with different model sizes (not reported in this paper) and found that model size scaling is not as straightforward as in the case of unsupervised learning (Kaplan et al., 2020).
We found that bigger models are better, in the sense that they consistently exhibit higher pass@1 .
But, they are also much more expensive to sample from.
And despite their pass@1 being higher, it is often the case that for a fixed amount of compute, sampling more attempts from a smaller model leads to a better final performance.
For the compute budget we had available, we estimated the model size we used to be a compelling trade-off.
We leave as future work a more thorough study of these dynamics to better understand the different compute frontiers involved.
Indicatively, with our 774m parameters model, running a full expert iteration to train θ full 9 required about 2000 A100 days of compute.
Running one full proof search ( a = 1 d = 512 e = 8 ) when properly parallelised, requires on average about 0 .
1 A100 hour of compute.
I E XAMPLE PROOFS FROM mathlib-train
We present in this section original proofs found by our models from mathlib-train , compared with their ground-truth version.
_eq_of_inverse
Statement lemma comap_eq_of_inverse {f : filter α } {g : filter β } { ϕ : α → β } ( ψ : β → α ) (eq : ψ ◦ ϕ
= id) (h ϕ : tendsto ϕ f g) (h ψ : tendsto ψ g f) : comap ϕ g = f :=
Ground-truth beginrefine ((comap_mono $ map_le_iff_le_comap.1 h ψ ).trans _).antisymm (map_le_iff_le_comap.1 h ϕ ), rw [comap_comap, eq, comap_id], exact le_rfl end
Model proof beginrefine le_antisymm _ (filter.map_le_iff_le_comap.1 h ϕ ), refine λ s hs, _, rw mem_comap, use [ ψ − ¹ ' s, h ψ hs], rw [ ← preimage_comp, eq, preimage_id] end
[Table]
beginrw[ ← sum_Ico_eq_sub _ hnm], congr,applyfinset.ext, simp[Ico.mem
[Table]
[Table]
tauto end
[Table]
lemma prod_inv_distrib : ( (cid:81) x in s, (f x) − ¹ ) =
( (cid:81) x
in s, f x) − ¹ :=
Ground-truth beginclassical, by_casesh : ∃ x ∈ s, f x = 0, { simpa [prod_eq_zero_iff.mpr h, prod_eq_zero_iff] using h }, { push_neg at h, have h' :
= prod_ne_zero_iff.mpr h, have hf : ∀ x ∈ s, (f x) − ¹ * f x = 1 := λ x hx, inv_mul_cancel (h x hx), apply mul right_cancel' _mul_distrib, prod rfl hf] } end Model proof
beginclassical; induction s using finset.induction_on with a s has ih, { simp, }, simp only [has, prod_insert has, mul_inv_rev'], finish end
J E XAMPLE PROOFS FROM miniF2F-{test, valid, curriculum}
We present in this section proofs found by our models from miniF2F-{test, valid, curriculum} , demonstrating some of the capabilities emerging from our training procedure.
J. Q UALITATIVE ANALYSIS OF PROOFS
We provide qualitative insights in the nature of the proofs found by our models, which we believe are useful to build a better intuition of their capabilities beyond pass rate numbers.
Throughout this section, we refer to statements and solutions found by our models that are presented in Appendix J along with comments describing the specificity of each proof.
First, we observe that a large number of olympiad problems that are designed to be computationally challenging for humans are rendered trivial for our models through the use of Lean tactics.
As an example, mathd_numbertheory_447 which is not necessarily considered straightforward for humans, can be closed in Lean by a simple refl (proof found by our models).
In recent years, Lean’s mathlib community has developed high-powered tactics such as linarith/nlinarith (solves (non)linear inequalities), norm_num (normalizes numerical expressions), simp (simplifies goals and hypotheses) and ring (normalizes expressions in a ring).
These tactics can be used with arguments to guide their underlying search procedure.
As mentioned in
Zheng et al.
, we confirm here that our models acquire advanced capabilities to leverage these high-level tactics by providing exogenous arguments which are not present in the current tactic state.
The generation of these exogenous arguments through language modeling seems to require a non-trivial amount of mathematical intuition.
imo_1964_p2 , imo_1961_p1 and aime_1990_p15 are good examples of such uses.
We have also observed a number of proofs that require multiple non-trivial reasoning steps through the use of lower-level tactics such as use , have , or by_cases that generally involve producing a witness or chaining implications, requiring the generation of context specific exogenous terms.
These interesting reasoning steps are structurally different from simple normalization, simplification and rewriting of hypotheses or goals because they heavily rely on our models ability to generate meaningful cuts or witnesses.
This capability is, in our opinion, the most exciting stepping stone towards solving more challenging mathematical problems.
See, aopsbook_v2_c8_ex1 , amc12b_2020_p and mathd_train_algebra_217 for examples of such proofs.
More generally, we also observe that proofs generated by our models have a distinctive style compared to proofs formalized by humans.
This stems in part from the model’s capability to leverage high-level tactics in a way that is challenging for humans as discussed in this section (e.g. one-liners such as nlinarith [sq_nonneg (x - y), sq_nonneg (y - z)] where humans would generally decompose the problem in a less machine-like way).
Additionally, as a result of our search procedure and despite the bias towards shorter proofs introduced by our value function, extraneous proofsteps (such as reversion/introduction of hypotheses, or no-op rewrites) are often interleaved with useful ones, which rarely happens in human formalizations.
imo_1961_p
Natural language
Solve the system of equations:
[Equation]
where a and b are constants.
Give the conditions that a and b must satisfy so that x, y, z (the solutions of the system) are distinct positive numbers.
Note : the formalized statement in miniF2F is a weaker problem as it focuses on the second part of the question, providing the actual conditions, and asking for a proof that the requirement entails them.
Model proof
theorem
[Equation]
beginrevert_all, intros,rwmul_comm, split,{nlinarith[sq_nonneg (x - y), sq_nonneg (y - z)], }, split,{nlinarith [sq_nonneg (z - 1)], }, revert h 3 h 4 , field_simp [mul_comm a b], rw [mul_comm, ← h 5 ], contrapose!, rwmul_commat h 6 , rw mul_comm, intro h, nlinarith [sq_nonneg (x - y), sq_nonneg (y - z)] end
Comments
The model is able to close this problem by splitting into cases, contraposing for the last case and using nlinarith .
It must be noted that the arguments for the first two nlinarith uses are not necessary, however the [sq_nonneg (x - y), sq_nonneg
[Table]
[Table]
imo_1964_p
Natural language
Suppose a, b, c are the sides of a triangle.
Prove that a 2 ( b + c − a ) + b 2 ( c + a − b ) + c 2 ( a + b − c ) ≤ 3 abc
Model proof theorem imo_1964_p2 (a b c : R )
(h 0 : 0 < a ∧ 0 <
b ∧ 0 < c) (h 1 : c < a + b) (h 2 : b < a + c) (h 3 : a < b + c) : a^2 * (b + c - a) + b^2 * (c + a - b) + c^2 * (a + b - c) ≤ 3 * a * b * c := beginnlinarith [sq_nonneg (b - a), sq_nonneg (c - b), sq_nonneg (a - c), sq_nonneg (c - a)] end
Comments
The model is able to close an IMO problem in one-line.
It correctly provides exogenous arguments to nlinarith , which are necessary to close the goal.
Note that either one of the last two arguments in the sequence [sq_nonneg (b - a), sq_nonneg (c - b), sq_nonneg (a - c), sq_nonneg (c - a)] can be omitted.
aime_1990_p
[Table]
Find ax 5 + by 5 if the real numbers a, b, x, and y satisfy the equations ax + by = 3 , ax 2 + by 2 = 7 , ax 3 + by 3 = 16 , ax 4 + by 4 = 42 .
Note : the formalized statement in miniF2F provides the answer and asks for a proof of it.
Model proof
theorem
[Table] [Table] [Table] : R ) (h
[Equation]
[Equation] (h 1 :
[Equation]
h
[Equation]
[Equation]
h
[Equation]
[Equation] [Equation] [Equation]
= 20 := beginrevert_all, introsabx y h [Equation] h
[Equation] h [Equation] , ring_nf at h 1 h
[Equation] , rw ← sub_eq_zero at h 1 , nlinarith [sq_nonneg (x - y), sq_nonneg (a + b - 2), sq_nonneg (x + y - (2: R )), sq_nonneg (a - b - 2)] end
Comments
The model is able to close a challenging AIME problem by providing crucial ex- ogenous arguments sq_nonneg (x - y) and sq_nonneg (x + y - ( 2 : R )) to nlinarith , which are required to close the goal (while the other two can be removed). mathd_train_
[Table]
Natural language
Let f ( x ) =
Ax + B and g ( x ) =
Bx + A , where A ̸ = B .
If f ( g ( x )) − g ( f ( x ))
=
B − A , what is A + B ?
Note : the formalized statement in our curriculum provides the answer and asks for a proof of it.
Model proof
theorem mathd_train_algebra_217 (a b : R ) (f g : R → R ) (h 0 : ∀ x, f x = a * x + b) (h 1 : ∀ x, f x = b * x + a) (h 2 : a ̸ = b) (h 3 : ∀ x, f (g x) - g (f x) =
b - a) : a + b = 0 :
= beginrevert_all, introsab,introsfg, contrapose!,rintro 〈 h 0 , 〈 h 1 , h 2 〉〉 , use (0 : R ), simp only [sub_eq_iff_eq_add, h 0 , mul_zero] at *, simp only [*, zero_add], norm_num at h 0 end
Comments
The model is able to close the goal by contraposing, supplying a witness by the use of use ( 0 : R ) and finally leveraging the simp and norm_num .
This example demonstrates the model’s ability to chain multiple non-trivial steps of reasoning including the generation of witnesses.
amc12b_2020_p
Natural language
For all integers n ≥ 9 , the value of ( n + 2)!
− ( n + 1)!
n ! is always which of the following?
a multiple
[Table]
theorem amc12b_2020_p6 (n : N ) (h 0 : 9 ≤ n) : ∃ x : N , (x: R )
^2 =
(nat.factorial (n + 2) - nat.factorial (n + 1)) / nat.factorial n := beginrevert_all, introsnhn,usen+1, field_simp [nat.factorial_ne_zero, pow_succ'], ring_exp end
Comments
The model exhibits here its ability to implicitly interpret factorial expressions by guessing the correct answer with the use of use n + 1 and straightforwardly proving its claim right after.
[Table]
The expression 24 x 2 − 19 x − 35 can be written as ( Ax − 5)(2 Bx + C ) , where A ,
B , and C are positive numbers.
Find AB − 3 C .
Note : the formalized statement in miniF2F provides the answer and asks for a proof of it.
[Table]
theorem mathd_algebra_140 (a b c : R ) (h 0 : 0 < a ∧ 0 <
b ∧ 0 < c) (h 1 : ∀ x, 24 * x^2 - 19 * x - {+
+} = (((a * x) - 5) * ((2 * (b * x))
+ c)))
: a * b - 3 * c = -9 :
= beginrevert_all, rintroabc h 1 , rw mul_comm, rw ← sub_eq_zero, field_simp, rwsub_eq_add_neg, rw ← eq_neg_iff_add_eq_zero, rw [mul_comm, ← sub_eq_add_neg, sub_eq_add_neg, ← sub_eq_add_neg], rw ← sub_eq_zero, simp only [mul_add, neg_mul_eq_neg_mul_symm, mul_comm, add_left_comm],norm_num, simponly[pow_two],introh 0 , rw ← sub_eq_zero, linarith [h 1 .2, h 1 .1, h 0 3, h 0 1, h 0 2] end
Comments
The model mostly "struggles" to make progress up to the last line.
There, it presents us with the idea to specialize the statement given in h 1 (which became h 0 in the process of the proof) at three consecutive natural numbers 1 , 2 , 3 which closes the goal with nlinarith .
This proof is interesting as it demonstrates the model’s ability to evaluate symbolic expressions implicitly.
[Table]
theorem aime_1984_p1 (
[Table] [Table]
) (h 0 : ∀ n, u (n + 1) =
u n + 1) (h 1 : (cid:80) k in finset.range 98, u k.succ = 137) : (cid:80) k in finset.range 49, u (2 * k.succ) =
93 := beginrevert_all, simp[finset.sum_range_succ],ring, simp[pow_succ,mul_comm,add_left_comm, add_assoc], simp [two_mul], rintro p, revert p, simp [add_comm], intros p hp, simp [hp], ring_nf, intros,
[Table]
Comments
The model finds a rather short proof of the statement compared to typ- ical human formalizations (see https://gist.github.com/MantasBaksys/ 3efd5c0d42f440d16c09076db42822f2 ).
aopsbook_v2_c8_ex
Natural language
Consider the sequence { nn +1 } = 12 , 23 , 34 , ...
Prove that this sequence tends to 1 as n → ∞
Model proof theorem aopsbook_v2_c8_ex1 (u : N → R ) (h 0 : ∀ n, u n = n / (n + 1)) : filter.tendsto u filter.at_top ( N 1) :
= beginrevert_all, simp[ ← nnreal.coe_one], norm_cast, intros,reverth 0 , assume h, simp [tendsto_const_nhds, ← nnreal.coe_one, h], revert u, assume f, norm_num,rwtendsto_iff_norm_tendsto_zero, assumeH,converttendsto_norm_zero.comp tendsto_one_div_add_at_top_nhds_0_nat,funextn, haveh 0 :
(((n: R ) + 1) :
R ) ̸ = 0, { norm_cast, exact n.succ_ne_zero, }, rw H, field_simp
[h 0 , norm_neg], ring,rw[ ← sub_eq_zero], simp [ ← sub_eq_zero], simp [sub_eq_add_neg], end
Comments
An interesting example of a generated have statement, denoted by h 0 , which is introduced as a cut in order to simplify the expression containing divisions by using field_simp[h 0 , norm_neg] at a later step.
mathd_numbertheory
Natural language
What is the sum of the units digits of all the multiples of 3 between 0 and 50 ?
Note : the formalized statement in miniF2F provides the answer and asks for a proof of it.
theorem mathd_numbertheory_447 : (cid:80) k in finset.filter ( λ x, 3 | x) (finset.erase (finset.range 50) 0),
[Table]
[Table]
beginreflend
Comments
Because the predicate λ x, 3|x is registered as decidable over N , we can state the problem by using finset.filter , which is computable.
Hence, refl is able to close the goal.